metadata
base_model: Snowflake/snowflake-arctic-embed-m
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:1490
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: >-
How can I list the 'kubernetes-cluster' resources that are accessible by
service connectors in my ZenML workspace?
sentences:
- >-
lly registered orchestrator `<ORCHESTRATOR_NAME>`.$ zenml
service-connector list-resources --resource-type kubernetes-cluster -e
The following 'kubernetes-cluster' resources can be accessed by service
connectors configured in your workspace:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │
CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨
┃ e33c9fac-5daa-48b2-87bb-0187d3782cde │ aws-iam-multi-eu │ 🔶
aws │ 🌀 kubernetes-cluster │ kubeflowmultitenant ┃
┃ │
│ │ │ zenbox ┃
┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨
┃ ed528d5a-d6cb-4fc4-bc52-c3d2d01643e5 │ aws-iam-multi-us │ 🔶
aws │ 🌀 kubernetes-cluster │ zenhacks-cluster ┃
┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨
┃ 1c54b32a-4889-4417-abbd-42d3ace3d03a │ gcp-sa-multi │ 🔵
gcp │ 🌀 kubernetes-cluster │ zenml-test-cluster ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┛
- >-
Name your pipeline runs
In the output logs of a pipeline run you will see the name of the run:
Pipeline run training_pipeline-2023_05_24-12_41_04_576473 has finished
in 3.742s.
This name is automatically generated based on the current date and time.
To change the name for a run, pass run_name as a parameter to the
with_options() method:
training_pipeline = training_pipeline.with_options(
run_name="custom_pipeline_run_name"
training_pipeline()
Pipeline run names must be unique, so if you plan to run your pipelines
multiple times or run them on a schedule, make sure to either compute
the run name dynamically or include one of the following placeholders
that ZenML will replace:
{{date}} will resolve to the current date, e.g. 2023_02_19
{{time}} will resolve to the current time, e.g. 11_07_09_326492
training_pipeline = training_pipeline.with_options(
run_name=f"custom_pipeline_run_name_{{date}}_{{time}}"
training_pipeline()
Be sure to include the f string prefix to allow for the placeholders to
be replaced, as shown in the example above. Without the f prefix, the
placeholders will not be replaced.
PreviousUsing a custom step invocation ID
NextUse failure/success hooks
Last updated 19 days ago
- >2-
a_new_local_stack -o default -a my_artifact_storestack : This is the CLI group that enables interactions with the stacks
register: Here we want to register a new stack. Explore other operations
withzenml stack --help.
a_new_local_stack : This is the unique name that the stack will have.
--orchestrator or -o are used to specify which orchestrator to use for
the stack
--artifact-store or -a are used to specify which artifact store to use
for the stack
The output for the command should look something like this:
Using the default local database.
Running with active workspace: 'default' (repository)
Stack 'a_new_local_stack' successfully registered!
You can inspect the stack with the following command:
zenml stack describe a_new_local_stack
Which will give you an output like this:
Stack Configuration
┏━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┓
┃ COMPONENT_TYPE │ COMPONENT_NAME ┃
┠────────────────┼───────────────────┨
┃ ORCHESTRATOR │ default ┃
┠────────────────┼───────────────────┨
┃ ARTIFACT_STORE │ my_artifact_store ┃
┗━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┛
'a_new_local_stack' stack
Stack 'a_new_local_stack' with id '...' is owned by user default and is
'private'.
Switch stacks with our VS Code extension
If you are using our VS Code extension, you can easily view and switch
your stacks by opening the sidebar (click on the ZenML icon). You can
then click on the stack you want to switch to as well as view the stack
components it's made up of.
Run a pipeline on the new local stack
Let's use the pipeline in our starter project from the previous guide to
see it in action.
If you have not already, clone the starter template:
pip install "zenml[templates,server]" notebook
zenml integration install sklearn -y
mkdir zenml_starter
cd zenml_starter
zenml init --template starter --template-with-defaults
pip install -r requirements.txt
The starter template is the same as the ZenML quickstart. You can clone
it like so:
- source_sentence: How can I explicitly name my model version in ZenML?
sentences:
- >-
strator supports specifying resources in what way.If you're using an
orchestrator which does not support this feature or its underlying
infrastructure does not cover your requirements, you can also take a
look at step operators which allow you to execute individual steps of
../...your pipeline in environments independent of your orchestrator.
Ensure your container is CUDA-enabled
To run steps or pipelines on GPUs, it's crucial to have the necessary
CUDA tools installed in the environment. This section will guide you on
how to configure your environment to utilize GPU capabilities
effectively.
Note that these configuration changes are required for the GPU hardware
to be properly utilized. If you don't update the settings, your steps
might run, but they will not see any boost in performance from the
custom hardware.
All steps running on GPU-backed hardware will be executed within a
containerized environment, whether you're using the local Docker
orchestrator or a cloud instance of Kubeflow. Therefore, you need to
make two amendments to your Docker settings for the relevant steps:
1. Specify a CUDA-enabled parent image in your DockerSettings
For complete details, refer to the containerization page that explains
how to do this. As an example, if you want to use the latest
CUDA-enabled official PyTorch image for your entire pipeline run, you
can include the following code:
from zenml import pipeline
from zenml.config import DockerSettings
docker_settings =
DockerSettings(parent_image="pytorch/pytorch:1.12.1-cuda11.3-cudnn8-runtime")
@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
...
For TensorFlow, you might use the tensorflow/tensorflow:latest-gpu
image, as detailed in the official TensorFlow documentation or their
DockerHub overview.
2. Add ZenML as an explicit pip requirement
- >-
Reuse Docker builds to speed up Docker build times
Avoid building Docker images each time a pipeline runs
When using containerized components in your stack, ZenML needs to build
Docker images to remotely execute your code. Building Docker images
without connecting a git repository includes your step code in the built
Docker image. This, however, means that new Docker images will be built
and pushed whenever you make changes to any of your source files.
One way of skipping Docker builds each time is to pass in the ID of a
build as you run the pipeline:
my_pipeline = my_pipeline.with_options(build=<BUILD_ID>)
or when running a pipeline from the CLI:
zenml pipeline run <PIPELINE_NAME> --build=<BUILD_ID>
Please note, that this means specifying a custom build when running a
pipeline will not run the code on your client machine but will use the
code included in the Docker images of the build. As a consequence, even
if you make local code changes, reusing a build will always execute the
code bundled in the Docker image, rather than the local code. Therefore,
if you would like to reuse a Docker build AND make sure your local code
changes are also downloaded into the image, you need to connect a git
repository.
PreviousWhich files are built into the image
NextUse code repositories to automate Docker build reuse
Last updated 19 days ago
- >-
Controlling Model versions
Each model can have many versions. Model versions are a way for you to
track different iterations of your training process, complete with some
extra dashboard and API functionality to support the full ML lifecycle.
E.g. Based on your business rules during training, you can associate
model version with stages and promote them to production. You have an
interface that allows you to link these versions with non-technical
artifacts and data, e.g. business data, datasets, or even stages in your
process and workflow.
Model versions are created implicitly as you are running your machine
learning training, so you don't have to immediately think about this. If
you want more control over versions, our API has you covered, with an
option to explicitly name your versions.
Explicitly name your model version
If you want to explicitly name your model version, you can do so by
passing in the version argument to the Model object. If you don't do
this, ZenML will automatically generate a version number for you.
from zenml import Model, step, pipeline
model= Model(
name="my_model",
version="1.0.5"
@step(model=model)
def svc_trainer(...) -> ...:
...
@pipeline(model=model)
def training_pipeline( ... ):
Here we are specifically setting the model configuration for a
particular step or for the pipeline as a whole.
Please note in the above example if the model version exists, it is
automatically associated with the pipeline and becomes active in the
pipeline context. Therefore, a user should be careful and intentional as
to whether you want to create a new pipeline, or fetch an existing one.
See below for an example of fetching a model from an existing
version/stage.
Fetching model versions by stage
- source_sentence: >-
What are the different roles available for users in an organization within
ZenML Pro?
sentences:
- >-
User Management
In ZenML Pro, there is a slightly different entity hierarchy as compared
to the open-source ZenML framework. This document walks you through the
key differences and new concepts that are pro-only.
Organizations, Tenants, and Roles
ZenML Pro arranges various aspects of your work experience around the
concept of an Organization. This is the top-most level structure within
the ZenML Cloud environment. Generally, an organization contains a group
of users and one or more tenants. Tenants are individual, isolated
deployments of the ZenML server.
Every user in an organization has a distinct role. Each role configures
what they can view, modify, and their level of involvement in
collaborative tasks. A role thus helps determine the level of access
that a user has within an organization.
The admin has all permissions on an organization. They are allowed to
add members, adjust the billing information and assign roles. The editor
can still fully manage tenants and members but is not allowed to access
the subscription information or delete the organization. The viewer Role
allows you to allow users to access the tenants within the organization
with only view permissions.
Inviting Team Members
Inviting users to your organization to work on the organization's
tenants is easy. Simply click Add Member in the Organization settings,
and give them an initial Role. The User will be sent an invitation
email. If a user is part of an organization, they can utilize their
login on all tenants they have authority to access.
PreviousZenML SaaS
NextStarter guide
Last updated 12 days ago
- >2-
more information.
Get the last run of a pipelineTo access the most recent run of a
pipeline, you can either use the last_run property or access it through
the runs list:
last_run = pipeline_model.last_run
If your most recent runs have failed, and you want to find the last run
that has succeeded, you can use the last_successful_run property
instead.
Get the latest run from a pipeline
Calling a pipeline executes it and then returns the response of the
freshly executed run.
run = training_pipeline()
The run that you get back is the model stored in the ZenML database at
the point of the method call. This means the pipeline run is still
initializing and no steps have been run. To get the latest state can get
a refreshed version from the client:
from zenml.client import Client
Client().get_pipeline_run(run.id) to get a refreshed version
Get a run via the client
If you already know the exact run that you want to fetch (e.g., from
looking at the dashboard), you can use the Client.get_pipeline_run()
method to fetch the run directly without having to query the pipeline
first:
from zenml.client import Client
pipeline_run =
Client().get_pipeline_run("first_pipeline-2023_06_20-16_20_13_274466")
Similar to pipelines, you can query runs by either ID, name, or name
prefix, and you can also discover runs through the Client or CLI via the
Client.list_pipeline_runs() or zenml pipeline runs list commands.
Run information
Each run has a collection of useful information which can help you
reproduce your runs. In the following, you can find a list of some of
the most useful pipeline run information, but there is much more
available. See the PipelineRunResponse definition for a comprehensive
list.
Status
The status of a pipeline run. There are five possible states:
initialized, failed, completed, running, and cached.
status = run.status
Configuration
- >-
ner(gamma=gamma, X_train=X_train, y_train=y_train)if __name__ ==
"__main__":
first_pipeline()
python run.py
...
Registered pipeline first_pipeline (version 2).
...
This will now create a single run for version 2 of the pipeline called
first_pipeline.
PreviousHyperparameter tuning
NextAccess secrets in a step
Last updated 15 days ago
- source_sentence: >-
How can I check which GCP Service Connectors can access a GCS bucket in my
ZenML deployment?
sentences:
- >-
dashboard.
Warning! Usage in remote orchestratorsThe current ZenML version has a
limitation in its base Docker image that requires a workaround for all
pipelines using Deepchecks with a remote orchestrator (e.g. Kubeflow ,
Vertex). The limitation being that the base Docker image needs to be
extended to include binaries that are required by opencv2, which is a
package that Deepchecks requires.
While these binaries might be available on most operating systems out of
the box (and therefore not a problem with the default local
orchestrator), we need to tell ZenML to add them to the containerization
step when running in remote settings. Here is how:
First, create a file called deepchecks-zenml.Dockerfile and place it on
the same level as your runner script (commonly called run.py). The
contents of the Dockerfile are as follows:
ARG ZENML_VERSION=0.20.0
FROM zenmldocker/zenml:${ZENML_VERSION} AS base
RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6 -y
Then, place the following snippet above your pipeline definition. Note
that the path of the dockerfile are relative to where the pipeline
definition file is. Read the containerization guide for more details:
import zenml
from zenml import pipeline
from zenml.config import DockerSettings
from pathlib import Path
import sys
docker_settings = DockerSettings(
dockerfile="deepchecks-zenml.Dockerfile",
build_options={
"buildargs": {
"ZENML_VERSION": f"{zenml.__version__}"
},
},
@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
...
From here on, you can continue to use the deepchecks integration as is
explained below.
The Deepchecks standard steps
ZenML wraps the Deepchecks functionality for tabular data in the form of
four standard steps:
DeepchecksDataIntegrityCheckStep: use it in your pipelines to run data
integrity tests on a single dataset
DeepchecksDataDriftCheckStep: use it in your pipelines to run data drift
tests on two datasets as input: target and reference.
- >2-
gs://zenml-core_cloudbuild ┃┃ │ gs://zenml-datasets ┃
┃ │
gs://zenml-internal-artifact-store ┃
┃ │
gs://zenml-kubeflow-artifact-store ┃
┃ │
gs://zenml-project-time-series-bucket ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 🌀 kubernetes-cluster │
zenml-test-cluster ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 🐳 docker-registry │
gcr.io/zenml-core ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Long-lived credentials (API keys, account keys)
This is the magic formula of authentication methods. When paired with
another ability, such as automatically generating short-lived API
tokens, or impersonating accounts or assuming roles, this is the ideal
authentication mechanism to use, particularly when using ZenML in
production and when sharing results with other members of your ZenML
team.
As a general best practice, but implemented particularly well for cloud
platforms, account passwords are never directly used as a credential
when authenticating to the cloud platform APIs. There is always a
process in place that exchanges the account/password credential for
another type of long-lived credential:
AWS uses the aws configure CLI command
GCP offers the gcloud auth application-default login CLI commands
Azure provides the az login CLI command
None of your original login information is stored on your local machine
or used to access workloads. Instead, an API key, account key or some
other form of intermediate credential is generated and stored on the
local host and used to authenticate to remote cloud service APIs.
- >2-
should pick the one that best fits your use case.If you already have one or more GCP Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the GCS bucket you want to use for your GCS Artifact Store by running e.g.:
zenml service-connector list-resources --resource-type gcs-bucket
Example Command Output
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR
TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼─────────────────────┼────────────────┼───────────────┼─────────────────────────────────────────────────┨
┃ 7f0c69ba-9424-40ae-8ea6-04f35c2eba9d │ gcp-user-account │ 🔵
gcp │ 📦 gcs-bucket │
gs://zenml-bucket-sl ┃
┃ │
│ │ │
gs://zenml-core.appspot.com ┃
┃ │
│ │ │
gs://zenml-core_cloudbuild ┃
┃ │
│ │ │
gs://zenml-datasets ┃
┃ │
│ │ │
gs://zenml-internal-artifact-store ┃
┃ │
│ │ │
gs://zenml-kubeflow-artifact-store ┃
┠──────────────────────────────────────┼─────────────────────┼────────────────┼───────────────┼─────────────────────────────────────────────────┨
┃ 2a0bec1b-9787-4bd7-8d4a-9a47b6f61643 │ gcs-zenml-bucket-sl │ 🔵
gcp │ 📦 gcs-bucket │
gs://zenml-bucket-sl ┃
- source_sentence: >-
Is it possible to update the local AWS CLI configuration with credentials
extracted from the AWS Service Connector?
sentences:
- >-
36a885: Pull complete
c9c0554c8e6a: Pull completebacdcd847a66: Pull complete
482033770844: Pull complete
Digest:
sha256:bf2cc3895e70dfa1ee1cd90bbfa599fa4cd8df837e27184bac1ce1cc239ecd3f
Status: Downloaded newer image for
715803424590.dkr.ecr.us-east-1.amazonaws.com/zenml-server:latest
715803424590.dkr.ecr.us-east-1.amazonaws.com/zenml-server:latest
It is also possible to update the local AWS CLI configuration with
credentials extracted from the AWS Service Connector:
zenml service-connector login aws-session-token --resource-type
aws-generic
Example Command Output
Configured local AWS SDK profile 'zenml-c0f8e857'.
The 'aws-session-token' AWS Service Connector connector was used to
successfully configure the local Generic AWS resource client/SDK.
A new profile is created in the local AWS CLI configuration holding the
credentials. It can be used to access AWS resources and services, e.g.:
aws --profile zenml-c0f8e857 s3 ls
Stack Components use
The S3 Artifact Store Stack Component can be connected to a remote AWS
S3 bucket through an AWS Service Connector.
The AWS Service Connector can also be used with any Orchestrator or
Model Deployer stack component flavor that relies on Kubernetes clusters
to manage workloads. This allows EKS Kubernetes container workloads to
be managed without the need to configure and maintain explicit AWS or
Kubernetes kubectl configuration contexts and credentials in the target
environment and in the Stack Component.
Similarly, Container Registry Stack Components can be connected to an
ECR Container Registry through an AWS Service Connector. This allows
container images to be built and published to ECR container registries
without the need to configure explicit AWS credentials in the target
environment or the Stack Component.
End-to-end examples
- >2-
┃┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ RESOURCE TYPES │ 🌀
kubernetes-cluster ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ RESOURCE NAME │
arn:aws:eks:us-east-1:715803424590:cluster/zenhacks-cluster ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ SECRET ID
│ ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ SESSION DURATION │
N/A ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ EXPIRES IN │
11h59m57s ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ OWNER │
default ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ WORKSPACE │
default ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ SHARED │
➖ ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ CREATED_AT │ 2023-06-16
10:17:46.931091 ┃
┠──────────────────┼─────────────────────────────────────────────────────────────────────┨
┃ UPDATED_AT │ 2023-06-16
10:17:46.931094 ┃
┗━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Configuration
- >-
e --authentication_secret. For example, you'd run:zenml secret create
argilla_secrets --api_key="<your_argilla_api_key>"
(Visit the Argilla documentation and interface to obtain your API key.)
Then register your annotator with ZenML:
zenml annotator register argilla --flavor argilla
--authentication_secret=argilla_secrets
When using a deployed instance of Argilla, the instance URL must be
specified without any trailing / at the end. If you are using a Hugging
Face Spaces instance and its visibility is set to private, you must also
set the extra_headers parameter which would include a Hugging Face
token. For example:
zenml annotator register argilla --flavor argilla
--authentication_secret=argilla_secrets
--instance_url="https://[your-owner-name]-[your_space_name].hf.space"
--extra_headers="{"Authorization": f"Bearer
{<your_hugging_face_token>}"}"
Finally, add all these components to a stack and set it as your active
stack. For example:
zenml stack copy default annotation
components are first registered
zenml stack update annotation -an <YOUR_ARGILLA_ANNOTATOR>
zenml stack set annotation
zenml stack describe
Now if you run a simple CLI command like zenml annotator dataset list
this should work without any errors. You're ready to use your annotator
in your ML workflow!
How do you use it?
ZenML supports access to your data and annotations via the zenml
annotator ... CLI command. We have also implemented an interface to some
of the common Argilla functionality via the ZenML SDK.
You can access information about the datasets you're using with the
zenml annotator dataset list. To work on annotation for a particular
dataset, you can run zenml annotator dataset annotate <dataset_name>.
What follows is an overview of some key components to the Argilla
integration and how it can be used.
Argilla Annotator Stack Component
model-index:
- name: zenml/finetuned-snowflake-arctic-embed-m
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 384
type: dim_384
metrics:
- type: cosine_accuracy@1
value: 0.29518072289156627
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.6204819277108434
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6927710843373494
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7891566265060241
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.29518072289156627
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.20682730923694775
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.13855421686746985
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.0789156626506024
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.29518072289156627
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.6204819277108434
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6927710843373494
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7891566265060241
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5524302146116403
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.4758486326257412
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4836255621339311
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 256
type: dim_256
metrics:
- type: cosine_accuracy@1
value: 0.28313253012048195
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.5963855421686747
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6807228915662651
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7771084337349398
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.28313253012048195
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.19879518072289157
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.136144578313253
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.07771084337349396
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.28313253012048195
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.5963855421686747
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6807228915662651
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7771084337349398
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5376005319054157
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.46014534327787354
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4690725321460052
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 128
type: dim_128
metrics:
- type: cosine_accuracy@1
value: 0.2710843373493976
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.5301204819277109
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.5963855421686747
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7409638554216867
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.2710843373493976
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.17670682730923692
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.11927710843373492
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.07409638554216866
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.2710843373493976
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.5301204819277109
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.5963855421686747
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7409638554216867
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.49831034220322973
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.4218158347676423
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.43128822737879013
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 64
type: dim_64
metrics:
- type: cosine_accuracy@1
value: 0.26506024096385544
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.4819277108433735
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.5662650602409639
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6566265060240963
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.26506024096385544
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1606425702811245
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.11325301204819276
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06566265060240963
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.26506024096385544
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.4819277108433735
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.5662650602409639
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6566265060240963
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.45413104746517285
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.38985465672212666
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4019553541721889
name: Cosine Map@100
zenml/finetuned-snowflake-arctic-embed-m
This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: Snowflake/snowflake-arctic-embed-m
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
- Language: en
- License: apache-2.0
Model Sources
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m")
sentences = [
'Is it possible to update the local AWS CLI configuration with credentials extracted from the AWS Service Connector?',
"36a885: Pull complete\n\nc9c0554c8e6a: Pull completebacdcd847a66: Pull complete\n\n482033770844: Pull complete\n\nDigest: sha256:bf2cc3895e70dfa1ee1cd90bbfa599fa4cd8df837e27184bac1ce1cc239ecd3f\n\nStatus: Downloaded newer image for 715803424590.dkr.ecr.us-east-1.amazonaws.com/zenml-server:latest\n\n715803424590.dkr.ecr.us-east-1.amazonaws.com/zenml-server:latest\n\nIt is also possible to update the local AWS CLI configuration with credentials extracted from the AWS Service Connector:\n\nzenml service-connector login aws-session-token --resource-type aws-generic\n\nExample Command Output\n\nConfigured local AWS SDK profile 'zenml-c0f8e857'.\n\nThe 'aws-session-token' AWS Service Connector connector was used to successfully configure the local Generic AWS resource client/SDK.\n\nA new profile is created in the local AWS CLI configuration holding the credentials. It can be used to access AWS resources and services, e.g.:\n\naws --profile zenml-c0f8e857 s3 ls\n\nStack Components use\n\nThe S3 Artifact Store Stack Component can be connected to a remote AWS S3 bucket through an AWS Service Connector.\n\nThe AWS Service Connector can also be used with any Orchestrator or Model Deployer stack component flavor that relies on Kubernetes clusters to manage workloads. This allows EKS Kubernetes container workloads to be managed without the need to configure and maintain explicit AWS or Kubernetes kubectl configuration contexts and credentials in the target environment and in the Stack Component.\n\nSimilarly, Container Registry Stack Components can be connected to an ECR Container Registry through an AWS Service Connector. This allows container images to be built and published to ECR container registries without the need to configure explicit AWS credentials in the target environment or the Stack Component.\n\nEnd-to-end examples",
'e --authentication_secret. For example, you\'d run:zenml secret create argilla_secrets --api_key="<your_argilla_api_key>"\n\n(Visit the Argilla documentation and interface to obtain your API key.)\n\nThen register your annotator with ZenML:\n\nzenml annotator register argilla --flavor argilla --authentication_secret=argilla_secrets\n\nWhen using a deployed instance of Argilla, the instance URL must be specified without any trailing / at the end. If you are using a Hugging Face Spaces instance and its visibility is set to private, you must also set the extra_headers parameter which would include a Hugging Face token. For example:\n\nzenml annotator register argilla --flavor argilla --authentication_secret=argilla_secrets --instance_url="https://[your-owner-name]-[your_space_name].hf.space" --extra_headers="{"Authorization": f"Bearer {<your_hugging_face_token>}"}"\n\nFinally, add all these components to a stack and set it as your active stack. For example:\n\nzenml stack copy default annotation\n\n# this must be done separately so that the other required stack components are first registered\n\nzenml stack update annotation -an <YOUR_ARGILLA_ANNOTATOR>\n\nzenml stack set annotation\n\n# optionally also\n\nzenml stack describe\n\nNow if you run a simple CLI command like zenml annotator dataset list this should work without any errors. You\'re ready to use your annotator in your ML workflow!\n\nHow do you use it?\n\nZenML supports access to your data and annotations via the zenml annotator ... CLI command. We have also implemented an interface to some of the common Argilla functionality via the ZenML SDK.\n\nYou can access information about the datasets you\'re using with the zenml annotator dataset list. To work on annotation for a particular dataset, you can run zenml annotator dataset annotate <dataset_name>. What follows is an overview of some key components to the Argilla integration and how it can be used.\n\nArgilla Annotator Stack Component',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
Evaluation
Metrics
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2952 |
| cosine_accuracy@3 |
0.6205 |
| cosine_accuracy@5 |
0.6928 |
| cosine_accuracy@10 |
0.7892 |
| cosine_precision@1 |
0.2952 |
| cosine_precision@3 |
0.2068 |
| cosine_precision@5 |
0.1386 |
| cosine_precision@10 |
0.0789 |
| cosine_recall@1 |
0.2952 |
| cosine_recall@3 |
0.6205 |
| cosine_recall@5 |
0.6928 |
| cosine_recall@10 |
0.7892 |
| cosine_ndcg@10 |
0.5524 |
| cosine_mrr@10 |
0.4758 |
| cosine_map@100 |
0.4836 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2831 |
| cosine_accuracy@3 |
0.5964 |
| cosine_accuracy@5 |
0.6807 |
| cosine_accuracy@10 |
0.7771 |
| cosine_precision@1 |
0.2831 |
| cosine_precision@3 |
0.1988 |
| cosine_precision@5 |
0.1361 |
| cosine_precision@10 |
0.0777 |
| cosine_recall@1 |
0.2831 |
| cosine_recall@3 |
0.5964 |
| cosine_recall@5 |
0.6807 |
| cosine_recall@10 |
0.7771 |
| cosine_ndcg@10 |
0.5376 |
| cosine_mrr@10 |
0.4601 |
| cosine_map@100 |
0.4691 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2711 |
| cosine_accuracy@3 |
0.5301 |
| cosine_accuracy@5 |
0.5964 |
| cosine_accuracy@10 |
0.741 |
| cosine_precision@1 |
0.2711 |
| cosine_precision@3 |
0.1767 |
| cosine_precision@5 |
0.1193 |
| cosine_precision@10 |
0.0741 |
| cosine_recall@1 |
0.2711 |
| cosine_recall@3 |
0.5301 |
| cosine_recall@5 |
0.5964 |
| cosine_recall@10 |
0.741 |
| cosine_ndcg@10 |
0.4983 |
| cosine_mrr@10 |
0.4218 |
| cosine_map@100 |
0.4313 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2651 |
| cosine_accuracy@3 |
0.4819 |
| cosine_accuracy@5 |
0.5663 |
| cosine_accuracy@10 |
0.6566 |
| cosine_precision@1 |
0.2651 |
| cosine_precision@3 |
0.1606 |
| cosine_precision@5 |
0.1133 |
| cosine_precision@10 |
0.0657 |
| cosine_recall@1 |
0.2651 |
| cosine_recall@3 |
0.4819 |
| cosine_recall@5 |
0.5663 |
| cosine_recall@10 |
0.6566 |
| cosine_ndcg@10 |
0.4541 |
| cosine_mrr@10 |
0.3899 |
| cosine_map@100 |
0.402 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 1,490 training samples
- Columns:
positive and anchor
- Approximate statistics based on the first 1000 samples:
|
positive |
anchor |
| type |
string |
string |
| details |
- min: 10 tokens
- mean: 21.19 tokens
- max: 64 tokens
|
- min: 21 tokens
- mean: 374.24 tokens
- max: 512 tokens
|
- Samples:
| positive |
anchor |
How do I use it to start the local Airflow server for ZenML? |
use it
To use the Airflow orchestrator, we need:The ZenML airflow integration installed. If you haven't done so, runCopyzenml integration install airflow
Docker installed and running.
The orchestrator registered and part of our active stack:
zenml orchestrator register <br>
--flavor=airflow <br>
--local=True # set this to False if using a remote Airflow deployment
# Register and activate a stack with the new orchestrator
zenml stack register -o ... --set
In the local case, we need to reinstall in a certain way for the local Airflow server:
pip install "apache-airflow-providers-docker<3.8.0" "apache-airflow==2.4.0" "pendulum<3.0.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.0/constraints-3.9.txt"
Please make sure to replace 3.9 with your Python (major) version in the constraints file URL given above.
Once that is installed, we can start the local Airflow server by running the following command in your terminal. See further below on an alternative way to set up the Airflow server manually since the zenml stack up command is deprecated.
zenml stack up
This command will start up an Airflow server on your local machine that's running in the same Python environment that you used to provision it. When it is finished, it will print a username and password which you can use to log in to the Airflow UI here.
As long as you didn't configure any custom value for the dag_output_dir attribute of your orchestrator, running a pipeline locally is as simple as calling:
python file_that_runs_a_zenml_pipeline.py
This call will produce a .zip file containing a representation of your ZenML pipeline to the Airflow DAGs directory. From there, the local Airflow server will load it and run your pipeline (It might take a few seconds until the pipeline shows up in the Airflow UI). |
What are the best practices for various authentication methods implemented by Service Connectors? |
Security best practices
Best practices concerning the various authentication methods implemented by Service Connectors.
Service Connector Types, especially those targeted at cloud providers, offer a plethora of authentication methods matching those supported by remote cloud platforms. While there is no single authentication standard that unifies this process, there are some patterns that are easily identifiable and can be used as guidelines when deciding which authentication method to use to configure a Service Connector.
This section explores some of those patterns and gives some advice regarding which authentication methods are best suited for your needs.
This section may require some general knowledge about authentication and authorization to be properly understood. We tried to keep it simple and limit ourselves to talking about high-level concepts, but some areas may get a bit too technical.
Username and password
The key takeaway is this: you should avoid using your primary account password as authentication credentials as much as possible. If there are alternative authentication methods that you can use or other types of credentials (e.g. session tokens, API keys, API tokens), you should always try to use those instead.
Ultimately, if you have no choice, be cognizant of the third parties you share your passwords with. If possible, they should never leave the premises of your local host or development environment.
This is the typical authentication method that uses a username or account name plus the associated password. While this is the de facto method used to log in with web consoles and local CLIs, this is the least secure of all authentication methods and never something you want to share with other members of your team or organization or use to authenticate automated workloads. |
How can I interact with a model deployer after deploying a model in ZenML? |
eline_step_name="huggingface_model_deployer_step",model_name="LLAMA-7B",
model_uri="s3://zenprojects/huggingface_model_deployer_step/output/884/huggingface",
revision="main",
task="text-classification",
region="us-east-1",
vendor="aws",
token="huggingface_token",
namespace="zenml-workloads",
endpoint_type="public",
print(f"Model server {service.config['model_name']} is deployed at {service.status['prediction_url']}")
How to Interact with a model deployer after deployment?
When a Model Deployer is part of the active ZenML Stack, it is also possible to interact with it from the CLI to list, start, stop, or delete the model servers that is managed:
$ zenml model-deployer models list
┏━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ STATUS │ UUID │ PIPELINE_NAME │ PIPELINE_STEP_NAME ┃
┠────────┼──────────────────────────────────────┼────────────────────────────────┼────────────────────────────┨
┃ ✅ │ 8cbe671b-9fce-4394-a051-68e001f92765 │ seldon_deployment_pipeline │ seldon_model_deployer_step ┃
┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
$ zenml model-deployer models describe 8cbe671b-9fce-4394-a051-68e001f92765
Properties of Served Model 8cbe671b-9fce-4394-a051-68e001f92765
┏━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ MODEL SERVICE PROPERTY │ VALUE ┃
┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┨
┃ MODEL_NAME │ mnist ┃
┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┨ |
- Loss:
MatryoshkaLoss with these parameters:{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
384,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1
],
"n_dims_per_step": -1
}
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy: epochper_device_train_batch_size: 32per_device_eval_batch_size: 16gradient_accumulation_steps: 16learning_rate: 2e-05num_train_epochs: 4lr_scheduler_type: cosinewarmup_ratio: 0.1bf16: Truetf32: Trueload_best_model_at_end: Trueoptim: adamw_torch_fusedbatch_sampler: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir: Falsedo_predict: Falseeval_strategy: epochprediction_loss_only: Trueper_device_train_batch_size: 32per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 16eval_accumulation_steps: Nonelearning_rate: 2e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 4max_steps: -1lr_scheduler_type: cosinelr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Truefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Truelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Trueremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Falsehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseeval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters: auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Nonedispatch_batches: Nonesplit_batches: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falsebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportional
Training Logs
| Epoch |
Step |
dim_128_cosine_map@100 |
dim_256_cosine_map@100 |
dim_384_cosine_map@100 |
dim_64_cosine_map@100 |
| 0.6667 |
1 |
0.3484 |
0.4016 |
0.4286 |
0.2904 |
| 2.0 |
3 |
0.4266 |
0.4650 |
0.4774 |
0.3927 |
| 2.6667 |
4 |
0.4313 |
0.4691 |
0.4836 |
0.402 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.10.14
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.3.1+cu121
- Accelerate: 0.31.0
- Datasets: 2.19.1
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MatryoshkaLoss
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
MultipleNegativesRankingLoss
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
