--- base_model: Snowflake/snowflake-arctic-embed-m datasets: [] language: - en library_name: sentence-transformers license: apache-2.0 metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:1490 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: - source_sentence: How can I deploy the ZenML server in different environments and manage pipelines with the new commands? sentences: - 'ed to update the way they are registered in ZenML.the updated ZenML server provides a new and improved collaborative experience. When connected to a ZenML server, you can now share your ZenML Stacks and Stack Components with other users. If you were previously using the ZenML Profiles or the ZenML server to share your ZenML Stacks, you should switch to the new ZenML server and Dashboard and update your existing workflows to reflect the new features. ZenML takes over the Metadata Store role ZenML can now run as a server that can be accessed via a REST API and also comes with a visual user interface (called the ZenML Dashboard). This server can be deployed in arbitrary environments (local, on-prem, via Docker, on AWS, GCP, Azure etc.) and supports user management, workspace scoping, and more. The release introduces a series of commands to facilitate managing the lifecycle of the ZenML server and to access the pipeline and pipeline run information: zenml connect / disconnect / down / up / logs / status can be used to configure your client to connect to a ZenML server, to start a local ZenML Dashboard or to deploy a ZenML server to a cloud environment. For more information on how to use these commands, see the ZenML deployment documentation. zenml pipeline list / runs / delete can be used to display information and about and manage your pipelines and pipeline runs. In ZenML 0.13.2 and earlier versions, information about pipelines and pipeline runs used to be stored in a separate stack component called the Metadata Store. Starting with 0.20.0, the role of the Metadata Store is now taken over by ZenML itself. This means that the Metadata Store is no longer a separate component in the ZenML architecture, but rather a part of the ZenML core, located wherever ZenML is deployed: locally on your machine or running remotely as a server.' - 'ntainer │ service-principal │ │ ┃┃ │ │ 🌀 kubernetes-cluster │ access-token │ │ ┃ ┃ │ │ 🐳 docker-registry │ │ │ ┃ ┠──────────────────────────────┼───────────────┼───────────────────────┼───────────────────┼───────┼────────┨ ┃ AWS Service Connector │ 🔶 aws │ 🔶 aws-generic │ implicit │ ✅ │ ✅ ┃ ┃ │ │ 📦 s3-bucket │ secret-key │ │ ┃ ┃ │ │ 🌀 kubernetes-cluster │ sts-token │ │ ┃ ┃ │ │ 🐳 docker-registry │ iam-role │ │ ┃ ┃ │ │ │ session-token │ │ ┃ ┃ │ │ │ federation-token │ │ ┃ ┠──────────────────────────────┼───────────────┼───────────────────────┼───────────────────┼───────┼────────┨ ┃ GCP Service Connector │ 🔵 gcp │ 🔵 gcp-generic │ implicit │ ✅ │ ✅ ┃ ┃ │ │ 📦 gcs-bucket │ user-account │ │ ┃ ┃ │ │ 🌀 kubernetes-cluster │ service-account │ │ ┃ ┃ │ │ 🐳 docker-registry │ oauth2-token │ │ ┃ ┃ │ │ │ impersonation │ │ ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛' - 'tional) Which Metadata to Extract for the ArtifactOptionally, you can override the extract_metadata() method to track custom metadata for all artifacts saved by your materializer. Anything you extract here will be displayed in the dashboard next to your artifacts. src.zenml.metadata.metadata_types that are displayed in a dedicated way in the dashboard. See src.zenml.metadata.metadata_types.MetadataType for more details. By default, this method will only extract the storage size of an artifact, but you can overwrite it to track anything you wish. E.g., the zenml.materializers.NumpyMaterializer overwrites this method to track the shape, dtype, and some statistical properties of each np.ndarray that it saves. If you would like to disable artifact metadata extraction altogether, you can set enable_artifact_metadata at either pipeline or step level via @pipeline(enable_artifact_metadata=False) or @step(enable_artifact_metadata=False). Skipping materialization Skipping materialization might have unintended consequences for downstream tasks that rely on materialized artifacts. Only skip materialization if there is no other way to do what you want to do. While materializers should in most cases be used to control how artifacts are returned and consumed from pipeline steps, you might sometimes need to have a completely unmaterialized artifact in a step, e.g., if you need to know the exact path to where your artifact is stored. An unmaterialized artifact is a zenml.materializers.UnmaterializedArtifact. Among others, it has a property uri that points to the unique path in the artifact store where the artifact is persisted. One can use an unmaterialized artifact by specifying UnmaterializedArtifact as the type in the step: from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact from zenml import step @step def my_step(my_artifact: UnmaterializedArtifact): # rather than pd.DataFrame pass Example' - source_sentence: How is the verification process different for multi-instance and single-instance Service Connectors? sentences: - 'Develop a Custom Annotator Learning how to develop a custom annotator. Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our general guide to writing custom component flavors in ZenML. This guide provides an essential understanding of ZenML''s component flavor concepts. Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks you have ready for use. Base abstraction in progress! We are actively working on the base abstraction for the annotators, which will be available soon. As a result, their extension is not possible at the moment. If you would like to use an annotator in your stack, please check the list of already available feature stores down below. PreviousProdigy NextModel Registries Last updated 15 days ago' - 'ld be accessible to larger audiences. TerminologyAs with any high-level abstraction, some terminology is needed to express the concepts and operations involved. In spite of the fact that Service Connectors cover such a large area of application as authentication and authorization for a variety of resources from a range of different vendors, we managed to keep this abstraction clean and simple. In the following expandable sections, you''ll learn more about Service Connector Types, Resource Types, Resource Names, and Service Connectors. This term is used to represent and identify a particular Service Connector implementation and answer questions about its capabilities such as "what types of resources does this Service Connector give me access to", "what authentication methods does it support" and "what credentials and other information do I need to configure for it". This is analogous to the role Flavors play for Stack Components in that the Service Connector Type acts as the template from which one or more Service Connectors are created. For example, the built-in AWS Service Connector Type shipped with ZenML supports a rich variety of authentication methods and provides access to AWS resources such as S3 buckets, EKS clusters and ECR registries. The zenml service-connector list-types and zenml service-connector describe-type CLI commands can be used to explore the Service Connector Types available with your ZenML deployment. Extensive documentation is included covering supported authentication methods and Resource Types. The following are just some examples: zenml service-connector list-types Example Command Output ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓ ┃ NAME │ TYPE │ RESOURCE TYPES │ AUTH METHODS │ LOCAL │ REMOTE ┃ ┠──────────────────────────────┼───────────────┼───────────────────────┼───────────────────┼───────┼────────┨' - 'ing resources: ┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓┃ RESOURCE TYPE │ RESOURCE NAMES ┃ ┠───────────────┼────────────────┨ ┃ 📦 s3-bucket │ s3://zenfiles ┃ ┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛ The following might help understand the difference between scopes: the difference between a multi-instance and a multi-type Service Connector is that the Resource Type scope is locked to a particular value during configuration for the multi-instance Service Connector similarly, the difference between a multi-instance and a multi-type Service Connector is that the Resource Name (Resource ID) scope is locked to a particular value during configuration for the single-instance Service Connector Service Connector Verification When registering Service Connectors, the authentication configuration and credentials are automatically verified to ensure that they can indeed be used to gain access to the target resources: for multi-type Service Connectors, this verification means checking that the configured credentials can be used to authenticate successfully to the remote service, as well as listing all resources that the credentials have permission to access for each Resource Type supported by the Service Connector Type. for multi-instance Service Connectors, this verification step means listing all resources that the credentials have permission to access in addition to validating that the credentials can be used to authenticate to the target service or platform. for single-instance Service Connectors, the verification step simply checks that the configured credentials have permission to access the target resource. The verification can also be performed later on an already registered Service Connector. Furthermore, for multi-type and multi-instance Service Connectors, the verification operation can be scoped to a Resource Type and a Resource Name. The following shows how a multi-type, a multi-instance and a single-instance Service Connector can be verified with multiple scopes after registration.' - source_sentence: How long did it take to generate 1800+ questions from documentation chunks using the local model on a GPU-enabled machine? sentences: - 'ns, especially using the basic setup we have here.To give you an indication of how long this process takes, generating 1800+ questions from an equivalent number of documentation chunks took a little over 45 minutes using the local model on a GPU-enabled machine with Ollama. You can view the generated dataset on the Hugging Face Hub here. This dataset contains the original document chunks, the generated questions, and the URL reference for the original document. Once we have the generated questions, we can then pass them to the retrieval component and check the results. For convenience we load the data from the Hugging Face Hub and then pass it to the retrieval component for evaluation. We shuffle the data and select a subset of it to speed up the evaluation process, but for a more thorough evaluation you could use the entire dataset. (The best practice of keeping a separate set of data for evaluation purposes is also recommended here, though we''re not doing that in this example.) @step def retrieval_evaluation_full( sample_size: int = 50, ) -> Annotated[float, "full_failure_rate_retrieval"]: dataset = load_dataset("zenml/rag_qa_embedding_questions", split="train") sampled_dataset = dataset.shuffle(seed=42).select(range(sample_size)) total_tests = len(sampled_dataset) failures = 0 for item in sampled_dataset: generated_questions = item["generated_questions"] question = generated_questions[ ] # Assuming only one question per item url_ending = item["filename"].split("/")[ 1 ] # Extract the URL ending from the filename _, _, urls = query_similar_docs(question, url_ending) if all(url_ending not in url for url in urls): logging.error( f"Failed for question: {question}. Expected URL ending: {url_ending}. Got: {urls}" failures += 1 logging.info(f"Total tests: {total_tests}. Failures: {failures}") failure_rate = (failures / total_tests) * 100 return round(failure_rate, 2)' - '😸Set up a project repository Setting your team up for success with a project repository. ZenML code typically lives in a git repository. Setting this repository up correctly can make a huge impact on collaboration and getting the maximum out of your ZenML deployment. This section walks users through some of the options available to create a project repository with ZenML. PreviousFinetuning LLMs with ZenML NextConnect your git repository Last updated 15 days ago' - 'GCP Service Connector Configuring GCP Service Connectors to connect ZenML to GCP resources such as GCS buckets, GKE Kubernetes clusters, and GCR container registries. The ZenML GCP Service Connector facilitates the authentication and access to managed GCP services and resources. These encompass a range of resources, including GCS buckets, GCR container repositories, and GKE clusters. The connector provides support for various authentication methods, including GCP user accounts, service accounts, short-lived OAuth 2.0 tokens, and implicit authentication. To ensure heightened security measures, this connector always issues short-lived OAuth 2.0 tokens to clients instead of long-lived credentials unless explicitly configured to do otherwise. Furthermore, it includes automatic configuration and detection of credentials locally configured through the GCP CLI. This connector serves as a general means of accessing any GCP service by issuing OAuth 2.0 credential objects to clients. Additionally, the connector can handle specialized authentication for GCS, Docker, and Kubernetes Python clients. It also allows for the configuration of local Docker and Kubernetes CLIs. $ zenml service-connector list-types --type gcp ┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓ ┃ NAME │ TYPE │ RESOURCE TYPES │ AUTH METHODS │ LOCAL │ REMOTE ┃ ┠───────────────────────┼────────┼───────────────────────┼──────────────────┼───────┼────────┨ ┃ GCP Service Connector │ 🔵 gcp │ 🔵 gcp-generic │ implicit │ ✅ │ ✅ ┃ ┃ │ │ 📦 gcs-bucket │ user-account │ │ ┃ ┃ │ │ 🌀 kubernetes-cluster │ service-account │ │ ┃ ┃ │ │ 🐳 docker-registry │ external-account │ │ ┃ ┃ │ │ │ oauth2-token │ │ ┃' - source_sentence: How can I load and render reports in a Jupyter notebook using ZenML? sentences: - '❗Alerters Sending automated alerts to chat services. Alerters allow you to send messages to chat services (like Slack, Discord, Mattermost, etc.) from within your pipelines. This is useful to immediately get notified when failures happen, for general monitoring/reporting, and also for building human-in-the-loop ML. Alerter Flavors Currently, the SlackAlerter and DiscordAlerter are the available alerter integrations. However, it is straightforward to extend ZenML and build an alerter for other chat services. Alerter Flavor Integration Notes Slack slack slack Interacts with a Slack channel Discord discord discord Interacts with a Discord channel Custom Implementation custom Extend the alerter abstraction and provide your own implementation If you would like to see the available flavors of alerters in your terminal, you can use the following command: zenml alerter flavor list How to use Alerters with ZenML Each alerter integration comes with specific standard steps that you can use out of the box. However, you first need to register an alerter component in your terminal: zenml alerter register ... Then you can add it to your stack using zenml stack register ... -al Afterward, you can import the alerter standard steps provided by the respective integration and directly use them in your pipelines. PreviousDevelop a Custom Step Operator NextDiscord Alerter Last updated 15 days ago' - 'ry_similar_docs( question: str, url_ending: str,use_reranking: bool = False, returned_sample_size: int = 5, ) -> Tuple[str, str, List[str]]: """Query similar documents for a given question and URL ending.""" embedded_question = get_embeddings(question) db_conn = get_db_conn() num_docs = 20 if use_reranking else returned_sample_size # get (content, url) tuples for the top n similar documents top_similar_docs = get_topn_similar_docs( embedded_question, db_conn, n=num_docs, include_metadata=True if use_reranking: reranked_docs_and_urls = rerank_documents(question, top_similar_docs)[ :returned_sample_size urls = [doc[1] for doc in reranked_docs_and_urls] else: urls = [doc[1] for doc in top_similar_docs] # Unpacking URLs return (question, url_ending, urls) We get the embeddings for the question being passed into the function and connect to our PostgreSQL database. If we''re using reranking, we get the top 20 documents similar to our query and rerank them using the rerank_documents helper function. We then extract the URLs from the reranked documents and return them. Note that we only return 5 URLs, but in the case of reranking we get a larger number of documents and URLs back from the database to pass to our reranker, but in the end we always choose the top five reranked documents to return. Now that we''ve added reranking to our pipeline, we can evaluate the performance of our reranker and see how it affects the quality of the retrieved documents. Code Example To explore the full code, visit the Complete Guide repository and for this section, particularly the eval_retrieval.py file. PreviousUnderstanding reranking NextEvaluating reranking performance Last updated 1 month ago' - 'n the respective artifact in the pipeline run DAG.Alternatively, if you are running inside a Jupyter notebook, you can load and render the reports using the artifact.visualize() method, e.g.: from zenml.client import Client def visualize_results(pipeline_name: str, step_name: str) -> None: pipeline = Client().get_pipeline(pipeline=pipeline_name) evidently_step = pipeline.last_run.steps[step_name] evidently_step.visualize() if __name__ == "__main__": visualize_results("text_data_report_pipeline", "text_report") visualize_results("text_data_test_pipeline", "text_test") PreviousDeepchecks NextWhylogs Last updated 19 days ago' - source_sentence: How do you deploy the Comet Experiment Tracker flavor provided by ZenML integration? sentences: - 'Comet Logging and visualizing experiments with Comet. The Comet Experiment Tracker is an Experiment Tracker flavor provided with the Comet ZenML integration that uses the Comet experiment tracking platform to log and visualize information from your pipeline steps (e.g., models, parameters, metrics). When would you want to use it? Comet is a popular platform that you would normally use in the iterative ML experimentation phase to track and visualize experiment results. That doesn''t mean that it cannot be repurposed to track and visualize the results produced by your automated pipeline runs, as you make the transition towards a more production-oriented workflow. You should use the Comet Experiment Tracker: if you have already been using Comet to track experiment results for your project and would like to continue doing so as you are incorporating MLOps workflows and best practices in your project through ZenML. if you are looking for a more visually interactive way of navigating the results produced from your ZenML pipeline runs (e.g., models, metrics, datasets) if you would like to connect ZenML to Comet to share the artifacts and metrics logged by your pipelines with your team, organization, or external stakeholders You should consider one of the other Experiment Tracker flavors if you have never worked with Comet before and would rather use another experiment tracking tool that you are more familiar with. How do you deploy it? The Comet Experiment Tracker flavor is provided by the Comet ZenML integration. You need to install it on your local machine to be able to register a Comet Experiment Tracker and add it to your stack: zenml integration install comet -y The Comet Experiment Tracker needs to be configured with the credentials required to connect to the Comet platform using one of the available authentication methods. Authentication Methods You need to configure the following credentials for authentication to the Comet platform:' - 'guration set up by the GCP CLI on your local host.The following is an example of lifting GCP user credentials granting access to the same set of GCP resources and services that the local GCP CLI is allowed to access. The GCP CLI should already be configured with valid credentials (i.e. by running gcloud auth application-default login). In this case, the GCP user account authentication method is automatically detected: zenml service-connector register gcp-auto --type gcp --auto-configure Example Command Output Successfully registered service connector `gcp-auto` with access to the following resources: ┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ RESOURCE TYPE │ RESOURCE NAMES ┃ ┠───────────────────────┼─────────────────────────────────────────────────┨ ┃ 🔵 gcp-generic │ zenml-core ┃ ┠───────────────────────┼─────────────────────────────────────────────────┨ ┃ 📦 gcs-bucket │ gs://zenml-bucket-sl ┃ ┃ │ gs://zenml-core.appspot.com ┃ ┃ │ gs://zenml-core_cloudbuild ┃ ┃ │ gs://zenml-datasets ┃ ┃ │ gs://zenml-internal-artifact-store ┃ ┃ │ gs://zenml-kubeflow-artifact-store ┃ ┃ │ gs://zenml-project-time-series-bucket ┃ ┠───────────────────────┼─────────────────────────────────────────────────┨ ┃ 🌀 kubernetes-cluster │ zenml-test-cluster ┃ ┠───────────────────────┼─────────────────────────────────────────────────┨ ┃ 🐳 docker-registry │ gcr.io/zenml-core ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ zenml service-connector describe gcp-auto Example Command Output' - 'er Image Builder stack component, or the Vertex AIOrchestrator and Step Operator. It should be accompanied by a matching set of GCP permissions that allow access to the set of remote resources required by the client and Stack Component. The resource name represents the GCP project that the connector is authorized to access. 📦 GCP GCS bucket (resource type: gcs-bucket) Authentication methods: implicit, user-account, service-account, oauth2-token, impersonation Supports resource instances: True Authentication methods: 🔒 implicit 🔒 user-account 🔒 service-account 🔒 oauth2-token 🔒 impersonation Allows Stack Components to connect to GCS buckets. When used by Stack Components, they are provided a pre-configured GCS Python client instance. The configured credentials must have at least the following GCP permissions associated with the GCS buckets that it can access: storage.buckets.list storage.buckets.get storage.objects.create storage.objects.delete storage.objects.get storage.objects.list storage.objects.update For example, the GCP Storage Admin role includes all of the required permissions, but it also includes additional permissions that are not required by the connector. If set, the resource name must identify a GCS bucket using one of the following formats: GCS bucket URI: gs://{bucket-name} GCS bucket name: {bucket-name} [...] ──────────────────────────────────────────────────────────────────────────────── Please select a resource type or leave it empty to create a connector that can be used to access any of the supported resource types (gcp-generic, gcs-bucket, kubernetes-cluster, docker-registry). []: gcs-bucket Would you like to attempt auto-configuration to extract the authentication configuration from your local environment ? [y/N]: y Service connector auto-configured successfully with the following configuration: Service connector ''gcp-interactive'' of type ''gcp'' is ''private''. ''gcp-interactive'' gcp Service Connector Details ┏━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━┓' model-index: - name: zenml/finetuned-snowflake-arctic-embed-m results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 384 type: dim_384 metrics: - type: cosine_accuracy@1 value: 0.28313253012048195 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.572289156626506 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.6807228915662651 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8012048192771084 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.28313253012048195 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.19076305220883527 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.13614457831325297 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08012048192771083 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.28313253012048195 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.572289156626506 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.6807228915662651 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8012048192771084 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.5407472416922913 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.45774765729585015 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.46523155503040436 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.29518072289156627 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.6024096385542169 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.6807228915662651 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.7951807228915663 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.29518072289156627 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2008032128514056 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.13614457831325297 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.0795180722891566 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.29518072289156627 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.6024096385542169 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.6807228915662651 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.7951807228915663 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.5458001676537428 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.46605230445591894 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.4728738562350596 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.2469879518072289 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.5843373493975904 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.6265060240963856 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.7409638554216867 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.2469879518072289 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.19477911646586343 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.12530120481927706 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.07409638554216866 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.2469879518072289 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.5843373493975904 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.6265060240963856 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.7409638554216867 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.4994853551416632 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.421791929623255 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.4323899020969096 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.23493975903614459 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.5 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.5783132530120482 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.6927710843373494 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.23493975903614459 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.16666666666666666 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.11566265060240961 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.06927710843373491 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.23493975903614459 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.5 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.5783132530120482 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.6927710843373494 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.4607453075643617 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.38742589405240024 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.3969546791348258 name: Cosine Map@100 --- # zenml/finetuned-snowflake-arctic-embed-m This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m") # Run inference sentences = [ 'How do you deploy the Comet Experiment Tracker flavor provided by ZenML integration?', "Comet\n\nLogging and visualizing experiments with Comet.\n\nThe Comet Experiment Tracker is an Experiment Tracker flavor provided with the Comet ZenML integration that uses the Comet experiment tracking platform to log and visualize information from your pipeline steps (e.g., models, parameters, metrics).\n\nWhen would you want to use it?\n\nComet is a popular platform that you would normally use in the iterative ML experimentation phase to track and visualize experiment results. That doesn't mean that it cannot be repurposed to track and visualize the results produced by your automated pipeline runs, as you make the transition towards a more production-oriented workflow.\n\nYou should use the Comet Experiment Tracker:\n\nif you have already been using Comet to track experiment results for your project and would like to continue doing so as you are incorporating MLOps workflows and best practices in your project through ZenML.\n\nif you are looking for a more visually interactive way of navigating the results produced from your ZenML pipeline runs (e.g., models, metrics, datasets)\n\nif you would like to connect ZenML to Comet to share the artifacts and metrics logged by your pipelines with your team, organization, or external stakeholders\n\nYou should consider one of the other Experiment Tracker flavors if you have never worked with Comet before and would rather use another experiment tracking tool that you are more familiar with.\n\nHow do you deploy it?\n\nThe Comet Experiment Tracker flavor is provided by the Comet ZenML integration. You need to install it on your local machine to be able to register a Comet Experiment Tracker and add it to your stack:\n\nzenml integration install comet -y\n\nThe Comet Experiment Tracker needs to be configured with the credentials required to connect to the Comet platform using one of the available authentication methods.\n\nAuthentication Methods\n\nYou need to configure the following credentials for authentication to the Comet platform:", "er Image Builder stack component, or the Vertex AIOrchestrator and Step Operator. It should be accompanied by a matching set of\n\nGCP permissions that allow access to the set of remote resources required by the\n\nclient and Stack Component.\n\nThe resource name represents the GCP project that the connector is authorized to\n\naccess.\n\n📦 GCP GCS bucket (resource type: gcs-bucket)\n\nAuthentication methods: implicit, user-account, service-account, oauth2-token,\n\nimpersonation\n\nSupports resource instances: True\n\nAuthentication methods:\n\n🔒 implicit\n\n🔒 user-account\n\n🔒 service-account\n\n🔒 oauth2-token\n\n🔒 impersonation\n\nAllows Stack Components to connect to GCS buckets. When used by Stack\n\nComponents, they are provided a pre-configured GCS Python client instance.\n\nThe configured credentials must have at least the following GCP permissions\n\nassociated with the GCS buckets that it can access:\n\nstorage.buckets.list\n\nstorage.buckets.get\n\nstorage.objects.create\n\nstorage.objects.delete\n\nstorage.objects.get\n\nstorage.objects.list\n\nstorage.objects.update\n\nFor example, the GCP Storage Admin role includes all of the required\n\npermissions, but it also includes additional permissions that are not required\n\nby the connector.\n\nIf set, the resource name must identify a GCS bucket using one of the following\n\nformats:\n\nGCS bucket URI: gs://{bucket-name}\n\nGCS bucket name: {bucket-name}\n\n[...]\n\n────────────────────────────────────────────────────────────────────────────────\n\nPlease select a resource type or leave it empty to create a connector that can be used to access any of the supported resource types (gcp-generic, gcs-bucket, kubernetes-cluster, docker-registry). []: gcs-bucket\n\nWould you like to attempt auto-configuration to extract the authentication configuration from your local environment ? [y/N]: y\n\nService connector auto-configured successfully with the following configuration:\n\nService connector 'gcp-interactive' of type 'gcp' is 'private'.\n\n'gcp-interactive' gcp Service\n\nConnector Details\n\n┏━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━┓", ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_384` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.2831 | | cosine_accuracy@3 | 0.5723 | | cosine_accuracy@5 | 0.6807 | | cosine_accuracy@10 | 0.8012 | | cosine_precision@1 | 0.2831 | | cosine_precision@3 | 0.1908 | | cosine_precision@5 | 0.1361 | | cosine_precision@10 | 0.0801 | | cosine_recall@1 | 0.2831 | | cosine_recall@3 | 0.5723 | | cosine_recall@5 | 0.6807 | | cosine_recall@10 | 0.8012 | | cosine_ndcg@10 | 0.5407 | | cosine_mrr@10 | 0.4577 | | **cosine_map@100** | **0.4652** | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.2952 | | cosine_accuracy@3 | 0.6024 | | cosine_accuracy@5 | 0.6807 | | cosine_accuracy@10 | 0.7952 | | cosine_precision@1 | 0.2952 | | cosine_precision@3 | 0.2008 | | cosine_precision@5 | 0.1361 | | cosine_precision@10 | 0.0795 | | cosine_recall@1 | 0.2952 | | cosine_recall@3 | 0.6024 | | cosine_recall@5 | 0.6807 | | cosine_recall@10 | 0.7952 | | cosine_ndcg@10 | 0.5458 | | cosine_mrr@10 | 0.4661 | | **cosine_map@100** | **0.4729** | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.247 | | cosine_accuracy@3 | 0.5843 | | cosine_accuracy@5 | 0.6265 | | cosine_accuracy@10 | 0.741 | | cosine_precision@1 | 0.247 | | cosine_precision@3 | 0.1948 | | cosine_precision@5 | 0.1253 | | cosine_precision@10 | 0.0741 | | cosine_recall@1 | 0.247 | | cosine_recall@3 | 0.5843 | | cosine_recall@5 | 0.6265 | | cosine_recall@10 | 0.741 | | cosine_ndcg@10 | 0.4995 | | cosine_mrr@10 | 0.4218 | | **cosine_map@100** | **0.4324** | #### Information Retrieval * Dataset: `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:----------| | cosine_accuracy@1 | 0.2349 | | cosine_accuracy@3 | 0.5 | | cosine_accuracy@5 | 0.5783 | | cosine_accuracy@10 | 0.6928 | | cosine_precision@1 | 0.2349 | | cosine_precision@3 | 0.1667 | | cosine_precision@5 | 0.1157 | | cosine_precision@10 | 0.0693 | | cosine_recall@1 | 0.2349 | | cosine_recall@3 | 0.5 | | cosine_recall@5 | 0.5783 | | cosine_recall@10 | 0.6928 | | cosine_ndcg@10 | 0.4607 | | cosine_mrr@10 | 0.3874 | | **cosine_map@100** | **0.397** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 1,490 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | positive | anchor | |:-----------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | How can I organize key-value pairs into cards in the ZenML dashboard? | Group metadata

Learn how to group key-value pairs in the dashboard.

When logging metadata passing a dictionary of dictionaries in the metadata parameter will group the metadata into cards in the ZenML dashboard. This feature helps organize metadata into logical sections, making it easier to visualize and understand.

Here's an example of grouping metadata into cards:

from zenml.metadata.metadata_types import StorageSize

log_artifact_metadata(

metadata={

"model_metrics": {

"accuracy": 0.95,

"precision": 0.92,

"recall": 0.90

},

"data_details": {

"dataset_size": StorageSize(1500000),

"feature_columns": ["age", "income", "score"]

In the ZenML dashboard, "model_metrics" and "data_details" would appear as separate cards, each containing their respective key-value pairs.

PreviousAttach metadata to steps

NextSpecial Metadata Types

Last updated 19 days ago | | How can I configure an S3 Artifact Store using implicit AWS authentication on my local machine? | other remote stack components also running in AWS.This method uses the implicit AWS authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure an S3 Artifact Store. You don't need to supply credentials explicitly when you register the S3 Artifact Store, as it leverages the local credentials and configuration that the AWS CLI stores on your local machine. However, you will need to install and set up the AWS CLI on your machine as a prerequisite, as covered in the AWS CLI documentation, before you register the S3 Artifact Store.

Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permission to access the filesystem.

The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to work. If these components are not running on your machine, they do not have access to the local AWS CLI configuration and will encounter authentication failures while trying to access the S3 Artifact Store:

Orchestrators need to access the Artifact Store to manage pipeline artifacts

Step Operators need to access the Artifact Store to manage step-level artifacts

Model Deployers need to access the Artifact Store to load served models

To enable these use-cases, it is recommended to use an AWS Service Connector to link your S3 Artifact Store to the remote S3 bucket.

To set up the S3 Artifact Store to authenticate to AWS and access an S3 bucket, it is recommended to leverage the many features provided by the AWS Service Connector such as auto-configuration, best security practices regarding long-lived credentials and fine-grained access control and reusing the same credentials across multiple stack components. | | How does the `sync_new_data_to_label_studio` step ensure that new annotations remain in sync with the ZenML artifact store? | th the appropriate label config with Label Studio.get_labeled_data step - This step will get all labeled data available for a particular dataset. Note that these are output in a Label Studio annotation format, which will subsequently be converted into a format appropriate for your specific use case.

sync_new_data_to_label_studio step - This step is for ensuring that ZenML is handling the annotations and that the files being used are stored and synced with the ZenML artifact store. This is an important step as part of a continuous annotation workflow since you want all the subsequent steps of your workflow to remain in sync with whatever new annotations are being made or have been created.

Helper Functions

Label Studio requires the use of what it calls 'label config' when you are creating/registering your dataset. These are strings containing HTML-like syntax that allow you to define a custom interface for your annotation. ZenML provides three helper functions that will construct these label config strings in the case of object detection, image classification, and OCR. See the integrations.label_studio.label_config_generators module for those three functions.

PreviousArgilla

NextPigeon

Last updated 15 days ago | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 384, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: True - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_384_cosine_map@100 | dim_64_cosine_map@100 | |:----------:|:-----:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| | 0.6667 | 1 | 0.3939 | 0.3945 | 0.4088 | 0.2782 | | 2.0 | 3 | 0.4298 | 0.4580 | 0.4630 | 0.3868 | | **2.6667** | **4** | **0.4324** | **0.4729** | **0.4652** | **0.397** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.14 - Sentence Transformers: 3.0.1 - Transformers: 4.41.2 - PyTorch: 2.3.1+cu121 - Accelerate: 0.31.0 - Datasets: 2.19.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```