diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -31,1070 +31,1140 @@ tags: - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: -- source_sentence: How does ZenML Cloud manage access to customer secrets while ensuring - security? +- source_sentence: Where is the global configuration directory located in ZenML's + default setup? sentences: - - 'redentials are always stored on the customer side.Even though they are stored - customer side, access to ZenML secrets is fully managed by the ZenML Cloud. The - ZenML Cloud is also allowed to use some of those credentials to connect directly - to customer infrastructure services to implement control plane features such as - artifact visualization or triggering pipelines. This implies that the secret values - are allowed to leave the customer environment to allow their access to be managed - centrally by the ZenML Cloud and to enforce access control policies, but the ZenML - users and pipelines never have direct access to the secret store. + - '''default'' ... - All access to customer secrets is, of course, regulated through authentication - and RBAC, so that only authorized users can access the secrets. This deployment - scenario is meant for customers who want to use the ZenML Cloud but want to keep - their secrets on their own infrastructure. + Creating default user ''default'' ...Creating default stack for user ''default'' + in workspace default... - Scenario 3: Fully On-prem + Active workspace not set. Setting it to the default. - In this scenario, all services, data, and secrets are deployed on the customer - cloud. This is the opposite of Scenario 1, and is meant for customers who require - completely airgapped deployments, for the tightest security standards. Reach out - to us if you want to set this up. + The active stack is not set. Setting the active stack to the default workspace + stack. - Are you interested in the ZenML Cloud? Sign up and get access to Scenario 1. with - a free 14 day trial now! + Using the default store for the global config. - PreviousZenML Pro + Unable to find ZenML repository in your current working directory (/tmp/folder) + or any parent directories. If you want to use an existing repository which is + in a different location, set the environment variable ''ZENML_REPOSITORY_PATH''. + If you want to create a new repository, run zenml init. - NextZenML SaaS + Running without an active repository root. - Last updated 15 days ago' - - 'Delete an artifact + Using the default local database. - Learn how to delete artifacts. + Running with active workspace: ''default'' (global) - There is currently no way to delete an artifact directly, because it may lead - to a broken state of the ZenML database (dangling references to pipeline runs - that produce artifacts). + ┏━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓ - However, it is possible to delete artifacts that are no longer referenced by any - pipeline runs: + ┃ ACTIVE │ STACK NAME │ SHARED │ OWNER │ ARTIFACT_STORE │ ORCHESTRATOR ┃ - zenml artifact prune + ┠────────┼────────────┼────────┼─────────┼────────────────┼──────────────┨ - By default, this method deletes artifacts physically from the underlying artifact - store AND also the entry in the database. You can control this behavior by using - the --only-artifact and --only-metadata flags. + ┃ 👉 │ default │ ❌ │ default │ default │ default ┃ - You might find that some artifacts throw errors when you try to prune them, likely - because they were stored locally and no longer exist. If you wish to continue - pruning and to ignore these errors, please add the --ignore-errors flag. Warning - messages will still be output to the terminal during this process. + ┗━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛ - PreviousReturn multiple outputs from a step + The following is an example of the layout of the global config directory immediately + after initialization: - NextOrganize data with tags + /home/stefan/.config/zenml <- Global Config Directory - Last updated 7 days ago' - - 'Linking model binaries/data to a Model + ├── config.yaml <- Global Configuration Settings - Artifacts generated during pipeline runs can be linked to models in ZenML. This - connecting of artifacts provides lineage tracking and transparency into what data - and models are used during training, evaluation, and inference. + └── local_stores <- Every Stack component that stores information - There are a few ways to link artifacts: + | locally will have its own subdirectory here. - Configuring the Model at a pipeline level + ├── a1a0d3d0-d552-4a80-be09-67e5e29be8ee <- e.g. Local Store path for the - The easiest way is to configure the model parameter on the @pipeline decorator - or @step decorator: + | `default` local Artifact Store - from zenml import Model, pipeline + └── default_zen_store - model = Model( + └── zenml.db <- SQLite database where ZenML data (stacks, - name="my_model", + components, etc) are stored by default. - version="1.0.0" + As shown above, the global config directory stores the following information:' + - 'Reranking for better retrieval - @pipeline(model=model) + Add reranking to your RAG inference for better retrieval performance. - def my_pipeline(): + Rerankers are a crucial component of retrieval systems that use LLMs. They help + improve the quality of the retrieved documents by reordering them based on additional + features or scores. In this section, we''ll explore how to add a reranker to your + RAG inference pipeline in ZenML. - ... + In previous sections, we set up the overall workflow, from data ingestion and + preprocessing to embeddings generation and retrieval. We then set up some basic + evaluation metrics to assess the performance of our retrieval system. A reranker + is a way to squeeze a bit of extra performance out of the system by reordering + the retrieved documents based on additional features or scores. - This will automatically link all artifacts from this pipeline run to the specified - model configuration. + As you can see, reranking is an optional addition we make to what we''ve already + set up. It''s not strictly necessary, but it can help improve the relevance and + quality of the retrieved documents, which in turn can lead to better responses + from the LLM. Let''s dive in! - Controlling artifact types and linkage + PreviousEvaluation in practice - A ZenML model supports linking three types of artifacts: + NextUnderstanding reranking - Data artifacts: These are the default artifacts. If nothing is specified, all - artifacts are grouped under this category. + Last updated 1 month ago' + - '─────────────────────────────────────────────────┨┃ RESOURCE TYPES │ 🔵 gcp-generic, + 📦 gcs-bucket, 🌀 kubernetes-cluster, 🐳 docker-registry ┃ - Model artifacts: If there is a physical model artifact like a .pkl file or a model - neural network weights file, it should be grouped in this category. + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - Deployment artifacts: These artifacts are to do with artifacts related to the - endpoints and deployments of the models. + ┃ RESOURCE NAME │ ┃ - You can also explicitly specify the linkage on a per-artifact basis by passing - a special configuration to the Annotated output: + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - from zenml import step, ArtifactConfig + ┃ SECRET ID │ 4694de65-997b-4929-8831-b49d5e067b97 ┃ - from typing import Tuple + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - from typing_extensions import Annotated + ┃ SESSION DURATION │ N/A ┃ - import pandas as pd + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - @step + ┃ EXPIRES IN │ 59m46s ┃ - def svc_trainer( + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - X_train: pd.DataFrame, + ┃ OWNER │ default ┃ - y_train: pd.Series, + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - gamma: float = 0.001, + ┃ WORKSPACE │ default ┃ + + + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ + + ┃ SHARED │ ➖ ┃ - ) -> Tuple[ + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - # This third argument marks this as a Model Artifact + ┃ CREATED_AT │ 2023-05-19 09:04:33.557126 ┃ - Annotated[ClassifierMixin, ArtifactConfig("trained_model", is_model_artifact=True)], + ┠──────────────────┼──────────────────────────────────────────────────────────────────────────┨ - # This third argument marks this as a Data Artifact + ┃ UPDATED_AT │ 2023-05-19 09:04:33.557127 ┃ - Annotated[str, ArtifactConfig("deployment_uri", is_deployment_artifact=True)], + ┗━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ - ]: + Configuration - ...' -- source_sentence: How can I load and render reports in a Jupyter notebook using ZenML? + + ┏━━━━━━━━━━━━┯━━━━━━━━━━━━┓' +- source_sentence: Where can I find the instructions to enable CUDA for GPU-backed + hardware in ZenML SDK Docs? sentences: - - '🗄️Handle Data/Artifacts + - 'Configure a code repository - Step outputs in ZenML are stored in the artifact store. This enables caching, - lineage and auditability. Using type annotations helps with transparency, passing - data between steps, and serializing/des + Connect a Git repository to ZenML to track code changes and collaborate on MLOps + projects. - For best results, use type annotations for your outputs. This is good coding practice - for transparency, helps ZenML handle passing data between steps, and also enables - ZenML to serialize and deserialize (referred to as ''materialize'' in ZenML) the - data. + Throughout the lifecycle of a MLOps pipeline, it can get quite tiresome to always + wait for a Docker build every time after running a pipeline (even if the local + Docker cache is used). However, there is a way to just have one pipeline build + and keep reusing it until a change to the pipeline environment is made: by connecting + a code repository. - @step + With ZenML, connecting to a Git repository optimizes the Docker build processes. + It also has the added bonus of being a better way of managing repository changes + and enabling better code collaboration. Here is how the flow changes when running + a pipeline: + + + You trigger a pipeline run on your local machine. ZenML parses the @pipeline function + to determine the necessary steps. + + + The local client requests stack information from the ZenML server, which responds + with the cloud stack configuration. + + + The local client detects that we''re using a code repository and requests the + information from the git repo. + + + Instead of building a new Docker image, the client checks if an existing image + can be reused based on the current Git commit hash and other environment metadata. + + + The client initiates a run in the orchestrator, which sets up the execution environment + in the cloud, such as a VM. - def load_data(parameter: int) -> Dict[str, Any]: + The orchestrator downloads the code directly from the Git repository and uses + the existing Docker image to run the pipeline steps. - # do something with the parameter here + Pipeline steps execute, storing artifacts in the cloud-based artifact store. - training_data = [[1, 2], [3, 4], [5, 6]] + Throughout the execution, the pipeline run status and metadata are reported back + to the ZenML server. - labels = [0, 1, 0] + By connecting a Git repository, you avoid redundant builds and make your MLOps + processes more efficient. Your team can work on the codebase simultaneously, with + ZenML handling the version tracking and ensuring that the correct code version + is always used for each run. - return {''features'': training_data, ''labels'': labels} + Creating a GitHub Repository' + - 'Migration guide 0.39.1 → 0.41.0 + + + How to migrate your ZenML pipelines and steps from version <=0.39.1 to 0.41.0. + + + ZenML versions 0.40.0 to 0.41.0 introduced a new and more flexible syntax to define + ZenML steps and pipelines. This page contains code samples that show you how to + upgrade your steps and pipelines to the new syntax. + + + Newer versions of ZenML still work with pipelines and steps defined using the + old syntax, but the old syntax is deprecated and will be removed in the future. + + + Overview + + + from typing import Optional + + + from zenml.steps import BaseParameters, Output, StepContext, step + + + from zenml.pipelines import pipeline + + + # Define a Step + + + class MyStepParameters(BaseParameters): + + + param_1: int + + + param_2: Optional[float] = None @step - def train_model(data: Dict[str, Any]) -> None: + def my_step( - total_features = sum(map(sum, data[''features''])) + params: MyStepParameters, context: StepContext, - total_labels = sum(data[''labels'']) + ) -> Output(int_output=int, str_output=str): - # Train some model here + result = int(params.param_1 * (params.param_2 or 1)) - print(f"Trained model using {len(data[''features''])} data points. " + result_uri = context.get_output_artifact_uri() - f"Feature sum is {total_features}, label sum is {total_labels}") + return result, result_uri - @pipeline + # Run the Step separately - def simple_ml_pipeline(parameter: int): + my_step.entrypoint() - dataset = load_data(parameter=parameter) # Get the output + # Define a Pipeline - train_model(dataset) # Pipe the previous step output into the downstream step + @pipeline - In this code, we define two steps: load_data and train_model. The load_data step - takes an integer parameter and returns a dictionary containing training data and - labels. The train_model step receives the dictionary from load_data, extracts - the features and labels, and trains a model (not shown here). + def my_pipeline(my_step): - Finally, we define a pipeline simple_ml_pipeline that chains the load_data and - train_model steps together. The output from load_data is passed as input to train_model, - demonstrating how data flows between steps in a ZenML pipeline. + my_step() - PreviousDisable colorful logging + step_instance = my_step(params=MyStepParameters(param_1=17)) - NextHow ZenML stores data + pipeline_instance = my_pipeline(my_step=step_instance) - Last updated 19 days ago' - - 'Generation evaluation + # Configure and run the Pipeline - Evaluate the generation component of your RAG pipeline. + pipeline_instance.configure(enable_cache=False) - Now that we have a sense of how to evaluate the retrieval component of our RAG - pipeline, let''s move on to the generation component. The generation component - is responsible for generating the answer to the question based on the retrieved - context. At this point, our evaluation starts to move into more subjective territory. - It''s harder to come up with metrics that can accurately capture the quality of - the generated answers. However, there are some things we can do. + schedule = Schedule(...) - As with the retrieval evaluation, we can start with a simple approach and then - move on to more sophisticated methods. + pipeline_instance.run(schedule=schedule) - Handcrafted evaluation tests + # Fetch the Pipeline Run - As in the retrieval evaluation, we can start by putting together a set of examples - where we know that our generated output should or shouldn''t include certain terms. - For example, if we''re generating answers to questions about which orchestrators - ZenML supports, we can check that the generated answers include terms like "Airflow" - and "Kubeflow" (since we do support them) and exclude terms like "Flyte" or "Prefect" - (since we don''t (yet!) support them). These handcrafted tests should be driven - by mistakes that you''ve already seen in the RAG output. The negative example - of "Flyte" and "Prefect" showing up in the list of supported orchestrators, for - example, shows up sometimes when you use GPT 3.5 as the LLM. + last_run = pipeline_instance.get_runs()[0] - As another example, when you make a query asking ''what is the default orchestrator - in ZenML?'' you would expect that the answer would include the word ''local'', - so we can make a test case to confirm that. + int_output = last_run.get_step["my_step"].outputs["int_output"].read() - You can view our starter set of these tests here. It''s better to start with something - small and simple and then expand as is needed. There''s no need for complicated - harnesses or frameworks at this stage. + from typing import Annotated, Optional, Tuple - bad_answers table:' - - 'n the respective artifact in the pipeline run DAG.Alternatively, if you are running - inside a Jupyter notebook, you can load and render the reports using the artifact.visualize() - method, e.g.: + from zenml import get_step_context, pipeline, step from zenml.client import Client - def visualize_results(pipeline_name: str, step_name: str) -> None: + # Define a Step - pipeline = Client().get_pipeline(pipeline=pipeline_name) + @step + + + def my_step( + + + param_1: int, param_2: Optional[float] = None + + + ) -> Tuple[Annotated[int, "int_output"], Annotated[str, "str_output"]]: + + + result = int(param_1 * (param_2 or 1)) + + + result_uri = get_step_context().get_output_artifact_uri() + + + return result, result_uri - evidently_step = pipeline.last_run.steps[step_name] + # Run the Step separately - evidently_step.visualize() + my_step() - if __name__ == "__main__": + # Define a Pipeline - visualize_results("text_data_report_pipeline", "text_report") + @pipeline' + - ' SDK Docs . - visualize_results("text_data_test_pipeline", "text_test") + Enabling CUDA for GPU-backed hardwareNote that if you wish to use this step operator + to run steps on a GPU, you will need to follow the instructions on this page to + ensure that it works. It requires adding some extra settings customization and + is essential to enable CUDA for the GPU to give its full acceleration. - PreviousDeepchecks + PreviousStep Operators - NextWhylogs + NextGoogle Cloud VertexAI Last updated 19 days ago' -- source_sentence: How does ZenML check if a path exists in custom stack component - implementations? +- source_sentence: What are the special metadata types supported by ZenML and how + are they used? sentences: - - '(self, path): + - 'Special Metadata Types - """Checks if a path exists.""" + Tracking your metadata. - ...As each component defines a different interface, make sure to check out the - base class definition of the component type that you want to implement and also - check out the documentation on how to extend specific stack components. + ZenML supports several special metadata types to capture specific kinds of information. + Here are examples of how to use the special types Uri, Path, DType, and StorageSize: - If you would like to automatically track some metadata about your custom stack - component with each pipeline run, you can do so by defining some additional methods - in your stack component implementation class as shown in the Tracking Custom Stack - Component Metadata section. + from zenml.metadata.metadata_types import StorageSize, DType - See the full code of the base StackComponent class here. + from zenml import log_artifact_metadata - Base Abstraction 2: StackComponentConfig + log_artifact_metadata( - As the name suggests, the StackComponentConfig is used to configure a stack component - instance. It is separated from the actual implementation on purpose. This way, - ZenML can use this class to validate the configuration of a stack component during - its registration/update, without having to import heavy (or even non-installed) - dependencies. + metadata={ - The config and settings of a stack component are two separate, yet related entities. - The config is the static part of your flavor''s configuration, defined when you - register your flavor. The settings are the dynamic part of your flavor''s configuration - that can be overridden at runtime. + "dataset_source": Uri("gs://my-bucket/datasets/source.csv"), - You can read more about the differences here. + "preprocessing_script": Path("/scripts/preprocess.py"), - Let us now continue with the base artifact store example from above and take a - look at the BaseArtifactStoreConfig: + "column_types": { - from zenml.stack import StackComponentConfig + "age": DType("int"), - class BaseArtifactStoreConfig(StackComponentConfig): + "income": DType("float"), - """Config class for `BaseArtifactStore`.""" + "score": DType("int") - path: str + }, - SUPPORTED_SCHEMES: ClassVar[Set[str]] + "processed_data_size": StorageSize(2500000) - ... + In this example: - Through the BaseArtifactStoreConfig, each artifact store will require users to - define a path variable. Additionally, the base config requires all artifact store - flavors to define a SUPPORTED_SCHEMES class variable that ZenML will use to check - if the user-provided path is actually supported by the flavor. + Uri is used to indicate a dataset source URI. - See the full code of the base StackComponentConfig class here. + Path is used to specify the filesystem path to a preprocessing script. - Base Abstraction 3: Flavor' - - 'otated[int, "remainder"] + DType is used to describe the data types of specific columns. - ]: + StorageSize is used to indicate the size of the processed data in bytes. - return a // b, a % bIf you do not give your outputs custom names, the created - artifacts will be named {pipeline_name}::{step_name}::output or {pipeline_name}::{step_name}::output_{i} - in the dashboard. See the documentation on artifact versioning and configuration - for more information. + These special types help standardize the format of metadata and ensure that it + is logged in a consistent and interpretable manner. - See Also: + PreviousGroup metadata - Learn more about output annotation here + NextFetch metadata within steps - For custom data types you should check these docs out + Last updated 19 days ago' + - 's is achieved using the log_model_metadata method:from zenml import get_step_context, + step, log_model_metadata - PreviousUse pipeline/step parameters + @step - NextControl caching behavior + def svc_trainer( - Last updated 16 days ago' - - 'pose -f /path/to/docker-compose.yml -p zenml up -dYou need to visit the ZenML - dashboard at http://localhost:8080 to activate the server by creating an initial - admin account. You can then connect your client to the server with the web login - flow: + X_train: pd.DataFrame, - zenml connect --url http://localhost:8080 + y_train: pd.Series, - Tearing down the installation is as simple as running: + gamma: float = 0.001, - docker-compose -p zenml down + ) -> Annotated[ClassifierMixin, "sklearn_classifier"],: - Database backup and recovery + # Train and score model - An automated database backup and recovery feature is enabled by default for all - Docker deployments. The ZenML server will automatically back up the database in-memory - before every database schema migration and restore it if the migration fails. + ... - The database backup automatically created by the ZenML server is only temporary - and only used as an immediate recovery in case of database migration failures. - It is not meant to be used as a long-term backup solution. If you need to back - up your database for long-term storage, you should use a dedicated backup solution. + model.fit(dataset[0], dataset[1]) - Several database backup strategies are supported, depending on where and how the - backup is stored. The strategy can be configured by means of the ZENML_STORE_BACKUP_STRATEGY - environment variable: + accuracy = model.score(dataset[0], dataset[1]) - disabled - no backup is performed + model = get_step_context().model - in-memory - the database schema and data are stored in memory. This is the fastest - backup strategy, but the backup is not persisted across container restarts, so - no manual intervention is possible in case the automatic DB recovery fails after - a failed DB migration. Adequate memory resources should be allocated to the ZenML - server container when using this backup strategy with larger databases. This is - the default backup strategy.' -- source_sentence: How do I configure a custom materializer for MyObj in ZenML? - sentences: - - 'Autogenerate a template yaml file + log_model_metadata( - To help you figure out what you can put in your configuration file, simply autogenerate - a template. + # Model name can be omitted if specified in the step or pipeline context - If you want to generate a template yaml file of your specific pipeline, you can - do so by using the .write_run_configuration_template() method. This will generate - a yaml file with all options commented out. This way you can pick and choose the - settings that are relevant to you. + model_name="iris_classifier", - from zenml import pipeline + # Passing None or omitting this will use the `latest` version - ... + version=None, - @pipeline(enable_cache=True) # set cache behavior at step level + # Metadata should be a dictionary of JSON-serializable values - def simple_ml_pipeline(parameter: int): + metadata={"accuracy": float(accuracy)} - dataset = load_data(parameter=parameter) + # A dictionary of dictionaries can also be passed to group metadata - train_model(dataset) + # in the dashboard - simple_ml_pipeline.write_run_configuration_template(path="") + # metadata = {"metrics": {"accuracy": accuracy}} - build: Union[PipelineBuildBase, UUID, NoneType] + from zenml.client import Client - enable_artifact_metadata: Optional[bool] + # Get an artifact version (in this the latest `iris_classifier`) - enable_artifact_visualization: Optional[bool] + model_version = Client().get_model_version(''iris_classifier'') - enable_cache: Optional[bool] + # Fetch it''s metadata - enable_step_logs: Optional[bool] + model_version.run_metadata["accuracy"].value - extra: Mapping[str, Any] + The ZenML Pro dashboard offers advanced visualization features for artifact exploration, + including a dedicated artifacts tab with metadata visualization: - model: + Choosing log metadata with artifacts or model versions depends on the scope and + purpose of the information you wish to capture. Artifact metadata is best for + details specific to individual outputs, while model version metadata is suitable + for broader information relevant to the overall model. By utilizing ZenML''s metadata + logging capabilities and special types, you can enhance the traceability, reproducibility, + and analysis of your ML workflows. - audience: Optional[str] + Once metadata has been logged to a model, we can retrieve it easily with the client: - description: Optional[str] + from zenml.client import Client - ethics: Optional[str] + client = Client() - license: Optional[str] + model = client.get_model_version("my_model", "my_version") - limitations: Optional[str] + print(model.run_metadata["metadata_key"].value)' + - 'Hugging Face - name: str + Deploying models to Huggingface Inference Endpoints with Hugging Face :hugging_face:. - save_models_to_registry: bool + Hugging Face Inference Endpoints provides a secure production solution to easily + deploy any transformers, sentence-transformers, and diffusers models on a dedicated + and autoscaling infrastructure managed by Hugging Face. An Inference Endpoint + is built from a model from the Hub. - suppress_class_validation_warnings: bool + This service provides dedicated and autoscaling infrastructure managed by Hugging + Face, allowing you to deploy models without dealing with containers and GPUs. - tags: Optional[List[str]] + When to use it? - trade_offs: Optional[str] + You should use Hugging Face Model Deployer: - use_cases: Optional[str] + if you want to deploy Transformers, Sentence-Transformers, or Diffusion models + on dedicated and secure infrastructure. - version: Union[ModelStages, int, str, NoneType] + if you prefer a fully-managed production solution for inference without the need + to handle containers and GPUs. - was_created_in_this_run: bool + if your goal is to turn your models into production-ready APIs with minimal infrastructure + or MLOps involvement - parameters: Optional[Mapping[str, Any]] + Cost-effectiveness is crucial, and you want to pay only for the raw compute resources + you use. - run_name: Optional[str] + Enterprise security is a priority, and you need to deploy models into secure offline + endpoints accessible only via a direct connection to your Virtual Private Cloud + (VPCs). - schedule: + If you are looking for a more easy way to deploy your models locally, you can + use the MLflow Model Deployer flavor. - catchup: bool + How to deploy it? - cron_expression: Optional[str] + The Hugging Face Model Deployer flavor is provided by the Hugging Face ZenML integration, + so you need to install it on your local machine to be able to deploy your models. + You can do this by running the following command: - end_time: Optional[datetime] + zenml integration install huggingface -y - interval_second: Optional[timedelta] + To register the Hugging Face model deployer with ZenML you need to run the following + command: - name: Optional[str] + zenml model-deployer register --flavor=huggingface --token= + --namespace= - run_once_start_time: Optional[datetime] + Here, - start_time: Optional[datetime] + token parameter is the Hugging Face authentication token. It can be managed through + Hugging Face settings.' +- source_sentence: What are the benefits of deploying stack components directly from + the ZenML CLI? + sentences: + - 'cess this particular Kubernetes cluster in AWS ?":zenml service-connector verify + aws-multi-type --resource-type s3-bucket - settings: + Example Command Output - docker: + Service connector ''aws-multi-type'' is correctly configured with valid credentials + and has access to the following resources: - apt_packages: List[str] + ┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ - build_context_root: Optional[str] + ┃ RESOURCE TYPE │ RESOURCE NAMES ┃ - build_options: Mapping[str, Any] + ┠───────────────┼───────────────────────────────────────┨ - copy_files: bool + ┃ 📦 s3-bucket │ s3://aws-ia-mwaa-715803424590 ┃ - copy_global_config: bool + ┃ │ s3://zenfiles ┃ - dockerfile: Optional[str] + ┃ │ s3://zenml-demos ┃ - dockerignore: Optional[str] + ┃ │ s3://zenml-generative-chat ┃ - environment: Mapping[str, Any] + ┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ - install_stack_requirements: bool + zenml service-connector verify aws-multi-type --resource-type kubernetes-cluster + --resource-id zenhacks-cluster - parent_image: Optional[str] + Example Command Output - python_package_installer: PythonPackageInstaller' - - 'How to configure a pipeline with a YAML + Service connector ''aws-multi-type'' is correctly configured with valid credentials + and has access to the following resources: - Specify a configuration file + ┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┓ - All configuration that can be specified in a YAML file can also be specified in - code itself. However, it is best practice to use a YAML file to separate config - from code. + ┃ RESOURCE TYPE │ RESOURCE NAMES ┃ - You can use the with_options(config_path=) pattern to apply your - configuration to a pipeline. Here is a minimal example of using a file based configuration - yaml. + ┠───────────────────────┼──────────────────┨ - enable_cache: False + ┃ 🌀 kubernetes-cluster │ zenhacks-cluster ┃ - # Configure the pipeline parameters + ┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┛ - parameters: + Verifying the multi-instance Service Connector displays all the resources that + it can access. We can also scope the verification to a single resource: - dataset_name: "best_dataset" + zenml service-connector verify aws-s3-multi-instance - steps: + Example Command Output - load_data: # Use the step name here + Service connector ''aws-s3-multi-instance'' is correctly configured with valid + credentials and has access to the following resources: - enable_cache: False # same as @step(enable_cache=False) + ┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ - from zenml import step, pipeline + ┃ RESOURCE TYPE │ RESOURCE NAMES ┃ - @step + ┠───────────────┼───────────────────────────────────────┨ - def load_data(dataset_name: str) -> dict: + ┃ 📦 s3-bucket │ s3://aws-ia-mwaa-715803424590 ┃ - ... + ┃ │ s3://zenfiles ┃ - @pipeline # This function combines steps together + ┃ │ s3://zenml-demos ┃ - def simple_ml_pipeline(dataset_name: str): + ┃ │ s3://zenml-generative-chat ┃' + - ' for GCS, Docker and Kubernetes Python clients. Italso allows for the configuration + of local Docker and Kubernetes CLIs. - load_data(dataset_name) + The GCP Service Connector is part of the GCP ZenML integration. You can either - if __name__=="__main__": + install the entire integration or use a pypi extra to install it independently - simple_ml_pipeline.with_options(config_path=)() + of the integration: - The above would run the simple_ml_pipeline with cache disabled for load_data and - the parameter dataset_name set to best_dataset. + pip install "zenml[connectors-gcp]" installs only prerequisites for the GCP - PreviousUse configuration files + Service Connector Type - NextWhat can be configured + zenml integration install gcp installs the entire GCP ZenML integration - Last updated 15 days ago' - - '_init__(self, name: str): + It is not required to install and set up the GCP CLI on your local machine to - self.name = name + use the GCP Service Connector to link Stack Components to GCP resources and - @stepdef my_first_step() -> MyObj: + services. However, it is recommended to do so if you are looking for a quick - """Step that returns an object of type MyObj.""" + setup that includes using the auto-configuration Service Connector features. - return MyObj("my_object") + ────────────────────────────────────────────────────────────────────────────────── - @step + Fetching details about the GCP kubernetes-cluster resource type (i.e. the GKE + cluster): - def my_second_step(my_obj: MyObj) -> None: + zenml service-connector describe-type gcp --resource-type kubernetes-cluster - """Step that logs the input object and returns nothing.""" + Example Command Output - logging.info( + ╔══════════════════════════════════════════════════════════════════════════════╗ - f"The following object was passed to this step: `{my_obj.name}`" + ║ 🌀 GCP GKE Kubernetes cluster (resource type: kubernetes-cluster) ║ - @pipeline + ╚══════════════════════════════════════════════════════════════════════════════╝ - def first_pipeline(): + Authentication methods: implicit, user-account, service-account, oauth2-token, - output_1 = my_first_step() + impersonation - my_second_step(output_1) + Supports resource instances: True - first_pipeline() + Authentication methods: - Running the above without a custom materializer will work but print the following - warning: + 🔒 implicit - No materializer is registered for type MyObj, so the default Pickle materializer - was used. Pickle is not production ready and should only be used for prototyping - as the artifacts cannot be loaded when running with a different Python version. - Please consider implementing a custom materializer for type MyObj + 🔒 user-account - To get rid of this warning and make our pipeline more robust, we will subclass - the BaseMaterializer class, listing MyObj in ASSOCIATED_TYPES, and overwriting - load() and save(): + 🔒 service-account - import os + 🔒 oauth2-token - from typing import Type + 🔒 impersonation - from zenml.enums import ArtifactType + Allows Stack Components to access a GKE registry as a standard Kubernetes - from zenml.materializers.base_materializer import BaseMaterializer + cluster resource. When used by Stack Components, they are provided a - class MyMaterializer(BaseMaterializer): + pre-authenticated Python Kubernetes client instance. - ASSOCIATED_TYPES = (MyObj,) + The configured credentials must have at least the following GCP permissions - ASSOCIATED_ARTIFACT_TYPE = ArtifactType.DATA + associated with the GKE clusters that it can access: - def load(self, data_type: Type[MyObj]) -> MyObj: + container.clusters.list - """Read from artifact store.""" + container.clusters.get - with self.artifact_store.open(os.path.join(self.uri, ''data.txt''), ''r'') as - f: + In addition to the above permissions, the credentials should include permissions' + - '⚒️Manage stacks - name = f.read() + Deploying your stack components directly from the ZenML CLI - return MyObj(name=name) + The first step in running your pipelines on remote infrastructure is to deploy + all the components that you would need, like an MLflow tracking server, a Seldon + Core model deployer, and more to your cloud. - def save(self, my_obj: MyObj) -> None: + This can bring plenty of benefits like scalability, reliability, and collaboration. + ZenML eases the path to production by providing a seamless way for all tools to + interact with others through the use of abstractions. However, one of the most + painful parts of this process, from what we see on our Slack and in general, is + the deployment of these stack components. - """Write to artifact store.""" + Deploying and managing MLOps tools is tricky 😭😵‍💫 - with self.artifact_store.open(os.path.join(self.uri, ''data.txt''), ''w'') as - f: + It is not trivial to set up all the different tools that you might need for your + pipeline. - f.write(my_obj.name) + 🌈 Each tool comes with a certain set of requirements. For example, a Kubeflow + installation will require you to have a Kubernetes cluster, and so would a Seldon + Core deployment. - Pro-tip: Use the self.artifact_store property to ensure your materialization logic - works across artifact stores (local and remote like S3 buckets). + 🤔 Figuring out the defaults for infra parameters is not easy. Even if you have + identified the backing infra that you need for a stack component, setting up reasonable + defaults for parameters like instance size, CPU, memory, etc., needs a lot of + experimentation to figure out. - Now, ZenML can use this materializer to handle the outputs and inputs of your - customs object. Edit the pipeline as follows to see this in action: + 🚧 Many times, standard tool installations don''t work out of the box. For example, + to run a custom pipeline in Vertex AI, it is not enough to just run an imported + pipeline. You might also need a custom service account that is configured to perform + tasks like reading secrets from your secret store or talking to other GCP services + that your pipeline might need. - my_first_step.configure(output_materializers=MyMaterializer) + 🔐 Some tools need an additional layer of installations to enable a more secure, + production-grade setup. For example, a standard MLflow tracking server deployment + comes without an authentication frontend which might expose all of your tracking + data to the world if deployed as-is.' +- source_sentence: What is the expiration time for the GCP OAuth2 token in the ZenML + configuration? + sentences: + - '━━━━━┛ - first_pipeline()' -- source_sentence: Where can I find the code logic for storing embeddings in PostgreSQL? - sentences: - - 'ponses to user queries in real-time. + Configuration - Code ExampleTo explore the full code, visit the Complete Guide repository. The - logic for storing the embeddings in PostgreSQL can be found here. + ┏━━━━━━━━━━━━┯━━━━━━━━━━━━┓┃ PROPERTY │ VALUE ┃ - PreviousEmbeddings generation + ┠────────────┼────────────┨ - NextBasic RAG inference pipeline + ┃ project_id │ zenml-core ┃ - Last updated 19 days ago' - - 'Local Image Builder + ┠────────────┼────────────┨ - Building container images locally. + ┃ token │ [HIDDEN] ┃ - The local image builder is an image builder flavor that comes built-in with ZenML - and uses the local Docker installation on your client machine to build container - images. + ┗━━━━━━━━━━━━┷━━━━━━━━━━━━┛ - ZenML uses the official Docker Python library to build and push your images. This - library loads its authentication credentials to push images from the default config - location: $HOME/.docker/config.json. If your Docker configuration is stored in - a different directory, you can use the environment variable DOCKER_CONFIG to override - this behavior: + Note the temporary nature of the Service Connector. It will expire and become + unusable in 1 hour: - export DOCKER_CONFIG=/path/to/config_dir + zenml service-connector list --name gcp-oauth2-token - The directory that you specify here must contain your Docker configuration in - a file called config.json. + Example Command Output - When to use it + ┏━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━┓ - You should use the local image builder if: + ┃ ACTIVE │ NAME │ ID │ TYPE │ + RESOURCE TYPES │ RESOURCE NAME │ SHARED │ OWNER │ EXPIRES IN │ LABELS + ┃ - you''re able to install and use Docker on your client machine. + ┠────────┼──────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼───────────────┼────────┼─────────┼────────────┼────────┨ - you want to use remote components that require containerization without the additional - hassle of configuring infrastructure for an additional component. + ┃ │ gcp-oauth2-token │ ec4d7d85-c71c-476b-aa76-95bf772c90da │ 🔵 gcp │ 🔵 + gcp-generic │ │ ➖ │ default │ 59m35s │ ┃ - How to deploy it + ┃ │ │ │ │ + 📦 gcs-bucket │ │ │ │ │ ┃ - The local image builder comes with ZenML and works without any additional setup. + ┃ │ │ │ │ + 🌀 kubernetes-cluster │ │ │ │ │ ┃ - How to use it + ┃ │ │ │ │ + 🐳 docker-registry │ │ │ │ │ ┃ - To use the Local image builder, we need: + ┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┛ - Docker installed and running. + Auto-configuration - The Docker client authenticated to push to the container registry that you intend - to use in the same stack. + The GCP Service Connector allows auto-discovering and fetching credentials and + configuration set up by the GCP CLI on your local host.' + - 'strator │ eks_seldon │ aws_secret_manager ┃┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼────────────────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┨ - We can then register the image builder and use it to create a new stack: + ┃ 👉 │ default │ fe913bb5-e631-4d4e-8c1b-936518190ebb │ │ + default │ │ default │ default │ │ ┃ - zenml image-builder register --flavor=local + ┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┛ - # Register and activate a stack with the new image builder + Example of migrating a profile into the default project using a name prefix: - zenml stack register -i ... --set + $ zenml profile migrate /home/stefan/.config/zenml/profiles/zenbytes --prefix + zenbytes_ - For more information and a full list of configurable attributes of the local image - builder, check out the SDK Docs . + No component flavors to migrate from /home/stefan/.config/zenml/profiles/zenbytes/stacks.yaml... - PreviousImage Builders + Migrating stack components from /home/stefan/.config/zenml/profiles/zenbytes/stacks.yaml... - NextKaniko Image Builder + Created artifact_store ''zenbytes_s3_store'' with flavor ''s3''. - Last updated 15 days ago' - - ' asked to access: + Created container_registry ''zenbytes_ecr_registry'' with flavor ''default''. - from zenml.client import Clientclient = Client() + Created experiment_tracker ''zenbytes_mlflow_tracker'' with flavor ''mlflow''. - # Get a Service Connector client for a particular S3 bucket + Created experiment_tracker ''zenbytes_mlflow_tracker_local'' with flavor ''mlflow''. - connector_client = client.get_service_connector_client( + Created model_deployer ''zenbytes_eks_seldon'' with flavor ''seldon''. - name_id_or_prefix="aws-federation-multi", + Created model_deployer ''zenbytes_mlflow'' with flavor ''mlflow''. - resource_type="s3-bucket", + Created orchestrator ''zenbytes_eks_orchestrator'' with flavor ''kubeflow''. - resource_id="s3://zenfiles" + Created secrets_manager ''zenbytes_aws_secret_manager'' with flavor ''aws''. - # Get the S3 boto3 python client pre-configured and pre-authenticated + Migrating stacks from /home/stefan/.config/zenml/profiles/zenbytes/stacks.yaml... - # from the Service Connector client + Created stack ''zenbytes_aws_kubeflow_stack''. - s3_client = connector_client.connect() + Created stack ''zenbytes_local_with_mlflow''. - # Verify access to the chosen S3 bucket using the temporary token that + $ zenml stack list - # was issued to the client. + Using the default local database. - s3_client.head_bucket(Bucket="zenfiles") + Running with active project: ''default'' (global)' + - 'Evaluation in 65 lines of code - # Try to access another S3 bucket that the original AWS long-lived credentials - can access. + Learn how to implement evaluation for RAG in just 65 lines of code. - # An error will be thrown indicating that the bucket is not accessible. + Our RAG guide included a short example for how to implement a basic RAG pipeline + in just 85 lines of code. In this section, we''ll build on that example to show + how you can evaluate the performance of your RAG pipeline in just 65 lines. For + the full code, please visit the project repository here. The code that follows + requires the functions from the earlier RAG pipeline code to work. - s3_client.head_bucket(Bucket="zenml-demos") + # ...previous RAG pipeline code here... - Example Output + # see https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/most_basic_rag_pipeline.py - >>> from zenml.client import Client + eval_data = [ - >>> + "question": "What creatures inhabit the luminescent forests of ZenML World?", - >>> client = Client() + "expected_answer": "The luminescent forests of ZenML World are inhabited by glowing + Zenbots.", - Unable to find ZenML repository in your current working directory (/home/stefan/aspyre/src/zenml) - or any parent directories. If you want to use an existing repository which is - in a different location, set the environment variable ''ZENML_REPOSITORY_PATH''. - If you want to create a new repository, run zenml init. + }, - Running without an active repository root. + "question": "What do Fractal Fungi do in the melodic caverns of ZenML World?", - >>> + "expected_answer": "Fractal Fungi emit pulsating tones that resonate through the + crystalline structures, creating a symphony of otherworldly sounds in the melodic + caverns of ZenML World.", - >>> # Get a Service Connector client for a particular S3 bucket + }, - >>> connector_client = client.get_service_connector_client( + "question": "Where do Gravitational Geckos live in ZenML World?", - ... name_id_or_prefix="aws-federation-multi", + "expected_answer": "Gravitational Geckos traverse the inverted cliffs of ZenML + World.", - ... resource_type="s3-bucket", + }, - ... resource_id="s3://zenfiles" + def evaluate_retrieval(question, expected_answer, corpus, top_n=2): - ... ) + relevant_chunks = retrieve_relevant_chunks(question, corpus, top_n) - >>> + score = any( - >>> # Get the S3 boto3 python client pre-configured and pre-authenticated + any(word in chunk for word in tokenize(expected_answer)) - >>> # from the Service Connector client + for chunk in relevant_chunks - >>> s3_client = connector_client.connect() + return score - >>> + def evaluate_generation(question, expected_answer, generated_answer): - >>> # Verify access to the chosen S3 bucket using the temporary token that + client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) - >>> # was issued to the client. + chat_completion = client.chat.completions.create( - >>> s3_client.head_bucket(Bucket="zenfiles")' + messages=[ + + + "role": "system", + + + "content": "You are an evaluation judge. Given a question, an expected answer, + and a generated answer, your task is to determine if the generated answer is relevant + and accurate. Respond with ''YES'' if the generated answer is satisfactory, or + ''NO'' if it is not.", + + + },' model-index: - name: zenml/finetuned-snowflake-arctic-embed-m results: @@ -1106,49 +1176,49 @@ model-index: type: dim_384 metrics: - type: cosine_accuracy@1 - value: 0.41566265060240964 + value: 0.3493975903614458 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.6144578313253012 + value: 0.572289156626506 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.7108433734939759 + value: 0.6445783132530121 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.8132530120481928 + value: 0.7530120481927711 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.41566265060240964 + value: 0.3493975903614458 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.2048192771084337 + value: 0.1907630522088353 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.14216867469879516 + value: 0.1289156626506024 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.08132530120481926 + value: 0.07530120481927709 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.41566265060240964 + value: 0.3493975903614458 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.6144578313253012 + value: 0.572289156626506 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.7108433734939759 + value: 0.6445783132530121 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.8132530120481928 + value: 0.7530120481927711 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.6022022499638306 + value: 0.5491448856982105 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.536194779116466 + value: 0.48445926563396463 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.5430625041344633 + value: 0.4935434037345097 name: Cosine Map@100 - task: type: information-retrieval @@ -1158,49 +1228,49 @@ model-index: type: dim_256 metrics: - type: cosine_accuracy@1 - value: 0.3855421686746988 + value: 0.3253012048192771 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.6144578313253012 + value: 0.5783132530120482 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.6566265060240963 + value: 0.6445783132530121 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.7650602409638554 + value: 0.7469879518072289 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.3855421686746988 + value: 0.3253012048192771 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.2048192771084337 + value: 0.19277108433734935 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.13132530120481925 + value: 0.1289156626506024 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.07650602409638554 + value: 0.07469879518072287 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.3855421686746988 + value: 0.3253012048192771 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.6144578313253012 + value: 0.5783132530120482 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.6566265060240963 + value: 0.6445783132530121 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.7650602409638554 + value: 0.7469879518072289 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.5703850381958874 + value: 0.5389489067913165 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.5086608338114361 + value: 0.4727792120864409 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.5186259851830761 + value: 0.4829032647176661 name: Cosine Map@100 - task: type: information-retrieval @@ -1210,49 +1280,49 @@ model-index: type: dim_128 metrics: - type: cosine_accuracy@1 - value: 0.3433734939759036 + value: 0.29518072289156627 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.5843373493975904 + value: 0.5421686746987951 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.6506024096385542 + value: 0.6325301204819277 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.7650602409638554 + value: 0.7289156626506024 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.3433734939759036 + value: 0.29518072289156627 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.19477911646586343 + value: 0.180722891566265 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.1301204819277108 + value: 0.12650602409638553 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.07650602409638552 + value: 0.07289156626506023 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.3433734939759036 + value: 0.29518072289156627 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.5843373493975904 + value: 0.5421686746987951 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.6506024096385542 + value: 0.6325301204819277 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.7650602409638554 + value: 0.7289156626506024 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.5495526049170634 + value: 0.5118201113899791 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.4812296806272709 + value: 0.4423407917383822 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.4888829758025171 + value: 0.45200042096222626 name: Cosine Map@100 - task: type: information-retrieval @@ -1262,49 +1332,49 @@ model-index: type: dim_64 metrics: - type: cosine_accuracy@1 - value: 0.2710843373493976 + value: 0.26506024096385544 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.5120481927710844 + value: 0.5240963855421686 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.5963855421686747 + value: 0.5843373493975904 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.6807228915662651 + value: 0.7048192771084337 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.2710843373493976 + value: 0.26506024096385544 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.17068273092369476 + value: 0.17469879518072287 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.11927710843373492 + value: 0.11686746987951806 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.06807228915662648 + value: 0.07048192771084334 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.2710843373493976 + value: 0.26506024096385544 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.5120481927710844 + value: 0.5240963855421686 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.5963855421686747 + value: 0.5843373493975904 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.6807228915662651 + value: 0.7048192771084337 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.4729495357950134 + value: 0.4807789244303294 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.4064567794989483 + value: 0.409741346337732 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.4159487751602236 + value: 0.41754560205770985 name: Cosine Map@100 --- @@ -1358,9 +1428,9 @@ from sentence_transformers import SentenceTransformer model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m") # Run inference sentences = [ - 'Where can I find the code logic for storing embeddings in PostgreSQL?', - 'ponses to user queries in real-time.\n\nCode ExampleTo explore the full code, visit the Complete Guide repository. The logic for storing the embeddings in PostgreSQL can be found here.\n\nPreviousEmbeddings generation\n\nNextBasic RAG inference pipeline\n\nLast updated 19 days ago', - ' asked to access:\n\nfrom zenml.client import Clientclient = Client()\n\n# Get a Service Connector client for a particular S3 bucket\n\nconnector_client = client.get_service_connector_client(\n\nname_id_or_prefix="aws-federation-multi",\n\nresource_type="s3-bucket",\n\nresource_id="s3://zenfiles"\n\n# Get the S3 boto3 python client pre-configured and pre-authenticated\n\n# from the Service Connector client\n\ns3_client = connector_client.connect()\n\n# Verify access to the chosen S3 bucket using the temporary token that\n\n# was issued to the client.\n\ns3_client.head_bucket(Bucket="zenfiles")\n\n# Try to access another S3 bucket that the original AWS long-lived credentials can access.\n\n# An error will be thrown indicating that the bucket is not accessible.\n\ns3_client.head_bucket(Bucket="zenml-demos")\n\nExample Output\n\n>>> from zenml.client import Client\n\n>>>\n\n>>> client = Client()\n\nUnable to find ZenML repository in your current working directory (/home/stefan/aspyre/src/zenml) or any parent directories. If you want to use an existing repository which is in a different location, set the environment variable \'ZENML_REPOSITORY_PATH\'. If you want to create a new repository, run zenml init.\n\nRunning without an active repository root.\n\n>>>\n\n>>> # Get a Service Connector client for a particular S3 bucket\n\n>>> connector_client = client.get_service_connector_client(\n\n... name_id_or_prefix="aws-federation-multi",\n\n... resource_type="s3-bucket",\n\n... resource_id="s3://zenfiles"\n\n... )\n\n>>>\n\n>>> # Get the S3 boto3 python client pre-configured and pre-authenticated\n\n>>> # from the Service Connector client\n\n>>> s3_client = connector_client.connect()\n\n>>>\n\n>>> # Verify access to the chosen S3 bucket using the temporary token that\n\n>>> # was issued to the client.\n\n>>> s3_client.head_bucket(Bucket="zenfiles")', + 'What is the expiration time for the GCP OAuth2 token in the ZenML configuration?', + '━━━━━┛\n\nConfiguration\n\n┏━━━━━━━━━━━━┯━━━━━━━━━━━━┓┃ PROPERTY │ VALUE ┃\n\n┠────────────┼────────────┨\n\n┃ project_id │ zenml-core ┃\n\n┠────────────┼────────────┨\n\n┃ token │ [HIDDEN] ┃\n\n┗━━━━━━━━━━━━┷━━━━━━━━━━━━┛\n\nNote the temporary nature of the Service Connector. It will expire and become unusable in 1 hour:\n\nzenml service-connector list --name gcp-oauth2-token\n\nExample Command Output\n\n┏��━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━┓\n\n┃ ACTIVE │ NAME │ ID │ TYPE │ RESOURCE TYPES │ RESOURCE NAME │ SHARED │ OWNER │ EXPIRES IN │ LABELS ┃\n\n┠────────┼──────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼───────────────┼────────┼─────────┼────────────┼────────┨\n\n┃ │ gcp-oauth2-token │ ec4d7d85-c71c-476b-aa76-95bf772c90da │ 🔵 gcp │ 🔵 gcp-generic │ │ ➖ │ default │ 59m35s │ ┃\n\n┃ │ │ │ │ 📦 gcs-bucket │ │ │ │ │ ┃\n\n┃ │ │ │ │ 🌀 kubernetes-cluster │ │ │ │ │ ┃\n\n┃ │ │ │ │ 🐳 docker-registry │ │ │ │ │ ┃\n\n┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┛\n\nAuto-configuration\n\nThe GCP Service Connector allows auto-discovering and fetching credentials and configuration set up by the GCP CLI on your local host.', + 'Evaluation in 65 lines of code\n\nLearn how to implement evaluation for RAG in just 65 lines of code.\n\nOur RAG guide included a short example for how to implement a basic RAG pipeline in just 85 lines of code. In this section, we\'ll build on that example to show how you can evaluate the performance of your RAG pipeline in just 65 lines. For the full code, please visit the project repository here. The code that follows requires the functions from the earlier RAG pipeline code to work.\n\n# ...previous RAG pipeline code here...\n\n# see https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/most_basic_rag_pipeline.py\n\neval_data = [\n\n"question": "What creatures inhabit the luminescent forests of ZenML World?",\n\n"expected_answer": "The luminescent forests of ZenML World are inhabited by glowing Zenbots.",\n\n},\n\n"question": "What do Fractal Fungi do in the melodic caverns of ZenML World?",\n\n"expected_answer": "Fractal Fungi emit pulsating tones that resonate through the crystalline structures, creating a symphony of otherworldly sounds in the melodic caverns of ZenML World.",\n\n},\n\n"question": "Where do Gravitational Geckos live in ZenML World?",\n\n"expected_answer": "Gravitational Geckos traverse the inverted cliffs of ZenML World.",\n\n},\n\ndef evaluate_retrieval(question, expected_answer, corpus, top_n=2):\n\nrelevant_chunks = retrieve_relevant_chunks(question, corpus, top_n)\n\nscore = any(\n\nany(word in chunk for word in tokenize(expected_answer))\n\nfor chunk in relevant_chunks\n\nreturn score\n\ndef evaluate_generation(question, expected_answer, generated_answer):\n\nclient = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))\n\nchat_completion = client.chat.completions.create(\n\nmessages=[\n\n"role": "system",\n\n"content": "You are an evaluation judge. Given a question, an expected answer, and a generated answer, your task is to determine if the generated answer is relevant and accurate. Respond with \'YES\' if the generated answer is satisfactory, or \'NO\' if it is not.",\n\n},', ] embeddings = model.encode(sentences) print(embeddings.shape) @@ -1406,21 +1476,21 @@ You can finetune this model on your own dataset. | Metric | Value | |:--------------------|:-----------| -| cosine_accuracy@1 | 0.4157 | -| cosine_accuracy@3 | 0.6145 | -| cosine_accuracy@5 | 0.7108 | -| cosine_accuracy@10 | 0.8133 | -| cosine_precision@1 | 0.4157 | -| cosine_precision@3 | 0.2048 | -| cosine_precision@5 | 0.1422 | -| cosine_precision@10 | 0.0813 | -| cosine_recall@1 | 0.4157 | -| cosine_recall@3 | 0.6145 | -| cosine_recall@5 | 0.7108 | -| cosine_recall@10 | 0.8133 | -| cosine_ndcg@10 | 0.6022 | -| cosine_mrr@10 | 0.5362 | -| **cosine_map@100** | **0.5431** | +| cosine_accuracy@1 | 0.3494 | +| cosine_accuracy@3 | 0.5723 | +| cosine_accuracy@5 | 0.6446 | +| cosine_accuracy@10 | 0.753 | +| cosine_precision@1 | 0.3494 | +| cosine_precision@3 | 0.1908 | +| cosine_precision@5 | 0.1289 | +| cosine_precision@10 | 0.0753 | +| cosine_recall@1 | 0.3494 | +| cosine_recall@3 | 0.5723 | +| cosine_recall@5 | 0.6446 | +| cosine_recall@10 | 0.753 | +| cosine_ndcg@10 | 0.5491 | +| cosine_mrr@10 | 0.4845 | +| **cosine_map@100** | **0.4935** | #### Information Retrieval * Dataset: `dim_256` @@ -1428,43 +1498,43 @@ You can finetune this model on your own dataset. | Metric | Value | |:--------------------|:-----------| -| cosine_accuracy@1 | 0.3855 | -| cosine_accuracy@3 | 0.6145 | -| cosine_accuracy@5 | 0.6566 | -| cosine_accuracy@10 | 0.7651 | -| cosine_precision@1 | 0.3855 | -| cosine_precision@3 | 0.2048 | -| cosine_precision@5 | 0.1313 | -| cosine_precision@10 | 0.0765 | -| cosine_recall@1 | 0.3855 | -| cosine_recall@3 | 0.6145 | -| cosine_recall@5 | 0.6566 | -| cosine_recall@10 | 0.7651 | -| cosine_ndcg@10 | 0.5704 | -| cosine_mrr@10 | 0.5087 | -| **cosine_map@100** | **0.5186** | +| cosine_accuracy@1 | 0.3253 | +| cosine_accuracy@3 | 0.5783 | +| cosine_accuracy@5 | 0.6446 | +| cosine_accuracy@10 | 0.747 | +| cosine_precision@1 | 0.3253 | +| cosine_precision@3 | 0.1928 | +| cosine_precision@5 | 0.1289 | +| cosine_precision@10 | 0.0747 | +| cosine_recall@1 | 0.3253 | +| cosine_recall@3 | 0.5783 | +| cosine_recall@5 | 0.6446 | +| cosine_recall@10 | 0.747 | +| cosine_ndcg@10 | 0.5389 | +| cosine_mrr@10 | 0.4728 | +| **cosine_map@100** | **0.4829** | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) -| Metric | Value | -|:--------------------|:-----------| -| cosine_accuracy@1 | 0.3434 | -| cosine_accuracy@3 | 0.5843 | -| cosine_accuracy@5 | 0.6506 | -| cosine_accuracy@10 | 0.7651 | -| cosine_precision@1 | 0.3434 | -| cosine_precision@3 | 0.1948 | -| cosine_precision@5 | 0.1301 | -| cosine_precision@10 | 0.0765 | -| cosine_recall@1 | 0.3434 | -| cosine_recall@3 | 0.5843 | -| cosine_recall@5 | 0.6506 | -| cosine_recall@10 | 0.7651 | -| cosine_ndcg@10 | 0.5496 | -| cosine_mrr@10 | 0.4812 | -| **cosine_map@100** | **0.4889** | +| Metric | Value | +|:--------------------|:----------| +| cosine_accuracy@1 | 0.2952 | +| cosine_accuracy@3 | 0.5422 | +| cosine_accuracy@5 | 0.6325 | +| cosine_accuracy@10 | 0.7289 | +| cosine_precision@1 | 0.2952 | +| cosine_precision@3 | 0.1807 | +| cosine_precision@5 | 0.1265 | +| cosine_precision@10 | 0.0729 | +| cosine_recall@1 | 0.2952 | +| cosine_recall@3 | 0.5422 | +| cosine_recall@5 | 0.6325 | +| cosine_recall@10 | 0.7289 | +| cosine_ndcg@10 | 0.5118 | +| cosine_mrr@10 | 0.4423 | +| **cosine_map@100** | **0.452** | #### Information Retrieval * Dataset: `dim_64` @@ -1472,21 +1542,21 @@ You can finetune this model on your own dataset. | Metric | Value | |:--------------------|:-----------| -| cosine_accuracy@1 | 0.2711 | -| cosine_accuracy@3 | 0.512 | -| cosine_accuracy@5 | 0.5964 | -| cosine_accuracy@10 | 0.6807 | -| cosine_precision@1 | 0.2711 | -| cosine_precision@3 | 0.1707 | -| cosine_precision@5 | 0.1193 | -| cosine_precision@10 | 0.0681 | -| cosine_recall@1 | 0.2711 | -| cosine_recall@3 | 0.512 | -| cosine_recall@5 | 0.5964 | -| cosine_recall@10 | 0.6807 | -| cosine_ndcg@10 | 0.4729 | -| cosine_mrr@10 | 0.4065 | -| **cosine_map@100** | **0.4159** | +| cosine_accuracy@1 | 0.2651 | +| cosine_accuracy@3 | 0.5241 | +| cosine_accuracy@5 | 0.5843 | +| cosine_accuracy@10 | 0.7048 | +| cosine_precision@1 | 0.2651 | +| cosine_precision@3 | 0.1747 | +| cosine_precision@5 | 0.1169 | +| cosine_precision@10 | 0.0705 | +| cosine_recall@1 | 0.2651 | +| cosine_recall@3 | 0.5241 | +| cosine_recall@5 | 0.5843 | +| cosine_recall@10 | 0.7048 | +| cosine_ndcg@10 | 0.4808 | +| cosine_mrr@10 | 0.4097 | +| **cosine_map@100** | **0.4175** |