Upload Llama-3-1-Varco-8B.ipynb (#5)

- Upload Llama-3-1-Varco-8B.ipynb (2d5e31ec7bac5dc1b3fc25f77a4cc7acd427f373)


Co-authored-by: Jooho Song <[email protected]>

Llama-3-1-Varco-8B.ipynb

@@ -0,0 +1,343 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Deploy Llama-VARCO-8B-Instruct Model from AWS Marketplace \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "Llama-VARCO-8B-Instruct is a generative model built with Llama, specifically designed to excel in Korean through additional training. The model uses continual pre-training with both Korean and English datasets to enhance its understanding and generation capabilites in Korean, while also maintaining its proficiency in English. It performs supervised fine-tuning (SFT) and direct preference optimization (DPO) in Korean to align with human preferences.\n",
+    "\n",
+    "This sample notebook shows you how to deploy [Llama-VARCO-8B-Instruct](https://aws.amazon.com/marketplace/pp/prodview-pynin2e23lb3e) using Amazon SageMaker.\n",
+    "\n",
+    "> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook.\n",
+    "\n",
+    "## Pre-requisites:\n",
+    "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
+    "1. Ensure that IAM role used has **AmazonSageMakerFullAccess**\n",
+    "1. To deploy this ML model successfully, ensure that:\n",
+    "    1. Either your IAM role has these three permissions and you have authority to make AWS Marketplace subscriptions in the AWS account used: \n",
+    "        1. **aws-marketplace:ViewSubscriptions**\n",
+    "        1. **aws-marketplace:Unsubscribe**\n",
+    "        1. **aws-marketplace:Subscribe**  \n",
+    "\n",
+    "## Contents:\n",
+    "1. [Subscribe to the model package](#1.-Subscribe-to-the-model-package)\n",
+    "2. [Create an endpoint and perform real-time inference](#2.-Create-an-endpoint-and-perform-real-time-inference)\n",
+    "3. [Clean-up](#3.-Clean-up)\n",
+    "\n",
+    "    \n",
+    "\n",
+    "## Usage instructions\n",
+    "You can run this notebook one cell at a time (By using Shift+Enter for running a cell)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Subscribe to the model package"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "To subscribe to the model package:\n",
+    "1. Open the model package [listing page](https://aws.amazon.com/marketplace/pp/prodview-pynin2e23lb3e)\n",
+    "1. On the AWS Marketplace listing, click on the **Continue to subscribe** button.\n",
+    "1. On the **Subscribe to this software** page, review and click on **\"Accept Offer\"** if you and your organization agrees with EULA, pricing, and support terms. \n",
+    "1. Once you click on **Continue to configuration button** and then choose a **region**, you will see a **Product Arn** displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3. Copy the ARN corresponding to your region and specify the same in the following cell."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "model_package_arn = \"arn:aws:sagemaker:us-west-2:594846645681:model-package/llama-varco-8b-ist-bedrock-37339dbb44f23f488e24f8671eaa0494\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import base64\n",
+    "import json\n",
+    "import uuid\n",
+    "from sagemaker import ModelPackage\n",
+    "import sagemaker as sage\n",
+    "from sagemaker import get_execution_role\n",
+    "from sagemaker import ModelPackage\n",
+    "import boto3\n",
+    "from IPython.display import Image\n",
+    "from PIL import Image as ImageEdit\n",
+    "import numpy as np\n",
+    "import io"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "role = get_execution_role()\n",
+    "\n",
+    "sagemaker_session = sage.Session()\n",
+    "\n",
+    "bucket = sagemaker_session.default_bucket()\n",
+    "runtime = boto3.client(\"runtime.sagemaker\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Create an endpoint and perform real-time inference"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you want to understand how real-time inference with Amazon SageMaker works, see [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "model_name = \"Llama-VARCO-8B-Instruct\"\n",
+    "\n",
+    "content_type = \"application/json\"\n",
+    "\n",
+    "real_time_inference_instance_type = (\n",
+    "    \"ml.g5.12xlarge\"\n",
+    ")\n",
+    "batch_transform_inference_instance_type = (\n",
+    "    \"ml.g4dn.12xlarge\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### A.Create an endpoint"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# create a deployable model from the model package.\n",
+    "model = ModelPackage(\n",
+    "    role=role, model_package_arn=model_package_arn, sagemaker_session=sagemaker_session\n",
+    ")\n",
+    "\n",
+    "# Deploy the model\n",
+    "predictor = model.deploy(1, real_time_inference_instance_type, endpoint_name=model_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Once endpoint has been created, you would be able to perform real-time inference."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "### B.Create input payload"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "input = {\n",
+    "    \"messages\": [\n",
+    "        {\n",
+    "            \"role\":\"user\",\n",
+    "            \"content\":\"안녕 넌 누구야?\"\n",
+    "        }\n",
+    "    ]\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### C. Perform real-time inference"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##### C-1. Stream Inference Example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "class VarcoInferenceStream():\n",
+    "    def __init__(self, sagemaker_runtime, endpoint_name):\n",
+    "        self.sagemaker_runtime = sagemaker_runtime\n",
+    "        self.endpoint_name = endpoint_name\n",
+    "\n",
+    "    def stream_inference(self, request_body):\n",
+    "        # Gets a streaming inference response\n",
+    "        # from the specified model endpoint:\n",
+    "        response = self.sagemaker_runtime\\\n",
+    "            .invoke_endpoint_with_response_stream(\n",
+    "                EndpointName=self.endpoint_name,\n",
+    "                Body=json.dumps(request_body),\n",
+    "                ContentType=\"application/json\"\n",
+    "        )\n",
+    "        # Gets the EventStream object returned by the SDK:\n",
+    "        for body in response[\"Body\"]:\n",
+    "            raw = body['PayloadPart']['Bytes']\n",
+    "            yield raw.decode()\n",
+    "\n",
+    "\n",
+    "sm_runtime = boto3.client(\"sagemaker-runtime\")\n",
+    "varco_inference_stream = VarcoInferenceStream(sm_runtime, model_name)\n",
+    "stream = varco_inference_stream.stream_inference(input)\n",
+    "for part in stream:\n",
+    "    print(part, end='')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "## 3. Clean-up"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that you have successfully performed a real-time inference, you do not need the endpoint any more. You can terminate the endpoint to avoid being charged."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### A. Delete the endpoint"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.sagemaker_session.delete_endpoint(model_name)\n",
+    "model.sagemaker_session.delete_endpoint_config(model_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### B. Delete the model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.delete_model()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### C. Unsubscribe to the listing (optional)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you would like to unsubscribe to the model package, follow these steps. Before you cancel the subscription, ensure that you do not have any [deployable model](https://console.aws.amazon.com/sagemaker/home#/models) created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model. \n",
+    "\n",
+    "**Steps to unsubscribe to product from AWS Marketplace**:\n",
+    "1. Navigate to __Machine Learning__ tab on [__Your Software subscriptions page__](https://aws.amazon.com/marketplace/ai/library?productType=ml&ref_=mlmp_gitdemo_indust)\n",
+    "2. Locate the listing that you want to cancel the subscription for, and then choose __Cancel Subscription__  to cancel the subscription.\n",
+    "\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "instance_type": "ml.t3.medium",
+  "kernelspec": {
+   "display_name": "conda_pytorch_p310",
+   "language": "python",
+   "name": "conda_pytorch_p310"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}