--- base_model: BAAI/bge-base-en-v1.5 datasets: [] language: - en library_name: sentence-transformers license: apache-2.0 metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:2231 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: - source_sentence: The fact that no customer noticed this major migration to Amazon S3 Glacier Instant Retrieval was a big win for us. It was a seamless experience for end users, and we had no production issues during the entire migration. ” Contact Sales Greater than 99. 99% Outcome | Gaining Insights on AWS to Prioritize Business Needs 한국어 Snap migrated more than 2 exabytes of data—roughly equivalent to 1. 5 trillion media files—seamlessly to Amazon S3 Glacier Instant Retrieval from Amazon S3 Standard-IA. “The fact that no customer noticed this major migration to Amazon S3 Glacier Instant Retrieval was a big win for us,” says Manoharan. “It was a seamless experience for Snapchatters, and we had no production issues during the entire migration. ” As a result of the migration, the company saved tens of millions of dollars on storage. Snap has configured Amazon S3 in 20 AWS Regions around the world so that customers anywhere can retrieve data in milliseconds. The AWS Global Infrastructure is the most secure, extensive, and reliable Global Cloud Infrastructure for a business’s applications. The global reach of AWS lets Snap store media closer to the place where Snapchatters are creating it for optimal performance. Snap is also able to deliver content efficiently using Amazon CloudFront, a content delivery network service built for high performance, security, and availability. “We’ve been able to off-load all of the regionalization work and costs to AWS so that we can focus on developing new features,” says Manoharan. As a result, Snapchat continues to meet its quarterly cost-optimization goals. Overview | Opportunity | Solution | Outcome | AWS Services Used 2 exabytes Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. … In 2016, Snap migrated its data to AWS. “We chose to migrate to AWS because of its global reach, excellent performance, and competitive pricing that, in turn, gave us the ability to reinvest in our business,” says Vijay Manoharan, manager of the media delivery platform team at Snap. Amazon S3 Glacier Instant Retrieval is an archive storage class that delivers the lowest-cost storage for long-lived data that is rarely accessed and requires retrieval in milliseconds. AWS Services Used In 2017, Snap migrated one of the app’s most central features—Snapchat Stories—to Amazon DynamoDB, a fully managed, serverless, NoSQL database designed to run high-performance applications at virtually any scale. Using Amazon DynamoDB, the company experienced greater than 99. sentences: - How did Snap save tens of millions of dollars on storage as a result of migrating to Amazon S3 Glacier Instant Retrieval from Amazon S3 Standard-IA? - How has Panasonic Avionics Corporation leveraged Amazon Aurora MySQL-Compatible Edition and other AWS services to improve the reliability and scalability of its databases for in-flight entertainment and communications systems? - How does Ground Truth Plus ensure the quality of image and video captions generated by human annotators? - source_sentence: ” 中文 (繁體) Bahasa Indonesia Contact Sales Ρусский Customer Stories / Software & Internet عربي 中文 (简体) Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today. Outcome | Expanding Intelligent Features of Virtual Care Amazon Transcribe is an automatic speech recognition service that makes it easy to add speech to text capabilities to any application. Learn more » Learn more » It is critical that video visits are secure, responsive, and reliable. Using AWS helps us provide all this in a performant and scalable way. " Overview With the Amazon Chime SDK, builders can easily add real-time voice, video, and messaging powered by machine learning into their applications. Get Started Beyond traditional use cases, Salesforce is adding capabilities in medication-therapy management, connectivity for care coordinators, and other approaches for patient engagement. The company is developing a new feature that will expand its support of Virtual Care sessions to multiple participants, instead of just clinician and patient. This will facilitate care-team coordination with multiple parties in a single meeting. Using AWS, Salesforce circumvented the heavy lifting that would have been required to build and maintain a video-calling solution from scratch. Patients self-schedule virtual appointments, coordinate previsit activities, and conduct virtual visits in a HIPAA-compliant environment. A patient’s appointment request gets routed to Amazon Chime SDK. Clinicians then review a patient’s intake form and correlate the patient to a Virtual Care session using Amazon Chime SDK messaging, which connects providers and patients with secure, scalable messaging in their web and mobile applications. The Amazon Chime SDK control plane sends event notifications through a default event bus to Amazon EventBridge, a serverless event bus that helps organizations receive, filter, transform, route, and deliver events. Healthcare professionals deliver care over the internet in near real time, which has significantly reduced no-shows for appointments. “Using Amazon Chime SDK, we don’t have to worry about the mechanics of the video call,” Daftari says. “We can focus on features and functions that help differentiate our product in the marketplace, while also significantly improving our speed to launch. ” Salesforce further supports accessibility through embedding closed-captioning of video calls using Amazon Chime SDK live transcription. Amazon Chime SDK sends live audio streams to Amazon Transcribe, which automatically converts speech to text. Salesforce Health Cloud customers can use the live transcription capability to display subtitles, create meeting transcripts, or analyze content. sentences: - How did DB Energie use Amazon SageMaker and AWS to enhance the sustainability and reliability of its power grid operations? - How did Provectus assist Earth.com in enhancing the AI-powered image recognition capabilities of EarthSnap and reducing engineering heavy lifting through the implementation of end-to-end ML pipelines and managed MLOps platform? - How does Salesforce use AWS services such as Amazon Chime SDK and Amazon Transcribe to enhance their Virtual Care sessions for healthcare professionals and patients? - source_sentence: It’s been a great success. ” Overview 93% Validate technical skills and cloud expertise to grow your career and business. Learn more » Amazon Web Services (AWS) Education Programs collaborate with education institutions and the public sector to provide access for individuals to develop cloud computing and digital skills. To help graduates boost their employability, Staffordshire University worked with the AWS team to introduce cloud computing skills training and add cloud courses to its credit-bearing computer science modules. Staffordshire University offers courses through AWS Academy, which empowers higher education institutions to prepare students for industry-recognized certifications and careers. Since the university added AWS Academy courses to its curriculum in 2017, several hundred students have participated. Of those students, 93 percent have achieved employment within 6 months of graduation. Empowered students Türkçe Solution | Learning by Doing Using AWS Learner Labs English With AWS Academy, our students love that they’re not just taking theory lessons. They get to work in actual environments with real AWS tools. ” Next up, Staffordshire University is expanding on the success of its cloud courses by launching additional programs of study developed in collaboration with the AWS team. Staffordshire University and the AWS team designed these programs by "Working Backwards" — an Amazon process that encourages companies to brainstorm solutions by using a customer challenge as the starting point — from the cloud skills employers are currently seeking in the United Kingdom and across the global labor market. One of these programs, which launches in September 2022, is a cloud computing course that features both cloud computing and cybersecurity modules and will offer students more opportunities to discover what’s possible with the AWS Cloud. “What we want to encourage is for students to play with AWS services as well as build confidence with the tools,” says Dr. Champion. to learn remotely using any hardware and earn AWS Certifications Staffordshire University added cloud computing skills training to its curriculum using AWS Education Programs, helping 93 percent of participants find employment within 6 months of graduation. covering cloud skills AWS Certification during the AWS Educate University Challenge Deutsch of graduates find jobs within 6 months Tiếng Việt Italiano ไทย Outcome | Developing New Cloud Coursework About Staffordshire University Staffordshire University is a public research university in Staffordshire, England. Founded in 1914, the university serves over 15,000 students across three schools and four campuses. The United Kingdom has experienced a technology boom in recent years, with technology funding tripling in the first 6 months of 2021 compared to the same period in 2020. In particular, employers need professionals with cloud computing skills ranging from cloud development to machine learning and data analytics. To meet demand, Staffordshire University offers students their choice of six AWS courses covering these key skills and more. sentences: - How has the collaboration between Staffordshire University and the AWS team impacted the employability of graduates in the field of cloud computing? - How can the confidence scores be used to verify the accuracy of sentiment assignments in the sentiment_results_final table, especially for any dubious sentiment assignments? - How did migrating to AWS help Travian Games improve the stability and reliability of their game servers, and what impact did this have on their players' experience? - source_sentence: Contact our experts and start your own AWS journey today. customer and agent experience 2022 Overview WaFd Bank Transforms Contact Centers Using Conversational AI on AWS Customer Stories / Financial Services WaFd uses a data lake on AWS to store and analyze data from phone and chatbot conversations. “We’re getting incredible data from AWS through the conversational logs,” says Hubbard. “That has given us insights into what our customers are asking for so that we can add more self-service functionality. ” The data also gives WaFd more insight into call volumes, so the call center can better manage staff schedules. Opportunity | Using Amazon Lex to Implement an AI-Powered Contact Center Solution Türkçe English WaFd is a US retail and commercial bank with over 200 branches in eight states. In 2019, WaFd founded subsidiary Pike Street Labs, a fintech startup, to drive client-facing digital innovation for the bank. “Banks need to meet customers’ digital expectations,” says Dustin Hubbard, chief technology officer at WaFd Bank and Pike Street Labs. “Every year, customers expect more innovation because that’s what they see from new entrants or in other markets. ” Pike Street Labs redesigned WaFd’s online banking solution to provide personalized customer experiences and began tackling the bank’s customer care center. The company’s previous contact center solution used dated technology with limited features spread across disparate systems. This led to long wait times for customers and frustration for agents, who had to answer incoming calls without prior knowledge of what the customer needed. Agents also bore the burden of identifying fraudulent calls. WaFd needed a solution to improve both the customer and agent experiences. Previously, WaFd used two different systems in its customer care center to manage its voice and chat-based customer interactions, with no way for one system to recognize that an agent was busy on the other. Chat messages remained unanswered because agents would forget to sign in to the chat system. The company implemented chatbots and voice bots powered by Amazon Lex. Now, the call and chat systems are interoperable, and chats can be escalated to agent assisted calls when needed. When a call gets passed to an agent, the system also passes the full chat record and an analysis of the customer’s tone so that the agent is prepared to address the client’s needs and be empathetic toward the caller’s sentiment. WaFd worked with the AWS and Talkdesk teams to create and launch its new contact center solution in July 2022. sentences: - How did Yellow Class optimize its video files and improve performance using AWS services such as AWS Elemental MediaConvert? - How has FanDuel ensured the redundancy and reliability of its live video streams through the use of AWS Elemental MediaConnect and AWS Elemental MediaLive? - How did WaFd Bank use data from phone and chatbot conversations stored in a data lake on AWS to improve self-service functionality and better manage call center staff schedules? - source_sentence: 'Alternatively, you can run the inference via code. Here is one example written in Python, using the requests library: import requests url = "https://. execute-api. . amazonaws. com/prod/question?question=\"What is the color of my car now?\"&context=\"My car used to be blue but I painted red\"" response = requests. request("GET", url, headers=headers, data=payload) print(response. text) The code outputs a string similar to the following: ''{"score":0. 6947233080863953,"start":38,"end":41,"answer":"red"}'' If you are interested in knowing more about deploying Generative AI and large language models on AWS, check out here: Deploy Serverless Generative AI on AWS Lambda with OpenLLaMa Deploy large language models on AWS Inferentia2 using large model inference containers Clean up Inside the root directory of your repository, run the following code to clean up your resources: make destroy Conclusion In this post, we introduced how you can use Lambda to deploy your trained ML model using your preferred web application framework, such as FastAPI. We provided a detailed code repository that you can deploy, and you retain the flexibility of switching to whichever trained model artifacts you process. The performance can depend on how you implement and deploy the model. You are welcome to try it out yourself, and we’re excited to hear your feedback! About the Authors Tingyi Li is an Enterprise Solutions Architect from AWS based out in Stockholm, Sweden supporting the Nordics customers. She enjoys helping customers with the architecture, design, and development of cloud-optimized infrastructure solutions. She is specialized in AI and Machine Learning and is interested in empowering customers with intelligence in their AI/ML applications. In her spare time, she is also a part-time illustrator who writes novels and plays the piano. Demir Catovic is a Machine Learning Engineer from AWS based in Zurich, Switzerland. He engages with customers and helps them implement scalable and fully-functional ML applications. He is passionate about building and productionizing machine learning applications for customers and is always keen to explore around new trends and cutting-edge technologies in the AI/ML world. TAGS: Generative AI , Natural Language Processing Comments View Comments Resources Getting Started What''s New Blog Topics Amazon Comprehend Amazon Kendra Amazon Lex Amazon Polly Amazon Rekognition Amazon SageMaker Amazon Textract Follow Twitter Facebook LinkedIn Twitch Email Updates.' sentences: - How did ALTBalaji use AWS Elemental MediaLive to handle a tenfold increase in viewership during the live streaming of Lock Upp, and what insights did they gain from this experience? - How has PayEye been able to accelerate their development process and enter the production phase within a few months using AWS services, and what impact has this had on their recruitment efforts and team focus? - How can Lambda be used to deploy trained ML models using a preferred web application framework? model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.5120967741935484 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8266129032258065 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9233870967741935 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9637096774193549 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.5120967741935484 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2755376344086021 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.18467741935483872 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09637096774193549 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.5120967741935484 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8266129032258065 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9233870967741935 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9637096774193549 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7538879073840729 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.6844038018433181 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6858592666542238 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.532258064516129 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8225806451612904 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9193548387096774 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.967741935483871 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.532258064516129 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.27419354838709675 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.18387096774193548 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09677419354838711 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.532258064516129 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8225806451612904 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9193548387096774 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.967741935483871 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7596718979684643 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.6912602406554021 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6924236134719179 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.5241935483870968 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8225806451612904 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9193548387096774 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9596774193548387 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.5241935483870968 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.27419354838709675 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.1838709677419355 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.0959677419354839 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.5241935483870968 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8225806451612904 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9193548387096774 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9596774193548387 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7527772429981233 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.6846406169994881 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6862769216923534 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.4959677419354839 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.7903225806451613 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8911290322580645 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9556451612903226 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.4959677419354839 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.26344086021505375 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17822580645161293 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09556451612903227 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.4959677419354839 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.7903225806451613 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8911290322580645 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9556451612903226 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.73375586078758 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.6613495263696876 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6630698645438532 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.4475806451612903 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.7661290322580645 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8790322580645161 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9475806451612904 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.4475806451612903 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2553763440860215 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17580645161290326 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09475806451612903 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.4475806451612903 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.7661290322580645 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8790322580645161 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9475806451612904 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7052651530890945 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.6260768689196109 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6277483838406475 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("anishareddyalla/bge-base-matryoshka-aws-casestudies") # Run inference sentences = [ 'Alternatively, you can run the inference via code. Here is one example written in Python, using the requests library: import requests url = "https://. execute-api. . amazonaws. com/prod/question?question=\\"What is the color of my car now?\\"&context=\\"My car used to be blue but I painted red\\"" response = requests. request("GET", url, headers=headers, data=payload) print(response. text) The code outputs a string similar to the following: \'{"score":0. 6947233080863953,"start":38,"end":41,"answer":"red"}\' If you are interested in knowing more about deploying Generative AI and large language models on AWS, check out here: Deploy Serverless Generative AI on AWS Lambda with OpenLLaMa Deploy large language models on AWS Inferentia2 using large model inference containers Clean up Inside the root directory of your repository, run the following code to clean up your resources: make destroy Conclusion In this post, we introduced how you can use Lambda to deploy your trained ML model using your preferred web application framework, such as FastAPI. We provided a detailed code repository that you can deploy, and you retain the flexibility of switching to whichever trained model artifacts you process. The performance can depend on how you implement and deploy the model. You are welcome to try it out yourself, and we’re excited to hear your feedback! About the Authors Tingyi Li is an Enterprise Solutions Architect from AWS based out in Stockholm, Sweden supporting the Nordics customers. She enjoys helping customers with the architecture, design, and development of cloud-optimized infrastructure solutions. She is specialized in AI and Machine Learning and is interested in empowering customers with intelligence in their AI/ML applications. In her spare time, she is also a part-time illustrator who writes novels and plays the piano. Demir Catovic is a Machine Learning Engineer from AWS based in Zurich, Switzerland. He engages with customers and helps them implement scalable and fully-functional ML applications. He is passionate about building and productionizing machine learning applications for customers and is always keen to explore around new trends and cutting-edge technologies in the AI/ML world. TAGS: Generative AI , Natural Language Processing Comments View Comments Resources Getting Started What\'s New Blog Topics Amazon Comprehend Amazon Kendra Amazon Lex Amazon Polly Amazon Rekognition Amazon SageMaker Amazon Textract Follow Twitter Facebook LinkedIn Twitch Email Updates.', 'How can Lambda be used to deploy trained ML models using a preferred web application framework?', 'How has PayEye been able to accelerate their development process and enter the production phase within a few months using AWS services, and what impact has this had on their recruitment efforts and team focus?', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.5121 | | cosine_accuracy@3 | 0.8266 | | cosine_accuracy@5 | 0.9234 | | cosine_accuracy@10 | 0.9637 | | cosine_precision@1 | 0.5121 | | cosine_precision@3 | 0.2755 | | cosine_precision@5 | 0.1847 | | cosine_precision@10 | 0.0964 | | cosine_recall@1 | 0.5121 | | cosine_recall@3 | 0.8266 | | cosine_recall@5 | 0.9234 | | cosine_recall@10 | 0.9637 | | cosine_ndcg@10 | 0.7539 | | cosine_mrr@10 | 0.6844 | | **cosine_map@100** | **0.6859** | #### Information Retrieval * Dataset: `dim_512` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.5323 | | cosine_accuracy@3 | 0.8226 | | cosine_accuracy@5 | 0.9194 | | cosine_accuracy@10 | 0.9677 | | cosine_precision@1 | 0.5323 | | cosine_precision@3 | 0.2742 | | cosine_precision@5 | 0.1839 | | cosine_precision@10 | 0.0968 | | cosine_recall@1 | 0.5323 | | cosine_recall@3 | 0.8226 | | cosine_recall@5 | 0.9194 | | cosine_recall@10 | 0.9677 | | cosine_ndcg@10 | 0.7597 | | cosine_mrr@10 | 0.6913 | | **cosine_map@100** | **0.6924** | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.5242 | | cosine_accuracy@3 | 0.8226 | | cosine_accuracy@5 | 0.9194 | | cosine_accuracy@10 | 0.9597 | | cosine_precision@1 | 0.5242 | | cosine_precision@3 | 0.2742 | | cosine_precision@5 | 0.1839 | | cosine_precision@10 | 0.096 | | cosine_recall@1 | 0.5242 | | cosine_recall@3 | 0.8226 | | cosine_recall@5 | 0.9194 | | cosine_recall@10 | 0.9597 | | cosine_ndcg@10 | 0.7528 | | cosine_mrr@10 | 0.6846 | | **cosine_map@100** | **0.6863** | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.496 | | cosine_accuracy@3 | 0.7903 | | cosine_accuracy@5 | 0.8911 | | cosine_accuracy@10 | 0.9556 | | cosine_precision@1 | 0.496 | | cosine_precision@3 | 0.2634 | | cosine_precision@5 | 0.1782 | | cosine_precision@10 | 0.0956 | | cosine_recall@1 | 0.496 | | cosine_recall@3 | 0.7903 | | cosine_recall@5 | 0.8911 | | cosine_recall@10 | 0.9556 | | cosine_ndcg@10 | 0.7338 | | cosine_mrr@10 | 0.6613 | | **cosine_map@100** | **0.6631** | #### Information Retrieval * Dataset: `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.4476 | | cosine_accuracy@3 | 0.7661 | | cosine_accuracy@5 | 0.879 | | cosine_accuracy@10 | 0.9476 | | cosine_precision@1 | 0.4476 | | cosine_precision@3 | 0.2554 | | cosine_precision@5 | 0.1758 | | cosine_precision@10 | 0.0948 | | cosine_recall@1 | 0.4476 | | cosine_recall@3 | 0.7661 | | cosine_recall@5 | 0.879 | | cosine_recall@10 | 0.9476 | | cosine_ndcg@10 | 0.7053 | | cosine_mrr@10 | 0.6261 | | **cosine_map@100** | **0.6277** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 2,231 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details |
  • min: 3 tokens
  • mean: 430.06 tokens
  • max: 512 tokens
|
  • min: 7 tokens
  • mean: 33.49 tokens
  • max: 65 tokens
| * Samples: | positive | anchor | |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | TCSG is helping students enter a competitive workforce as educated cloud professionals and providing opportunities for success. TCSG built its Cloud Academy using AWS Academy, which provides higher education institutions with a free, ready-to-teach cloud computing curriculum that prepares students to pursue industry-recognized certifications and in-demand cloud jobs. TCSG launched the TCSG Cloud Academy in two forms: one as a specialization within an existing associate’s degree and the second as a stand-alone technical certificate of credit. For the technical certificate of credit, students who have existing degrees can enter the curriculum to focus on cloud computing and participate in hands-on cloud experiences using AWS services. Tiếng Việt Italiano ไทย The Technical College System of Georgia is the state government agency that supervises workforce development of more than 294,000 students across 22 technical colleges, 88 campuses, and more than 600 programs. Using the AWS curriculum and technology as the foundation for its courses, TCSG is preparing students to earn industry-recognized AWS Certifications to increase employability while improving accessibility to cloud education by offering the academy virtually and remotely. Learn more » TCSG is the state of Georgia government agency that supervises workforce development of hundreds of thousands of students across 22 technical colleges, 88 campuses, and more than 600 programs. The agency aims to run a system of technical education using the latest technology that’s accessible to all adults and corporate citizens in the state. To develop and deploy its new cloud-focused curriculum, it worked with AWS Education Programs, which helps TCSG institutions develop initiatives that align education to careers in the cloud and promote student employability, preparing diverse learners for in-demand cloud roles around the world. In 2020, the organization officially launched the TCSG Cloud Academy—a virtual program for students pursuing expertise and certifications in cloud computing—on its eCampus virtual learning system. Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today. Português. | How has the use of AWS Academy by TCSG helped prepare students for pursuing industry-recognized certifications and in-demand cloud jobs in Georgia's workforce? | | This prompt is then provided to the LLM for generating an answer to the user question. @router. post("/rag") async def rag_handler(req: Request) -> Dict[str, Any]: # dump the received request for debugging purposes logger. info(f"req={req}") # initialize vector db and SageMaker Endpoint _init(req) # Use the vector db to find similar documents to the query # the vector db call would automatically convert the query text # into embeddings docs = _vector_db. similarity_search(req. q, k=req. max_matching_docs) logger. info(f"here are the {req. max_matching_docs} closest matching docs to the query=\"{req. q}\"") for d in docs: logger. info(f"---------") logger. info(d) logger. info(f"---------") # now that we have the matching docs, lets pack them as a context # into the prompt and ask the LLM to generate a response prompt_template = """Answer based on context:\n\n{context}\n\n{question}""" prompt = PromptTemplate( template=prompt_template, input_variables=["context", "question"] ) logger. info(f"prompt sent to llm = \"{prompt}\"") chain = load_qa_chain(llm=_sm_llm, prompt=prompt) answer = chain({"input_documents": docs, "question": req. q}, return_only_outputs=True)['output_text'] logger. info(f"answer received from llm,\nquestion: \"{req. q}\"\nanswer: \"{answer}\"") resp = {'question': req. q, 'answer': answer} if req. verbose is True: resp['docs'] = docs return resp Clean up To avoid incurring future charges, delete the resources. You can do this by deleting the CloudFormation stack as shown in the following screenshot. | What resources need to be deleted to avoid future charges, and how can they be deleted? | | append(input_1_s3_location) async_response = base_model_predictor. predict_async(input_path=input_1_s3_location) output_locations. append(async_response. output_path) if i > max_images: break This may take up to 30 minutes or more depending on how much data you have uploaded for asynchronous inference. You can visualize one of these inferences as follows: plot_response('data/single. out') Convert the asynchronous inference output to a Ground Truth input manifest In this step, we create an input manifest for a bounding box verification job on Ground Truth. We upload the Ground Truth UI template and label categories file, and create the verification job. The notebook linked to this post uses a private workforce to perform the labeling; you can change this if you’re using other types of workforces. For more details, refer to the full code in the notebook. Verify labels from the auto-labeling process in Ground Truth In this step, we complete the verification by accessing the labeling portal. For more details, refer to here. When you access the portal as a workforce member, you will be able to see the bounding boxes created by the JumpStart model and make adjustments as required. You can use this template to repeat auto-labeling with many task-specific models, potentially merge labels, and use the resulting labeled dataset in downstream tasks. Clean up In this step, we clean up by deleting the endpoint and the model created in previous steps: # Delete the SageMaker endpoint base_model_predictor. delete_model() base_model_predictor. delete_endpoint() Conclusion In this post, we walked through an auto-labeling process involving JumpStart and asynchronous inference. We used the results of the auto-labeling process to convert and visualize labeled data on a real-world dataset. You can use the solution to perform auto-labeling with many task-specific models, potentially merge labels, and use the resulting labeled dataset in downstream tasks. You can also explore using tools like the Segment Anything Model for generating segment masks as part of the auto-labeling process. In future posts in this series, we will cover the perception module and segmentation. | How can you visualize the inferences generated by the asynchronous inference process using the provided solution? | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 | |:----------:|:-----:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:| | 0.9143 | 4 | - | 0.6663 | 0.6851 | 0.7027 | 0.6120 | 0.6998 | | **1.8286** | **8** | **-** | **0.6758** | **0.6822** | **0.6966** | **0.6311** | **0.6941** | | 2.2857 | 10 | 1.883 | - | - | - | - | - | | 2.9714 | 13 | - | 0.6631 | 0.6881 | 0.6904 | 0.6245 | 0.6873 | | 3.6571 | 16 | - | 0.6631 | 0.6863 | 0.6924 | 0.6277 | 0.6859 | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.0.1 - Transformers: 4.42.4 - PyTorch: 2.3.1+cu121 - Accelerate: 0.32.1 - Datasets: 2.20.0 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```