metadata

language:
  - en
inference: false
tags:
  - text-classification
  - onnx
  - int8
  - optimum
  - multi-class-classification
  - ONNXRuntime
license: apache-2.0

LLM user flow classification

This model identifies common events and patterns within the conversation flow. Such events include, for example, complaint, when a user expresses dissatisfaction. The flow labels can serve as foundational elements for sophisticated LLM analytics.

It is ONNX quantized and is a fined-tune of MiniLMv2-L6-H384. The base model can be found here

This model is used only for the user texts. For the LLM texts in the dialog use this agent model.

Optimum

Installation

Install from source:

python -m pip install optimum[onnxruntime]@git+https://github.com/huggingface/optimum.git

Run the Model

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

model = ORTModelForSequenceClassification.from_pretrained('minuva/MiniLMv2-userflow-v2-onnx', provider="CPUExecutionProvider")
tokenizer = AutoTokenizer.from_pretrained('minuva/MiniLMv2-userflow-v2-onnx', use_fast=True, model_max_length=256, truncation=True, padding='max_length')

pipe = pipeline(task='text-classification', model=model, tokenizer=tokenizer, )
texts = ["that's wrong", "can you please answer me?"]
pipe(texts)
# [{'label': 'model_wrong_or_try_again', 'score': 0.9737648367881775},
# {'label': 'user_wants_agent_to_answer', 'score': 0.9105103015899658}]

ONNX Runtime only

A lighter solution for deployment

Installation

pip install tokenizers
pip install onnxruntime
git clone https://huggingface.co/minuva/MiniLMv2-userflow-v2-onnx

Run the Model

import os
import numpy as np
import json

from tokenizers import Tokenizer
from onnxruntime import InferenceSession


model_name = "minuva/MiniLMv2-userflow-v2-onnx"

tokenizer = Tokenizer.from_pretrained(model_name)
tokenizer.enable_padding(
    pad_token="<pad>",
    pad_id=1,
)
tokenizer.enable_truncation(max_length=256)
batch_size = 16

texts = ["that's wrong", "can you please answer me?"]


outputs = []
model = InferenceSession("MiniLMv2-userflow-v2-onnx/model_optimized_quantized.onnx", providers=['CPUExecutionProvider'])

with open(os.path.join("MiniLMv2-userflow-v2-onnx", "config.json"), "r") as f:
            config = json.load(f)

output_names = [output.name for output in model.get_outputs()]
input_names = [input.name for input in model.get_inputs()]

for subtexts in np.array_split(np.array(texts), len(texts) // batch_size + 1):
            encodings = tokenizer.encode_batch(list(subtexts))
            inputs = {
                "input_ids": np.vstack(
                    [encoding.ids for encoding in encodings],
                ),
                "attention_mask": np.vstack(
                    [encoding.attention_mask for encoding in encodings],
                ),
                "token_type_ids": np.vstack(
                    [encoding.type_ids for encoding in encodings],
                ),
            }

            for input_name in input_names:
                if input_name not in inputs:
                    raise ValueError(f"Input name {input_name} not found in inputs")

            inputs = {input_name: inputs[input_name] for input_name in input_names}
            output = np.squeeze(
                np.stack(
                    model.run(output_names=output_names, input_feed=inputs)
                ),
                axis=0,
            )
            outputs.append(output)

outputs = np.concatenate(outputs, axis=0)
scores = 1 / (1 + np.exp(-outputs))
results = []
for item in scores:
    labels = []
    scores = []
    for idx, s in enumerate(item):
        labels.append(config["id2label"][str(idx)])
        scores.append(float(s))
    results.append({"labels": labels, "scores": scores})


res = []

for result in results:
    joined = list(zip(result['labels'], result['scores']))
    max_score = max(joined, key=lambda x: x[1])    
    res.append(max_score)

res
#[('model_wrong_or_try_again', 0.9737648367881775),
# ('user_wants_agent_to_answer', 0.9105103015899658)]

Categories Explanation

Click to expand!

OTHER: Responses that do not fit into any predefined categories or are outside the scope of the specific interaction types listed.
agrees_praising_thanking: When the user agrees with the provided information, offers praise, or expresses gratitude.
asks_source: The user requests the source of the information or the basis for the answer provided.
continue: Indicates a prompt for the conversation to proceed or continue without a specific directional change.
continue_or_finnish_code: Signals either to continue with the current line of discussion or code execution, or to conclude it.
improve_or_modify_answer: The user requests an improvement or modification to the provided answer.
lack_of_understandment: Reflects the user's or agent confusion or lack of understanding regarding the information provided.
model_wrong_or_try_again: Indicates that the model's response was incorrect or unsatisfactory, suggesting a need to attempt another answer.
more_listing_or_expand: The user requests further elaboration, expansion from the given list by the agent.
repeat_answers_or_question: The need to reiterate a previous answer or question.
request_example: The user asks for examples to better understand the concept or answer provided.
user_complains_repetition: The user notes that the information or responses are repetitive, indicating a need for new or different content.
user_doubts_answer: The user expresses skepticism or doubt regarding the accuracy or validity of the provided answer.
user_goodbye: The user says goodbye to the agent.
user_reminds_question: The user reiterates the question.
user_wants_agent_to_answer: The user explicitly requests a response from the agent, when the agent refuses to do so.
user_wants_explanation: The user seeks an explanation behind the information or answer provided.
user_wants_more_detail: Indicates the user's desire for more comprehensive or detailed information on the topic.
user_wants_shorter_longer_answer: The user requests that the answer be condensed or expanded to better meet their informational needs.
user_wants_simplier_explanation: The user seeks a simpler, more easily understood explanation.
user_wants_yes_or_no: The user is asking for a straightforward affirmative or negative answer, without additional detail or explanation.

Metrics in our private test dataset

Model (params)	Loss	Accuracy	F1
minuva/MiniLMv2-userflow-v2 (33M)	0.6738	0.7236	0.7313
minuva/MiniLMv2-userflow-v2-onnx (33M)	-	0.7195	0.7189

Deployment

Check our llm-flow-classification repository for a FastAPI and ONNX based server to deploy this model on CPU devices.