Felladrin/Llama-68M-Chat-v1-GGUF

Quantized GGUF model files for Llama-68M-Chat-v1 from Felladrin

Name	Quant method	Size
llama-68m-chat-v1.fp16.gguf	fp16	136.79 MB
llama-68m-chat-v1.q2_k.gguf	q2_k	35.88 MB
llama-68m-chat-v1.q3_k_m.gguf	q3_k_m	40.66 MB
llama-68m-chat-v1.q4_k_m.gguf	q4_k_m	46.10 MB
llama-68m-chat-v1.q5_k_m.gguf	q5_k_m	51.16 MB
llama-68m-chat-v1.q6_k.gguf	q6_k	56.54 MB
llama-68m-chat-v1.q8_0.gguf	q8_0	73.02 MB

Original Model Card:

A Llama Chat Model of 68M Parameters

• Base model: JackFram/llama-68m
• Datasets:
  • THUDM/webglm-qa
  • databricks/databricks-dolly-15k
  • cognitivecomputations/wizard_vicuna_70k_unfiltered
  • totally-not-an-llm/EverythingLM-data-V3
  • Amod/mental_health_counseling_conversations
  • sablo/oasst2_curated
  • starfishmedical/webGPT_x_dolly
  • Open-Orca/OpenOrca
  • mlabonne/chatml_dpo_pairs
• Availability in other ML formats:
  • ONNX: Felladrin/onnx-Llama-68M-Chat-v1

Recommended Prompt Format

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Recommended Inference Parameters

penalty_alpha: 0.5
top_k: 4