Aryanne/Astrohermes-3B-GGUF

Quantized GGUF model files for Astrohermes-3B from Aryanne

Name Quant method Size
astrohermes-3b.fp16.gguf fp16 5.59 GB
astrohermes-3b.q2_k.gguf q2_k 1.20 GB
astrohermes-3b.q3_k_m.gguf q3_k_m 1.39 GB
astrohermes-3b.q4_k_m.gguf q4_k_m 1.71 GB
astrohermes-3b.q5_k_m.gguf q5_k_m 1.99 GB
astrohermes-3b.q6_k.gguf q6_k 2.30 GB
astrohermes-3b.q8_0.gguf q8_0 2.97 GB

Original Model Card:

This model is a mix of PAIXAI/Astrid-3B + jondurbin/airoboros-3b-3p0 + cxllin/StableHermes-3b, as shown in the yaml(see Astrohermes.yml or below). Aryanne/Astridboros-3B = PAIXAI/Astrid-3B + jondurbin/airoboros-3b-3p0

slices:
  - sources:
    - model: Aryanne/Astridboros-3B
      layer_range: [0, 15]
  - sources:
    - model: cxllin/StableHermes-3b
      layer_range: [15, 16]
  - sources:
    - model: Aryanne/Astridboros-3B
      layer_range: [16, 17]
  - sources:
    - model: cxllin/StableHermes-3b
      layer_range: [17, 18]
  - sources:
    - model: Aryanne/Astridboros-3B
      layer_range: [18, 19]
  - sources:
    - model: cxllin/StableHermes-3b
      layer_range: [19, 20]
  - sources:
    - model: Aryanne/Astridboros-3B
      layer_range: [20, 21]
  - sources:
    - model: cxllin/StableHermes-3b
      layer_range: [21, 22]
  - sources:
    - model: Aryanne/Astridboros-3B
      layer_range: [22, 23]
  - sources:
    - model: cxllin/StableHermes-3b
      layer_range: [23, 24]
  - sources:
    - model: Aryanne/Astridboros-3B
      layer_range: [24, 32]
merge_method: passthrough
dtype: float16
Downloads last month
17
GGUF
Model size
2.8B params
Architecture
stablelm

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Examples
Inference API (serverless) has been turned off for this model.

Model tree for afrideva/Astrohermes-3B-GGUF

Quantized
(1)
this model