Error might be related to Obabagoo and not the model.

Did you find a fix for this error? I'm seeing the exact same error (yes Im using Oobabooga as the Web UI) when launching without specifying wbits, group size (python server.py --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --chat --gpu-memory 4294967296 --cpu-memory 10GB --auto-devices --verbose)? When I do use wbits, groupsize args I get outofmemory error (running 3060Ti 8GB VRAM)

seeing this too

It is fixed here but I am not sure which steps actually solved it and if the problem is related to this model or Oobabooga.

You can try to download this model by using the download-model.bat

1. Choose option L None of the above
2. Input anon8231489123/vicuna-13b-GPTQ-4bit-128g

This is still a bug and has not been resolved.

@CR2022 -- you're likely referring to the guide posted by troublechute (https://youtu.be/ByV5w1ES38A), that is what I used and I did choose L and input the anon8231489123/vicuna-13b-GPTQ-4bit-128g and if you run the server.py file with the default args, for me atleast it fails and complains of OOM [torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 8.00 GiB total capacity; 7.08 GiB already allocated; 0 bytes free; 7.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Output generated in 3.30 seconds (0.00 tokens/s, 0 tokens, context 43)]

Just here to share I have the same issue, and using download-model.bat with 'L' and specifying anon8231489123/vicuna-13b-GPTQ-4bit-128g (which does download) did not resolve the issue when choosing the model, you get the same type of message.

FileNotFoundError: [Errno 2] No such file or directory: 'models\\anon8231489123_vicuna-13b-GPTQ-4bit-128g\\pytorch_model-00001-of-00003.bin'

It seems like in this process we should find a set of .bin files which appear to be a memory usage limiting option which splits the model files out into smaller partitions based on this link https://huggingface.co/docs/transformers/big_models

the file pytorch_model.bin.index.json refers to a set of entries like below, but in my models\anon8231489123_vicuna-13b-GPTQ-4bit-128g folder for example, I do not find these .bin files

    "lm_head.weight": "pytorch_model-00003-of-00003.bin",
    "model.embed_tokens.weight": "pytorch_model-00001-of-00003.bin",
    "model.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00003.bin",
    "model.layers.0.mlp.down_proj.weight": "pytorch_model-00001-of-00003.bin",

Same here.

FileNotFoundError: [Errno 2] No such file or directory: 'models\anon8231489123_vicuna-13b-GPTQ-4bit-128g\pytorch_model-00001-of-00003.bin'

I found the issue and perhaps not the best "fix", because it requires a lot of extra space.. but it appears that the script is looking for the original "vicuna-13b-delta-v0" that "anon8231489123_vicuna-13b-GPTQ-4bit-128g" was based on. It contained "pytorch_model-0000X-of-00003.bin" and I just downloaded that repo and created symbolic links (so both share the same files). It takes a while to cache/load but it works now. Pardon my ignorance, I'm likely doing something wrong but it doesn't feel like the extra 24gb of data shouldn't be needed. Thanks

Just want to share that this issue is now gone for me. Yesterday I downloaded the https://github.com/oobabooga/text-generation-webui both oobabooga-windows.zip, oobabooga-linux.zip (for WSL as an experiment) and just run the start script once or twice, and when time to pick a model, say "L" and specify anon8231489123/vicuna-13b-GPTQ-4bit-128g and wait for the site to start.

Both in Windows and WSL the packages worked first time through.

When running this in Windows WSL Linux with Text Generation Web UI -- I get "No module named ‘llama_inference_offload'" error - this was a clean install of oobabooga-linux manual install - second release

I got help on reddit -

cd text-generation-webui and then:

python server.py --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type llama

This worked for me. Note: I also installed the GPTQ conversion repository - I don't know if that helped. I would tri the above command first.

hy Yall,

Getting this result after placing the previous command
"python server.py --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type llama"

C:\AIProject\oobabooga_windows\text-generation-webui>python server.py --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type llama
Traceback (most recent call last):
File "C:\AIProject\oobabooga_windows\text-generation-webui\server.py", line 17, in
import gradio as gr
ModuleNotFoundError: No module named 'gradio'

C:\AIProject\oobabooga_windows\text-generation-webui>

What else could I try to get this up and running? ANy thought?

hy Yall,

Getting this result after placing the previous command
"python server.py --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type llama"

C:\AIProject\oobabooga_windows\text-generation-webui>python server.py --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type llama
Traceback (most recent call last):
File "C:\AIProject\oobabooga_windows\text-generation-webui\server.py", line 17, in
import gradio as gr
ModuleNotFoundError: No module named 'gradio'

C:\AIProject\oobabooga_windows\text-generation-webui>

What else could I try to get this up and running? ANy thought?

Run pip install -r requirements.txt inside\text-generation-webui and try again

I follow the One-click installers and run the start_windows.bat.
And then download the anon8231489123/vicuna-13b-GPTQ-4bit-128g model by using web UI.
But I got the same error after loading this model.
Is there any solution or suggestion?