|
# Addition from the Hugging Face team |
|
|
|
> [!IMPORTANT] |
|
> |
|
> This repository corresponds to the original Llama format and codebase, not the transformers library |
|
|
|
The weights were originally distributed in the following format: |
|
|
|
``` |
|
weights/ |
|
consolidated.00.pth |
|
consolidated.01.pth |
|
... |
|
``` |
|
|
|
Unfortunately, the files themselves were too large for the Hub to handle, so we had to shard them. |
|
In order to keep the same structure that was originally given, the sharding is done as follows: |
|
|
|
|
|
``` |
|
weights/ |
|
consolidated.00/ |
|
consolidated-00001-of-00011.pth |
|
consolidated-00002-of-00011.pth |
|
... |
|
consolidated.01/ |
|
consolidated-00001-of-00011.pth |
|
consolidated-00002-of-00011.pth |
|
... |
|
... |
|
``` |
|
|
|
If trying to run the code that was given with the original weights, we recommend running this script to join the files together once again: |
|
|
|
|
|
```py |
|
import os |
|
import torch |
|
|
|
from pathlib import Path |
|
|
|
path_to_files = Path('.../weights') |
|
|
|
folders = [folder for folder in os.listdir(path_to_files) if os.path.isdir(path_to_files / folder) and folder.startswith('consolidated')] |
|
for folder in folders: |
|
state_dict = {} |
|
files = [file for file in os.listdir(path_to_files / folder) if file.endswith(".pth")] |
|
|
|
for file in files: |
|
state_dict_partial = torch.load(path_to_files / folder / file, map_location="cpu") |
|
for key, value in state_dict_partial.items(): |
|
state_dict[key]=value |
|
|
|
torch.save(state_dict, path_to_files / f"{folder}.pth") |
|
``` |
|
|