{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿš€ Welcome to the Hugging Face Uploader Notebook!\n", "\n", "This notebook provides a user-friendly tool for uploading files directly to your Hugging Face repositories. Here's how to get started:\n", "\n", "**๐Ÿ”‘ Initial Setup (One Time Only)**\n", "\n", "1. **Authenticate:**\n", " * Run the `notebook_login()` cell *once* to securely store your Hugging Face API token.\n", " * **Important:** For security, avoid sharing your notebook file or system state after you have run `notebook_login()`. Do not commit your notebook file to a shared repo, as this could expose your API token.\n", "\n", "**๐Ÿ—‚๏ธ Using the Uploader**\n", "\n", "1. **Repository Details:**\n", " * Enter your Hugging Face Organization or Username in the \"Owner\" field.\n", " * Enter your repository name in the \"Repo\" field.\n", "\n", "2. **Directory Selection:**\n", " * Enter a directory path where your files are located.\n", " * Select the 'Update Dir' button to set that path.\n", "\n", "3. **File Selection:**\n", " * Select the appropriate file type from the dropdown menu.\n", " * Select the files you want to upload from the list. You can sort them by name or date modified.\n", "\n", "4. **Commit Message (Optional):**\n", " * Add a commit message to your upload, or use the default message.\n", "\n", "5. **Upload Options:**\n", " * Choose whether to create a pull request or upload directly to the main branch.\n", " * Select whether to clear the output after a successful upload.\n", "\n", "6. **Start Upload:**\n", " * Click the \"โฌ†๏ธ Upload\" button.\n", "\n", "**๐Ÿ’ก Important Notes**\n", "\n", "* **Direct Uploads:** This uploader uses the Hugging Face API for direct file uploads, bypassing the need for traditional Git operations for core functionality.\n", "* **Git LFS:** Most users will not need to interact with Git or Git LFS. If you need to clone or push changes to a repository *outside this notebook*, you will require a separate Git credential setup (e.g., your operating system's credential manager or SSH keys). This is separate from your Hugging Face API token, and you should not share any git credentials in your notebook.\n", "* **Troubleshooting:** If you encounter any issues, please review the steps, or double-check that you have write access to the repository, and that your API token has the correct scope of access.\n", "\n", "**๐Ÿ“ฃ Updates & Community**\n", "\n", "* This tool will continue to be updated.\n", "* For the latest patches, fixes, and community contributions, visit [https://github.com/duskfallcrew/HuggingFace_Backup](https://github.com/duskfallcrew/HuggingFace_Backup)\n", "\n", "We hope this notebook makes your Hugging Face uploads easier! If you have any questions or suggestions, please reach out." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "!pip install -q huggingface_hub ipywidgets tqdm" ] }, { "cell_type": "markdown", "metadata": { "id": "Xs1mb1VKLuUW" }, "source": [ "# โœจ Connecting to Hugging Face: Authentication Required\n", "\n", "To upload files to Hugging Face using this notebook, you'll need to authenticate using your Hugging Face API token. This token acts as your secure access key to the Hugging Face Hub.\n", "\n", "**๐Ÿ”‘ Obtaining Your API Token**\n", "\n", "1. **Go to Your Settings:** Navigate to your [Hugging Face account settings](https://huggingface.co/settings/tokens).\n", "2. **Access Tokens Section:** Find the \"Access Tokens\" section.\n", "3. **Create or Copy Token:**\n", " * If you don't have one already, create a new access token.\n", " * Copy your existing access token.\n", "\n", "**๐Ÿ”’ Logging In**\n", "\n", "1. **Run the Cell:** Execute the cell below this section.\n", "2. **Paste Token:** When prompted, paste your Hugging Face API token into the input field.\n", "3. **Confirm:** Press `Enter` to complete the login process. No characters will be displayed as you type your token.\n", " * **Security Note:** This ensures your token is kept private while pasting.\n", "\n", "**Important Notes**\n", "\n", "* **Purpose:** This login step authorizes your notebook environment to interact with the Hugging Face Hub. This is required for uploading your models and data.\n", "* **Session-Based:** You will need to perform this login procedure each time you restart the notebook or start a new session (the token is stored locally on your computer, but not in the notebook, and it doesn't persist between sessions).\n", "* **Security:** After you have logged in using `notebook_login()`, **do not** share your notebook or your system state with untrusted users, and do not commit your notebook to a shared repository, as this could compromise your API token.\n", "\n", "After completing this step, you'll be ready to use the upload functionality of this notebook." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b8aa868174ad469e86777458ea77c214", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
๐Ÿ“š Repository Details'), HBox(children=(Text(value='', description='Owner:', โ€ฆ" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import glob\n", "import os\n", "from pathlib import Path\n", "from huggingface_hub import HfApi\n", "from ipywidgets import Text, Dropdown, Button, SelectMultiple, VBox, HBox, Output, Layout, Checkbox, HTML, Textarea, Label, IntProgress\n", "from IPython.display import display, clear_output\n", "import time\n", "from tqdm import tqdm\n", "\n", "class HuggingFaceUploader:\n", " def __init__(self):\n", " self.api = HfApi()\n", " self.file_types = [\n", " # AI Model Files ๐Ÿค–\n", " ('SafeTensors', 'safetensors'),\n", " ('PyTorch Models', 'pt'),\n", " ('PyTorch Legacy', 'pth'),\n", " ('ONNX Models', 'onnx'),\n", " ('TensorFlow Models', 'pb'),\n", " ('Keras Models', 'h5'),\n", "\n", " # Checkpoint Files ๐ŸŽฏ\n", " ('Checkpoints', 'ckpt'),\n", " ('Binary Files', 'bin'),\n", "\n", " # Config & Data Files ๐Ÿ“\n", " ('JSON Files', 'json'),\n", " ('YAML Files', 'yaml'),\n", " ('YAML Alt', 'yml'),\n", " ('Text Files', 'txt'),\n", " ('CSV Files', 'csv'),\n", " ('Pickle Files', 'pkl'),\n", "\n", " # Image Files ๐ŸŽจ\n", " ('PNG Images', 'png'),\n", " ('JPEG Images', 'jpg'),\n", " ('JPEG Alt', 'jpeg'),\n", " ('WebP Images', 'webp'),\n", " ('GIF Images', 'gif'),\n", "\n", " # Archive Files ๐Ÿ“ฆ\n", " ('ZIP Archives', 'zip'),\n", " ('TAR Files', 'tar'),\n", " ('GZ Archives', 'gz')\n", " ]\n", " self.current_directory = os.getcwd()\n", " self._create_widgets()\n", " self._bind_events()\n", "\n", "\n", " def _create_widgets(self):\n", " # Repository info section\n", " self.repo_info = HTML(value=\"๐Ÿ“š Repository Details\")\n", " self.org_name = Text(\n", " placeholder='Organization or Username',\n", " description='Owner:',\n", " style={'description_width': 'initial'}\n", " )\n", " self.repo_name = Text(\n", " placeholder='Repository Name',\n", " description='Repo:',\n", " style={'description_width': 'initial'}\n", " )\n", "\n", " # File handling section\n", " self.file_section = HTML(value=\"๐Ÿ—‚๏ธ File Selection\")\n", " self.file_type = Dropdown(\n", " options=self.file_types,\n", " value='safetensors',\n", " description='File Type:',\n", " style={'description_width': 'initial'}\n", " )\n", " self.sort_by = Dropdown(\n", " options=['name', 'date'],\n", " value='name',\n", " description='Sort By:'\n", " )\n", " self.directory_label = Label(value=f\"Current Directory: {self.current_directory}\")\n", " self.directory_text = Text(\n", " value=self.current_directory,\n", " description=\"Path:\",\n", " style={'description_width': 'initial'},\n", " layout=Layout(width=\"400px\")\n", " )\n", " self.directory_update_btn = Button(\n", " description='๐Ÿ”„ Update Dir',\n", " button_style='info',\n", " tooltip='Refresh directory'\n", " )\n", "\n", "\n", " # Custom commit message\n", " self.commit_section = HTML(value=\"๐Ÿ’ญ Commit Details\")\n", " self.commit_msg = Textarea(\n", " value=\"Upload with Earth & Dusk Huggingface ๐Ÿค— Backup\",\n", " placeholder='Enter your commit message (optional)',\n", " description='Message:',\n", " layout=Layout(width='400px', height='60px')\n", " )\n", "\n", " # Upload options section\n", " self.upload_section = HTML(value=\"๐Ÿš€ Upload Settings\")\n", " self.create_pr = Checkbox(\n", " value=False,\n", " description='Create Pull Request',\n", " indent=False,\n", " tooltip='When checked, creates a Pull Request instead of direct upload'\n", " )\n", " self.clear_after = Checkbox(\n", " value=True,\n", " description='Clear output after upload',\n", " indent=False,\n", " tooltip='Clears output area after successful upload'\n", " )\n", "\n", " # Progress tracking\n", " self.progress_text = HTML(value=\"\")\n", "\n", " # Action buttons with style\n", " self.update_btn = Button(\n", " description='๐Ÿ”„ Update Files',\n", " button_style='info',\n", " tooltip='Refresh file list'\n", " )\n", " self.upload_btn = Button(\n", " description='โฌ†๏ธ Upload',\n", " button_style='success',\n", " tooltip='Start upload process',\n", " layout=Layout(width='auto', height='auto'),\n", " )\n", " self.clear_btn = Button(\n", " description='๐Ÿงน Clear Output',\n", " button_style='warning',\n", " tooltip='Clear output area'\n", " )\n", "\n", " # File selector and output\n", " self.ckpt_picker = SelectMultiple(\n", " options=[],\n", " layout=Layout(width=\"600px\", height=\"200px\")\n", " )\n", " self.out = Output(layout=Layout(padding='10px'))\n", " \n", " # Create debug output:\n", " self.debug_output = Output()\n", " self.debug_box = VBox([HTML(value=\"Debug Output\"), self.debug_output],\n", " layout=Layout(border='2px solid #2ecc71', padding='10px', margin_top='10px'))\n", " self.debug_toggle = Checkbox(\n", " value=False,\n", " description='Show Debug',\n", " indent=False,\n", " tooltip='Show debug information'\n", " )\n", " self.debug_box.layout.visibility = 'hidden'\n", "\n", "\n", " def _bind_events(self):\n", " self.update_btn.on_click(self._update_files)\n", " self.upload_btn.on_click(self._upload_ckpts)\n", " self.clear_btn.on_click(lambda _: self.out.clear_output())\n", " self.file_type.observe(self._update_files, names='value')\n", " self.directory_update_btn.on_click(self._update_directory)\n", " self.debug_toggle.observe(self._toggle_debug, names='value')\n", "\n", " def _toggle_debug(self,_):\n", " if self.debug_toggle.value == True:\n", " self.debug_box.layout.visibility = 'visible'\n", " else:\n", " self.debug_box.layout.visibility = 'hidden'\n", "\n", "\n", " def _update_directory(self,_):\n", " new_dir = self.directory_text.value\n", " if os.path.isdir(new_dir):\n", " self.current_directory = new_dir\n", " self.directory_label.value = f\"Current Directory: {self.current_directory}\"\n", " self._update_files(None)\n", " else:\n", " with self.out:\n", " print(\"โŒ Invalid Directory\")\n", "\n", " def _update_files(self, _):\n", " file_extension = self.file_type.value\n", " try:\n", " # Glob files based on file extension\n", " all_files = glob.glob(os.path.join(self.current_directory, f\"*.{file_extension}\"))\n", "\n", " with self.debug_output:\n", " print(f\"DEBUG _update_files: File type is {file_extension}\")\n", " print(f\"DEBUG _update_files: All files found by glob: {all_files}\")\n", "\n", " # Filter out symlinks, old files and ignore patterns\n", " filtered_files = []\n", " for file_path in all_files:\n", " if os.path.islink(file_path):\n", " with self.debug_output:\n", " print(f\"DEBUG _update_files: Skipping symlink {file_path}\")\n", " continue\n", " if not os.path.isfile(file_path):\n", " with self.debug_output:\n", " print(f\"DEBUG _update_files: Skipping non file {file_path}\")\n", " continue\n", "\n", " filtered_files.append(file_path)\n", " \n", " # Sort the files\n", " all_ckpts = sorted(\n", " filtered_files,\n", " key=os.path.getmtime if self.sort_by.value == 'date' else str\n", " )\n", "\n", " self.ckpt_picker.options = all_ckpts\n", "\n", " with self.out:\n", " print(f\"โœจ Found {len(all_ckpts)} {file_extension} files in {self.current_directory}\")\n", "\n", " except Exception as e:\n", " with self.out:\n", " print(f\"โŒ Error listing files: {str(e)}\")\n", "\n", "\n", " def _format_size(self, size):\n", " for unit in ['B', 'KB', 'MB', 'GB']:\n", " if size < 1024:\n", " return f\"{size:.2f} {unit}\"\n", " size /= 1024\n", " return f\"{size:.2f} TB\"\n", "\n", " def _print_file_info(self, file_path, file_size, index, total):\n", " with self.out:\n", " print(f\"๐Ÿ“ฆ File {index}/{total}: {file_path} ({self._format_size(file_size)})\")\n", "\n", "\n", " def _upload_ckpts(self, _):\n", " if not self.org_name.value or not self.repo_name.value:\n", " with self.out:\n", " print(\"โ— Please fill in both Organization/Username and Repository name\")\n", " return\n", "\n", " repo_id = f\"{self.org_name.value}/{self.repo_name.value}\"\n", " selected_files = self.ckpt_picker.value\n", " total_files = len(selected_files)\n", "\n", " with self.out:\n", " if not selected_files:\n", " print(\"๐Ÿ“ Nothing selected for upload. Please select files from the list.\")\n", " return\n", "\n", " print(f\"๐ŸŽฏ Starting upload to: huggingface.co/{repo_id}\")\n", "\n", " commit_msg = self.commit_msg.value if self.commit_msg.value else \"Uploaded with Earth & Dusk Huggingface ๐Ÿค— Backup\"\n", "\n", " with self.debug_output:\n", " print(f\"DEBUG TQDM: Is TQDM working? This should be above the progress bar.\")\n", "\n", " progress_bar = IntProgress( # Create a progress bar for this upload\n", " value=0,\n", " min=0,\n", " max=total_files,\n", " description='Upload Progress:',\n", " style={'description_width': 'initial'},\n", " layout=Layout(width=\"400px\"),\n", " )\n", " display(progress_bar) # Display progress bar\n", "\n", " for idx, ckpt in enumerate(selected_files, 1):\n", " try:\n", " file_size = os.path.getsize(ckpt)\n", " self._print_file_info(ckpt, file_size, idx, total_files)\n", " start_time = time.time()\n", " with self.debug_output:\n", " print(f\"๐Ÿš€ Attempting upload of {ckpt}\")\n", " response = self.api.upload_file(\n", " path_or_fileobj=ckpt,\n", " path_in_repo=os.path.basename(ckpt),\n", " repo_id=repo_id,\n", " repo_type=None,\n", " create_pr=self.create_pr.value,\n", " commit_message=commit_msg\n", " )\n", " with self.debug_output:\n", " print(f\"โœ… Upload Response: {response}\")\n", " duration = time.time() - start_time\n", " with self.out:\n", " print(f\"โœ… Upload completed in {duration:.1f} seconds\")\n", " display(response)\n", "\n", "\n", " progress_bar.value = idx\n", " display(progress_bar) # Force a render of the progress bar\n", " except Exception as e:\n", " with self.out:\n", " print(f\"โŒ Error uploading {ckpt}: {str(e)}\")\n", " continue\n", "\n", "\n", " with self.out:\n", " print(\"\\nโœจ All uploads completed! โœจ\")\n", " if self.create_pr.value:\n", " print(\"๐ŸŽ‰ Check your repository for the new Pull Request!\")\n", " else:\n", " print(\"๐ŸŽ‰ Files have been uploaded directly to your repository!\")\n", "\n", " if self.clear_after.value:\n", " time.sleep(3)\n", " self.out.clear_output()\n", "\n", " def display(self):\n", " box = VBox([\n", " self.repo_info,\n", " HBox([self.org_name, self.repo_name]),\n", " self.file_section,\n", " HBox([self.file_type, self.sort_by]),\n", " HBox([self.directory_label, self.directory_text, self.directory_update_btn]),\n", " self.commit_section,\n", " self.commit_msg,\n", " self.upload_section,\n", " HBox([self.create_pr, self.clear_after]),\n", " self.update_btn,\n", " self.ckpt_picker,\n", " HBox([self.upload_btn, self.clear_btn, self.debug_toggle], layout=Layout(align_items='center')),\n", " self.progress_text,\n", " self.out,\n", " self.debug_box,\n", " ])\n", " display(box)\n", "\n", "# Create and display the uploader - just one line to rule them all! โœจ\n", "uploader = HuggingFaceUploader()\n", "uploader.display()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "colab": { "collapsed_sections": [ "IZ_JYwvBLrg-", "PNF2kdyeO3Dn" ], "private_outputs": true, "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 4 }