Upload 140 files

.gitattributes

@@ -1,2 +1,18 @@
 /web/assets/** linguist-generated
 /web/** linguist-vendored
+custom_nodes/ComfyUI-N-Nodes/img/image-5.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/img/image-6.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/img/image-7.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/assets/demo-1.jpg filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/assets/demo-2.jpg filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/moondream/assets/demo-1.jpg filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/moondream/assets/demo-2.jpg filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/I0_0.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/I0_1.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/I0_slomo_clipped.gif filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/i0.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/i1.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/I2_0.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/I2_1.png filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/demo/I2_slomo_clipped.gif filter=lfs diff=lfs merge=lfs -text
+custom_nodes/ComfyUI-N-Nodes/libs/rifle/train_log/flownet.pkl filter=lfs diff=lfs merge=lfs -text

custom_nodes/ComfyUI-N-Nodes/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,37 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Full log**
+This is MANDATORY. By log I mean all the text in the console from the time ComfyUI was started until the time of the reported bug.
+>>Bug reports that do not have this log will be closed.<<
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+
+**Additional context**
+Add any other context about the problem here.

custom_nodes/ComfyUI-N-Nodes/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

custom_nodes/ComfyUI-N-Nodes/.github/workflows/publish.yml

@@ -0,0 +1,21 @@
+name: Publish to Comfy registry
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - "pyproject.toml"
+
+jobs:
+  publish-node:
+    name: Publish Custom Node to registry
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v4
+      - name: Publish Custom Node
+        uses: Comfy-Org/publish-node-action@main
+        with:
+          ## Add your own personal access token to your Github Repository secrets and reference it here.
+          personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}

custom_nodes/ComfyUI-N-Nodes/.gitignore

@@ -0,0 +1,322 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+libs/moondream_repo
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+*.pyc

custom_nodes/ComfyUI-N-Nodes/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 pythongosssss
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

custom_nodes/ComfyUI-N-Nodes/README.md

@@ -0,0 +1,369 @@
+[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/C0C0AJECJ)
+
+# ComfyUI-N-Suite
+A suite of custom nodes for ComfyUI that includes Integer, string and float variable nodes, GPT nodes and video nodes.
+
+> [!IMPORTANT]  
+> These nodes were tested primarily in Windows in the default environment provided by ComfyUI and in the environment created by the [notebook](https://github.com/comfyanonymous/ComfyUI/blob/master/notebooks/comfyui_colab.ipynb) for paperspace specifically with the cyberes/gradient-base-py3.10:latest docker image.
+**Any other environment has not been tested.**
+
+
+# Installation
+
+1. Clone the repository:
+`git clone https://github.com/Nuked88/ComfyUI-N-Nodes.git`  
+to your ComfyUI `custom_nodes` directory
+
+2. ~~IMPORTANT: If you want the GPT nodes on GPU you'll need to run **install_dependency bat files**. 
+There are 2 versions: ***install_dependency_ggml_models.bat*** for the old ggmlv3 models and ***install_dependency_gguf_models.bat*** for all the new models (GGUF).
+YOU CAN ONLY USE ONE OF THEM AT A TIME!
+Since _llama-cpp-python_ needs to be compiled from source code to enable it to use the GPU, you will first need to have [CUDA](https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64)  and visual studio 2019 or 2022  (in the case of my bat) installed to compile it. For details and the full guide you can go [HERE](https://github.com/abetlen/llama-cpp-python).~~ 
+
+3. If you intend to use GPTLoaderSimple with the Moondream model, you'll need to execute the 'install_extra.bat' script, which will install transformers version 4.36.2.
+4. Reboot ComfyUI
+
+In case you need to revert these changes (due to incompatibility with other nodes), you can utilize the 'remove_extra.bat' script.
+
+ComfyUI will automatically load all custom scripts and nodes at startup.
+
+> [!NOTE]  
+> The llama-cpp-python installation will be done automatically by the script. If you have an NVIDIA GPU NO MORE CUDA BUILD IS NECESSARY thanks to [jllllll](https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/) repo. I've also dropped the support to GGMLv3 models since all notable models should have switched to the latest version of GGUF by now.
+
+
+> [!NOTE]  
+> Since 14/02/2024, the node has undergone a massive rewrite, which also led to the change of all node names in order to avoid any conflicts with other extensions in the future (or at least I hope so). Consequently, the old workflows are no longer compatible and will require manual replacement of each node.
+> To avoid this, I have created a tool that allows for automatic replacement.
+> On Windows, simply drag any *.json workflow onto the migrate.bat file located in (custom_nodes/ComfyUI-N-Nodes), and another workflow with the suffix _migrated will be created in the same folder as the current workflow.
+> On Linux, you can use the script in the following way: python libs/migrate.py path/to/original/workflow/.
+> For security reasons, the original workflow will not be deleted."
+> For install the last version of this repository before this changes from the Comfyui-N-Suite execute **git checkout 29b2e43baba81ee556b2930b0ca0a9c978c47083**
+
+
+- For uninstallation:
+  - Delete the `ComfyUI-N-Nodes` folder in `custom_nodes`
+  - Delete the `comfyui-n-nodes` folder in  `ComfyUI\web\extensions`
+  - Delete the `n-styles.csv` and `n-styles.csv.backup` file in `ComfyUI\styles`
+  - Delete the `GPTcheckpoints` folder in `ComfyUI\models`
+
+
+
+
+
+# Update
+1. Navigate to the cloned repo e.g. `custom_nodes/ComfyUI-N-Nodes`
+2. `git pull`
+
+# Features
+
+## 📽️ Video Nodes 📽️
+
+### LoadVideo
+
+![alt text](./img/image-13.png)
+
+The LoadVideoAdvanced node allows loading a video file and extracting frames from it.
+The name has been changed from `LoadVideo` to `LoadVideoAdvanced` in order to avoid conflicts with the `LoadVideo` animatediff node.
+
+
+#### Input Fields
+- `video`: Select the video file to load.
+- `framerate`: Choose whether to keep the original framerate or reduce to half or quarter speed.
+- `resize_by`: Select how to resize frames - 'none', 'height', or 'width'.
+- `size`: Target size if resizing by height or width.
+- `images_limit`: Limit number of frames to extract.
+- `batch_size`: Batch size for encoding frames.
+- `starting_frame`: Select which frame to start from.
+- `autoplay`: Select whether to autoplay the video.
+- `use_ram`: Use RAM instead of disk for decompressing video frames.  
+
+#### Output
+
+- `IMAGES`: Extracted frame images as PyTorch tensors.
+- `LATENT`: Empty latent vectors.
+- `METADATA`: Video metadata - FPS and number of frames.
+- `WIDTH:` Frame width.
+- `HEIGHT`: Frame height.
+- `META_FPS`: Frame rate.
+- `META_N_FRAMES`: Number of frames.
+
+
+The node extracts frames from the input video at the specified framerate. It resizes frames if chosen and returns them as batches of PyTorch image tensors along with latent vectors, metadata, and frame dimensions.
+
+### SaveVideo
+The SaveVideo node takes in extracted frames and saves them back as a video file.
+![alt text](./img/image-3.png)
+
+#### Input Fields
+- `images`: Frame images as tensors.
+- `METADATA`: Metadata from LoadVideo node.
+- `SaveVideo`: Toggle saving output video file.
+- `SaveFrames`: Toggle saving frames to a folder.
+- `CompressionLevel`: PNG compression level for saving frames.
+#### Output
+Saves output video file and/or extracted frames.
+
+The node takes extracted frames and metadata and can save them as a new video file and/or individual frame images. Video compression and frame PNG compression can be configured.
+NOTE: If you are using **LoadVideo** as source of the frames, the audio of the original file will be maintained but only in case **images_limit** and **starting_frame** are equal to Zero.
+
+### LoadFramesFromFolder
+![alt text](./img/image.png)
+
+The LoadFramesFromFolder node allows loading image frames from a folder and returning them as a batch.
+
+
+#### Input Fields
+- `folder`: Path to the folder containing the frame images.Must be png format, named with a number (eg. 1.png or even 0001.png).The images will be loaded sequentially.
+- `fps`: Frames per second to assign to the loaded frames.
+
+#### Output
+- `IMAGES`: Batch of loaded frame images as PyTorch tensors.
+- `METADATA`: Metadata containing the set FPS value.
+- `MAX_WIDTH`: Maximum frame width.
+- `MAX_HEIGHT`: Maximum frame height.
+- `FRAME COUNT`: Number of frames in the folder.
+- `PATH`: Path to the folder containing the frame images.
+- `IMAGE LIST`: List of frame images in the folder (not a real list just a string divided by \n).
+
+The node loads all image files from the specified folder, converts them to PyTorch tensors, and returns them as a batched tensor along with simple metadata containing the set FPS value.
+
+This allows easily loading a set of frames that were extracted and saved previously, for example, to reload and process them again. By setting the FPS value, the frames can be properly interpreted as a video sequence.
+
+### SetMetadataForSaveVideo
+
+![alt text](./img/image-1.png)
+
+The SetMetadataForSaveVideo node allows setting metadata for the SaveVideo node.
+
+### FrameInterpolator
+
+![alt text](./img/image-4.png)
+
+The FrameInterpolator node allows interpolating between extracted video frames to increase the frame rate and smooth motion.
+
+
+#### Input Fields
+
+- `images`: Extracted frame images as tensors.
+- `METADATA`: Metadata from video - FPS and number of frames.
+- `multiplier`: Factor by which to increase frame rate. 
+
+#### Output  
+
+- `IMAGES`: Interpolated frames as image tensors.
+- `METADATA`: Updated metadata with new frame rate.
+
+The node takes extracted frames and metadata as input. It uses an interpolation model (RIFE) to generate additional in-between frames at a higher frame rate. 
+
+The original frame rate in the metadata is multiplied by the `multiplier` value to get the new interpolated frame rate.
+
+The interpolated frames are returned as a batch of image tensors, along with updated metadata containing the new frame rate.
+
+This allows increasing the frame rate of an existing video to achieve smoother motion and slower playback. The interpolation model creates new realistic frames to fill in the gaps rather than just duplicating existing frames.
+
+The original code has been taken from [HERE](https://github.com/hzwer/Practical-RIFE/tree/main)
+
+## Variables
+Since the primitive node has limitations in links (for example at the time i'm writing you cannot link "start_at_step" and "steps" of another ksampler toghether), I decided to create these simple node-variables to bypass this limitation
+The node-variables are:
+- Integer
+- Float
+- String
+
+
+## 🤖 GPTLoaderSimple and GPTSampler 🤖
+
+These custom nodes are designed to enhance the capabilities of the ConfyUI framework by enabling text generation using GGUF GPT models. This README provides an overview of the two custom nodes and their usage within ConfyUI.
+
+You can add in the _extra_model_paths.yaml_ the path where your model GGUF are in this way (example):
+
+`other_ui:
+          base_path: I:\\text-generation-webui
+          GPTcheckpoints: models/`
+          
+Otherwise it will create a GPTcheckpoints folder in the model folder of ComfyUI where you can place your .gguf models.
+
+Two folders have also been created within the 'Llava' directory in the 'GPTcheckpoints' folder for the LLava model:
+
+`clips`: This folder is designated for storing the clips for your LLava models (usually, files that start with **mm** in the repository).
+`models`: This folder is designated for storing the LLava models.
+
+This nodes actually supports 4 different models: 
+ - All the GGUF supported by [llama.cpp](https://github.com/ggerganov/llama.cpp) 
+ - Llava 
+ - Moondream
+ - Joytag
+
+
+#### GGUF LLM
+
+The GGUF models can be downloaded from the [Huggingface Hub](https://huggingface.co/models?search=gguf)
+
+[HERE](https://www.youtube.com/watch?v=gzTqXbF0S-w) a video of an example of how to use the GGUF models by [boricuapab](https://github.com/boricuapab)
+
+
+#### Llava
+Here a small list of the models supported by this nodes:
+
+[LlaVa 1.5 7B](https://huggingface.co/mys/ggml_llava-v1.5-7b/)
+[LlaVa 1.5 13B](https://huggingface.co/mys/ggml_llava-v1.5-13b)
+[LlaVa 1.6 Mistral 7B](https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/)
+[BakLLaVa](https://huggingface.co/mys/ggml_bakllava-1)
+[Nous Hermes 2 Vision](https://huggingface.co/billborkowski/llava-NousResearch_Nous-Hermes-2-Vision-GGUF)
+
+####Example with Llava model:
+![alt text](./img/image-5.png)
+
+#### Moondream
+The model will be automatically downloaded when you run the first time.
+Anyway, it is available [HERE](https://huggingface.co/vikhyatk/moondream1/tree/main)
+The code taken from [this repository](https://github.com/vikhyat/moondream)
+
+####Example with Moondream model:
+![alt text](./img/image-6.png)
+
+#### Joytag
+The model will be automatically downloaded when you run the first time.
+Anyway, it is available [HERE](https://huggingface.co/fancyfeast/joytag/tree/main)
+The code taken from [this repository](https://github.com/fpgaminer/joytag)
+
+####Example with Joytag model:
+![alt text](./img/image-7.png)
+
+### GPTLoaderSimple
+
+![alt text](./img/image11.png)
+
+The `GPTLoaderSimple` node is responsible for loading GPT model checkpoints and creating an instance of the Llama library for text generation. It provides an interface to configure GPU layers, the number of threads, and maximum context for text generation.
+
+
+
+#### Input Fields
+
+- `ckpt_name`: Select the GPT checkpoint name from the available options (joytag and moondream will be automatically downloaded used the first time).
+- `gpu_layers`: Specify the number of GPU layers to use (default: 27).
+- `n_threads`: Specify the number of threads for text generation (default: 8).
+- `max_ctx`: Specify the maximum context length for text generation (default: 2048).
+
+#### Output
+
+The node returns an instance of the Llama library (MODEL) and the path to the loaded checkpoint (STRING).
+
+### GPTSampler
+
+![alt text](./img/image-8.png)
+
+The `GPTSampler` node facilitates text generation using GPT models based on the input prompt and various generation parameters. It allows you to control aspects like temperature, top-p sampling, penalties, and more.
+
+
+#### Input Fields
+
+- `prompt`: Enter the input prompt for text generation.
+- `image`: Image input for Joytag, moondream and llava models.
+- `model`: Choose the GPT model to use for text generation.
+- `max_tokens`: Set the maximum number of tokens in the generated text (default: 128).
+- `temperature`: Set the temperature parameter for randomness (default: 0.7).
+- `top_p`: Set the top-p probability for nucleus sampling (default: 0.5).
+- `logprobs`: Specify the number of log probabilities to output (default: 0).
+- `echo`: Enable or disable printing the input prompt alongside the generated text.
+- `stop_token`: Specify the token at which text generation stops.
+- `frequency_penalty`, `presence_penalty`, `repeat_penalty`: Control word generation penalties.
+- `top_k`: Set the top-k tokens to consider during generation (default: 40).
+- `tfs_z`: Set the temperature scaling factor for top frequent samples (default: 1.0).
+- `print_output`: Enable or disable printing the generated text to the console.
+- `cached`: Choose whether to use cached generation (default: NO).
+- `prefix`, `suffix`: Specify text to prepend and append to the prompt.
+- `max_tags`: This only affect the max number of tags generated by joydag.
+
+#### Output
+
+The node returns the generated text along with a UI-friendly representation.
+
+
+## Image Pad For Outpainting Advanced 
+![alt text](./img/image-14.png)
+
+The `ImagePadForOutpaintingAdvanced` node is an alternative to the `ImagePadForOutpainting` node that applies the technique seen in [this video](https://www.youtube.com/@robadams2451) under the outpainting mask.
+The color correction part was taken from [this](https://github.com/sipherxyz/comfyui-art-venture) custom node from Sipherxyz
+
+#### Input Fields
+
+- `image`: Image input.
+- `left`: pixel to extend from left,
+- `top`: pixel to extend from top,
+- `right`: pixel to extend from right,
+- `bottom`: pixel to extend from bottom.
+- `feathering`: feathering strength
+- `noise`: blend strenght from noise and the copied border
+- `pixel_size`: how big will be the pixel in the pixellated effect
+- `pixel_to_copy`: how many pixels to copy (from each side)
+- `temperature`: color correction setting that is only applied to the mask part.
+- `hue`: color correction setting that is only applied to the mask part.
+- `brightness`: color correction setting that is only applied to the mask part.
+- `contrast`: color correction setting that is only applied to the mask part.
+- `saturation`: color correction setting that is only applied to the mask part.
+- `gamma`: color correction setting that is only applied to the mask part.
+
+#### Output
+
+The node returns the processed image and the mask.
+
+## Dynamic Prompt
+
+![alt text](./img/image-9.png)
+
+The `DynamicPrompt` node generates prompts by combining a fixed prompt with a random selection of tags from a variable prompt. This enables flexible and dynamic prompt generation for various use cases.
+
+#### Input Fields
+
+- `variable_prompt`: Enter the variable prompt for tag selection.
+- `cached`: Choose whether to cache the generated prompt (default: NO).
+- `number_of_random_tag`: Choose between "Fixed" and "Random" for the number of random tags to include.
+- `fixed_number_of_random_tag`: If `number_of_random_tag` if "Fixed" Specify the number of random tags to include (default: 1).
+- `fixed_prompt` (Optional): Enter the fixed prompt for generating the final prompt.
+
+#### Output
+
+The node returns the generated prompt, which is a combination of the fixed prompt and selected random tags.
+
+#### Example Usage
+
+- Just fill the `variable_prompt` field with tag comma separated, the `fixed_prompt` is optional
+
+
+## CLIP Text Encode Advanced (Experimental)
+
+![alt text](./img/image-10.png)
+
+The `CLIP Text Encode Advanced` node is an alternative to the standard `CLIP Text Encode` node. It offers support for Add/Replace/Delete styles, allowing for the inclusion of both positive and negative prompts within a single node.
+
+The base style file is called `n-styles.csv` and is located in the `ComfyUI\styles` folder.
+The styles file follows the same format as the current `styles.csv` file utilized in A1111 (at the time of writing).
+
+NOTE: this note is experimental and still have alot of bugs
+
+#### Input Fields
+
+- `clip`: clip input 
+- `style`: it will automatically fill the positive and negative prompts based on the choosen style
+
+#### Output
+- `positive`: positive conditions
+- `negative`: negative conditions
+
+
+
+
+
+
+## Troubleshooting
+
+- ~~**SaveVideo - Preview not working**: is related to a conflict with animateDiff, i've already opened a [PR](https://github.com/ArtVentureX/comfyui-animatediff/pull/64) to solve this issue. Meanwhile you can download my patched version from [here](https://github.com/Nuked88/comfyui-animatediff)~~ pull has been merged so this problem should be fixed now!
+
+## Contributing
+
+Feel free to contribute to this project by reporting issues or suggesting improvements. Open an issue or submit a pull request on the GitHub repository.
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+

custom_nodes/ComfyUI-N-Nodes/__init__.py

@@ -0,0 +1,74 @@
+import importlib.util
+import os
+import sys
+from .nnodes import init, get_ext_dir,check_and_install,downloader,get_commit,color
+import folder_paths
+import traceback
+from pathlib import Path
+NODE_CLASS_MAPPINGS = {}
+NODE_DISPLAY_NAME_MAPPINGS = {}
+
+
+if init():
+    print("------------------------------------------")
+    print(f"{color.BLUE}### N-Suite Revision:{color.END} {color.GREEN}{get_commit()} {color.END}")
+    py = Path(get_ext_dir("py"))
+    files = list(py.glob("*.py"))
+    check_and_install('packaging')
+    check_and_install('py-cpuinfo',"cpuinfo")
+    check_and_install('gitpython','git')
+    check_and_install('moviepy')
+    check_and_install("opencv-python","cv2") 
+    check_and_install('scikit-build',"skbuild")
+    #LLAMA DEPENTENCIES
+    check_and_install('typing')
+    check_and_install('diskcache')
+    check_and_install('llama_cpp')
+    check_and_install('timm',"timm","0.9.12",reboot=True)
+    #check_and_install('gitpython',"git")
+    #check_and_install('sentencepiece')
+    #check_and_install("accelerate")
+    #check_and_install('transformers','transformers',"4.36.2")
+    
+    
+
+
+    #git clone https://github.com/hzwer/Practical-RIFE.git
+    from git import Repo
+    if not os.path.exists(os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","rifle")):
+        Repo.clone_from("https://github.com/hzwer/Practical-RIFE.git", os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","rifle"))
+
+    if not os.path.exists(os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","moondream_repo")):
+        repo = Repo.clone_from("https://github.com/Nuked88/moondream.git", os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","moondream_repo"))
+
+        #commit_hash = "38af98596e59f2a6c25c6b52b2bd5a672dab4144"
+        #repo.git.checkout(commit_hash)
+
+    #if file moondream.py not exist 
+    #if not os.path.exists(os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","moondream_repo","moondream","moondream.py")):
+    #    #delete moondream_repo and download repo again
+    #    shutil.rmtree(os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","moondream_repo"))
+    #    repo = Repo.clone_from("https://github.com/Nuked88/moondream.git", os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","moondream_repo"))
+
+    #if train_log folder not exists
+    if not os.path.exists(os.path.join(os.path.dirname(os.path.realpath(__file__)),"libs","rifle","train_log")):
+        downloader("https://github.com/Nuked88/DreamingAI/raw/main/RIFE_trained_model_v4.7.zip")
+                   
+    # code based on pysssss repo
+    for file in files:
+        try:
+            name = os.path.splitext(file)[0]
+            spec = importlib.util.spec_from_file_location(name, os.path.join(py, file))
+            module = importlib.util.module_from_spec(spec)
+            sys.modules[name] = module
+            spec.loader.exec_module(module)
+            if hasattr(module, "NODE_CLASS_MAPPINGS") and getattr(module, "NODE_CLASS_MAPPINGS") is not None:
+                NODE_CLASS_MAPPINGS.update(module.NODE_CLASS_MAPPINGS)
+                if hasattr(module, "NODE_DISPLAY_NAME_MAPPINGS") and getattr(module, "NODE_DISPLAY_NAME_MAPPINGS") is not None:
+                    NODE_DISPLAY_NAME_MAPPINGS.update(module.NODE_DISPLAY_NAME_MAPPINGS)
+        except Exception as e:
+            traceback.print_exc()
+
+
+__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
+WEB_DIRECTORY = "./js"

custom_nodes/ComfyUI-N-Nodes/config.json

@@ -0,0 +1,4 @@
+{
+	"name": "N-Suite",
+	"logging": false
+}

custom_nodes/ComfyUI-N-Nodes/install_extra.bat

@@ -0,0 +1,15 @@
+@echo off
+set "python_exec=..\..\..\python_embeded\python.exe"
+
+echo Installing dependency for moondream_repo...
+if exist "%python_exec%" (
+    echo Installing with ComfyUI Portable
+    "%python_exec%" -s -m pip install transformers==4.36.2
+    echo Done. Please reboot ComfyUI.
+) else (
+    echo Installing with system Python
+    pip install  transformers==4.36.2
+    echo Done. Please reboot ComfyUI.
+)
+
+pause

custom_nodes/ComfyUI-N-Nodes/js/cte_advanced.js

@@ -0,0 +1,213 @@
+import { app } from "/scripts/app.js";
+import { api } from "/scripts/api.js"
+const MultilineSymbol = Symbol();
+const MultilineResizeSymbol = Symbol();
+
+function getStyles(name) {
+	//console.log("getStyles called " + name);
+	
+	return api.fetchApi('/nsuite/styles')
+	  .then(response => response.json())
+	  .then(data => {
+		// Eseguire l'elaborazione dei dati
+		const styles = data.styles;
+		//console.log('Styles:', styles);
+		let positive_prompt = "";
+		let negative_prompt = "";
+  
+		// Funzione per ottenere positive_prompt e negative_prompt dato il name
+		for (let i = 0; i < styles[0].length; i++) {
+		  const style = styles[0][i];
+		  if (style.name === name) {
+			positive_prompt = style.prompt;
+			negative_prompt = style.negative_prompt;
+			//console.log('Style:', style.name);
+			break;
+		  }
+		}
+  
+		if (positive_prompt !== "") {
+		  //console.log("Positive prompt:", positive_prompt);
+		  //console.log("Negative prompt:", negative_prompt);
+		  return { positive_prompt: positive_prompt, negative_prompt: negative_prompt };
+		} else {
+		  return { positive_prompt: "", negative_prompt: "" };
+		}
+	  })
+	  .catch(error => {
+		console.error('Error:', error);
+		throw error; // Rilancia l'errore per consentire al chiamante di gestirlo
+	  });
+  }
+
+  function addStyles(name, positive_prompt, negative_prompt) {
+	  return api.fetchApi('/nsuite/styles/add', {
+		method: 'POST',
+		headers: {
+		  'Content-Type': 'application/json',
+		},
+		body: JSON.stringify({
+		  name: name,
+		  positive_prompt: positive_prompt,
+		  negative_prompt: negative_prompt
+		}),
+		  
+	  })
+  }
+
+  function updateStyles(name, positive_prompt, negative_prompt) {
+	return api.fetchApi('/nsuite/styles/update', {
+		method: 'POST',
+		headers: {
+			'Content-Type': 'application/json',
+		},
+		body: JSON.stringify({
+			name: name,
+			positive_prompt: positive_prompt,
+			negative_prompt: negative_prompt
+		}),
+	})
+}
+
+function removeStyles(name) {
+	//confirmation
+	let ok = confirm("Are you sure you want to remove this style?");
+	if (!ok) {
+		return;
+	}
+
+	return api.fetchApi('/nsuite/styles/remove', {
+		method: 'POST',
+		headers: {
+			'Content-Type': 'application/json',
+		},
+		body: JSON.stringify({
+			name: name
+		}),
+	})
+}
+
+app.registerExtension({
+	name: "n.CLIPTextEncodeAdvancedNSuite",
+	async beforeRegisterNodeDef(nodeType, nodeData, app) {
+
+		const onAdded = nodeType.prototype.onAdded;
+		if (nodeData.name === "CLIPTextEncodeAdvancedNSuite [n-suite]") {
+		nodeType.prototype.onAdded = function () {
+			onAdded?.apply(this, arguments);
+			const styles = this.widgets.find((w) => w.name === "styles");
+			const p_prompt  = this.widgets.find((w) => w.name === "positive_prompt");
+			const n_prompt = this.widgets.find((w) => w.name === "negative_prompt");
+			const cb = nodeData.callback;
+			let addedd_positive_prompt = "";
+			let addedd_negative_prompt = "";
+			styles.callback = function () {
+				let index = styles.options.values.indexOf(styles.value); 
+				 
+		
+				if (addedd_positive_prompt == "" && addedd_negative_prompt == "") {
+					getStyles(styles.options.values[index-1]).then(style_prompts => {
+						//wait 4 seconds
+					
+						console.log(style_prompts);
+				
+						addedd_positive_prompt =  style_prompts.positive_prompt;
+						addedd_negative_prompt =  style_prompts.negative_prompt;
+						//alert("Addedd positive prompt: " + addedd_positive_prompt + "\nAddedd negative prompt: " + addedd_negative_prompt);
+					})
+				}
+
+
+				let current_positive_prompt = p_prompt.value;
+				let current_negative_prompt = n_prompt.value;
+				
+				getStyles(styles.value).then(style_prompts => {
+					//console.log(style_prompts)
+
+					if ((current_positive_prompt.trim() != addedd_positive_prompt.trim() || current_negative_prompt.trim() != addedd_negative_prompt.trim())) {
+						
+						let ok = confirm("Style has been changed. Do you want to change style without saving?");
+
+						
+						if (!ok) {
+							if (styles.value === styles.options.values[0]) {
+								value = styles.options.values[0];
+							}
+							styles.value = styles.options.values[index-1];
+
+							
+							return;
+						}
+					}
+
+					// add the addedd prompt to the current prompt
+					p_prompt.value =  style_prompts.positive_prompt;
+					n_prompt.value =  style_prompts.negative_prompt;
+
+
+					addedd_positive_prompt = style_prompts.positive_prompt;
+					addedd_negative_prompt = style_prompts.negative_prompt;
+					if (cb) {
+						return cb.apply(this, arguments);
+					}
+				  })
+				  .catch(error => {
+					console.error('Error:', error);
+				  });
+				
+			};
+
+			
+
+			let savestyle;
+			let replacestyle;
+			let deletestyle;
+		
+			
+			// Create the button widget for selecting the files
+			savestyle = this.addWidget("button", "New", "image", () => {
+				////console.log("Save called");
+				//ask input name style
+				let inputName = prompt("Enter a name for the style:", styles.value);
+				if (inputName === null) {
+					return;
+				}
+			
+				
+				addStyles(inputName, p_prompt.value, n_prompt.value);
+				// Add the file to the dropdown list and update the widget value
+				
+				if (!styles.options.values.includes(inputName)) {
+					styles.options.values.push(inputName);
+				  }
+
+			},{
+				cursor: "grab",
+			},);
+			replacestyle = this.addWidget("button", "Replace", "image", () => {
+				//console.log("Replace called");
+				updateStyles(styles.value, p_prompt.value, n_prompt.value);
+			},{
+				cursor: "grab",
+			},);
+			deletestyle = this.addWidget("button", "Delete", "image", () => {
+				//console.log("Delete called");
+				removeStyles(styles.value);
+
+				// Remove the file from the dropdown list
+				styles.options.values = styles.options.values.filter((value) => value !== styles.value);
+			},{
+				cursor: "grab",
+			},);
+			savestyle.serialize = false;	
+		
+		}
+	
+
+		
+
+
+		};
+		
+	},
+});

custom_nodes/ComfyUI-N-Nodes/js/dynamicPrompt.js

@@ -0,0 +1,44 @@
+import { app } from "/scripts/app.js";
+import { ComfyWidgets } from "/scripts/widgets.js";
+
+app.registerExtension({
+	name: "n.DynamicPrompt",
+	async beforeRegisterNodeDef(nodeType, nodeData, app) {
+		
+		if (nodeData.name === "DynamicPrompt") {
+			console.warn("DynamicPrompt detected")
+			
+			const onExecuted = nodeType.prototype.onExecuted;
+
+	
+			nodeType.prototype.onExecuted = function (message) {
+				onExecuted?.apply(this, arguments);
+				
+				const pos_cached = this.widgets.findIndex((w) => w.name === "cached");
+				console.warn("value:"+pos_cached)
+				
+				if (this.widgets) {
+					const pos_text = this.widgets.findIndex((w) => w.name === "text");
+					if (pos_text !== -1) {
+						for (let i = pos_text; i < this.widgets.length; i++) {
+							this.widgets[i].onRemove?.();
+						}
+						this.widgets.length = pos_text;
+					}
+				}
+
+
+				if (this.widgets[pos_cached].value === "NO") {
+					
+					const w = ComfyWidgets["STRING"](this, "text", ["STRING", { multiline: true }], app);
+					//random seed
+					var rnm = Math.floor(Math.random() * 10000)
+					w.widget.value = rnm;
+					
+				
+				}
+			
+			};
+		}
+	},
+});

custom_nodes/ComfyUI-N-Nodes/js/extended_widgets.js

@@ -0,0 +1,329 @@
+//extended_widgets.js
+import { api } from "/scripts/api.js"
+import { ComfyWidgets } from "/scripts/widgets.js";
+
+const MultilineSymbol = Symbol();
+const MultilineResizeSymbol = Symbol();
+async function uploadFile(file, updateNode, node, pasted = false) {
+	const videoWidget = node.widgets.find((w) => w.name === "video");
+
+
+	try {
+		// Wrap file in formdata so it includes filename
+		const body = new FormData();
+		body.append("image", file);
+		if (pasted) {
+			body.append("subfolder", "pasted");
+		}
+		else {
+			body.append("subfolder", "n-suite");
+		}
+
+		const resp = await api.fetchApi("/upload/image", {
+			method: "POST",
+			body,
+		});
+
+		if (resp.status === 200) {
+			const data = await resp.json();
+			// Add the file to the dropdown list and update the widget value
+			let path = data.name;
+
+
+			if (!videoWidget.options.values.includes(path)) {
+				videoWidget.options.values.push(path);
+			}
+
+			if (updateNode) {
+
+				videoWidget.value = path;
+				if (data.subfolder) path = data.subfolder + "/" + path;
+				showVideoInput(path,node);
+
+			}
+		} else {
+			alert(resp.status + " - " + resp.statusText);
+		}
+	} catch (error) {
+		alert(error);
+	}
+}
+
+function addVideo(node, name,src, app,autoplay_value) {
+	const MIN_SIZE = 50;
+	
+	function computeSize(size) {
+		try{
+	
+		if (node.widgets[0].last_y == null) return;
+
+		let y = node.widgets[0].last_y;
+		let freeSpace = size[1] - y;
+
+		// Compute the height of all non customvideo widgets
+		let widgetHeight = 0;
+		const multi = [];
+		for (let i = 0; i < node.widgets.length; i++) {
+			const w = node.widgets[i];
+			if (w.type === "customvideo") {
+				multi.push(w);
+			} else {
+				if (w.computeSize) {
+					widgetHeight += w.computeSize()[1] + 4;
+				} else {
+					widgetHeight += LiteGraph.NODE_WIDGET_HEIGHT + 4;
+				}
+			}
+		}
+	
+		// See how large each text input can be
+		freeSpace -= widgetHeight;
+		freeSpace /= multi.length + (!!node.imgs?.length);
+
+		if (freeSpace < MIN_SIZE) {
+			// There isnt enough space for all the widgets, increase the size of the node
+			freeSpace = MIN_SIZE;
+			node.size[1] = y + widgetHeight + freeSpace * (multi.length + (!!node.imgs?.length));
+			node.graph.setDirtyCanvas(true);
+		}
+
+		// Position each of the widgets
+		for (const w of node.widgets) {
+			w.y = y;
+			if (w.type === "customvideo") {
+				y += freeSpace;
+				w.computedHeight = freeSpace - multi.length*4;
+			} else if (w.computeSize) {
+				y += w.computeSize()[1] + 4;
+			} else {
+				y += LiteGraph.NODE_WIDGET_HEIGHT + 4;
+			}
+		}
+
+		node.inputHeight = freeSpace;
+	}catch(e){
+		
+	}
+	}
+	const widget = {
+		type: "customvideo",
+		name,
+		get value() {
+			return this.inputEl.value;
+		},
+		set value(x) {
+			this.inputEl.value = x;
+		},
+		draw: function (ctx, _, widgetWidth, y, widgetHeight) {
+			if (!this.parent.inputHeight) {
+				// If we are initially offscreen when created we wont have received a resize event
+				// Calculate it here instead
+				node.setSizeForImage?.();
+				
+			}
+			const visible = app.canvas.ds.scale > 0.5 && this.type === "customvideo";
+			const margin = 10;
+			let top_offset = 5
+			//hack for top menu
+			if (localStorage.getItem("Comfy.Settings.Comfy.UseNewMenu") === '"Top"') {
+				top_offset = 40;
+			}
+			
+			const elRect = ctx.canvas.getBoundingClientRect();
+			const transform = new DOMMatrix()
+				.scaleSelf(elRect.width / ctx.canvas.width, elRect.height / ctx.canvas.height)
+				.multiplySelf(ctx.getTransform())
+				.translateSelf(margin, margin + y);
+
+			const scale = new DOMMatrix().scaleSelf(transform.a, transform.d)
+			Object.assign(this.inputEl.style, {
+				transformOrigin: "0 0",
+				transform: scale,
+				left: `${transform.a + transform.e}px`,
+				top: `${transform.d +top_offset+ transform.f}px`,
+				width: `${widgetWidth - (margin * 2)}px`,
+				height: `${this.parent.inputHeight - (margin * 2)}px`,
+				position: "absolute",
+				background: (!node.color)?'':node.color,
+				color: (!node.color)?'':'white',
+				zIndex: app.graph._nodes.indexOf(node),
+			});
+			this.inputEl.hidden = !visible;
+		},
+	};
+
+
+	widget.inputEl = document.createElement("video");
+
+	
+	// Set the video attributes
+	Object.assign(widget.inputEl, {
+		controls: true,
+		src: src,
+		poster: "",
+		width: 400,
+		height: 300,
+		loop: true,
+		muted: true,
+		autoplay: autoplay_value,
+		type : "video/mp4"
+		
+	});
+
+
+
+	
+	// Add video element to the body
+	document.body.appendChild(widget.inputEl);
+
+
+
+	widget.parent = node;
+	document.body.appendChild(widget.inputEl);
+
+	node.addCustomWidget(widget);
+
+	app.canvas.onDrawBackground = function () {
+		// Draw node isnt fired once the node is off the screen
+		// if it goes off screen quickly, the input may not be removed
+		// this shifts it off screen so it can be moved back if the node is visible.
+		for (let n in app.graph._nodes) {
+			n = graph._nodes[n];
+			for (let w in n.widgets) {
+				let wid = n.widgets[w];
+				if (Object.hasOwn(wid, "inputEl")) {
+					wid.inputEl.style.left = -8000 + "px";
+					wid.inputEl.style.position = "absolute";
+				}
+			}
+		}
+	};
+
+	node.onRemoved = function () {
+		// When removing this node we need to remove the input from the DOM
+		for (let y in this.widgets) {
+			if (this.widgets[y].inputEl) {
+				this.widgets[y].inputEl.remove();
+			}
+		}
+	};
+
+	widget.onRemove = () => {
+		widget.inputEl?.remove();
+
+		// Restore original size handler if we are the last
+		if (!--node[MultilineSymbol]) {
+			node.onResize = node[MultilineResizeSymbol];
+			delete node[MultilineSymbol];
+			delete node[MultilineResizeSymbol];
+		}
+	};
+
+	if (node[MultilineSymbol]) {
+		node[MultilineSymbol]++;
+	} else {
+		node[MultilineSymbol] = 1;
+		const onResize = (node[MultilineResizeSymbol] = node.onResize);
+
+		node.onResize = function (size) {
+	
+			computeSize(size);
+			// Call original resizer handler
+			if (onResize) {
+				onResize.apply(this, arguments);
+			}
+		};
+	}
+
+	return { minWidth: 400, minHeight: 200, widget };
+}
+
+
+export function showVideoInput(name,node) {
+	const videoWidget = node.widgets.find((w) => w.name === "videoWidget");
+	const temp_web_url = node.widgets.find((w) => w.name === "local_url");
+	
+	
+	let folder_separator = name.lastIndexOf("/");
+	let subfolder = "n-suite";
+	if (folder_separator > -1) {
+		subfolder = name.substring(0, folder_separator);
+		name = name.substring(folder_separator + 1);
+	}
+
+	let url_video = api.apiURL(`/view?filename=${encodeURIComponent(name)}&type=input&subfolder=${subfolder}${app.getPreviewFormatParam()}`);
+	videoWidget.inputEl.src = url_video
+	temp_web_url.value = url_video
+}
+
+export function showVideoOutput(name,node) {
+	const videoWidget = node.widgets.find((w) => w.name === "videoOutWidget");
+
+	
+	
+	let folder_separator = name.lastIndexOf("/");
+	let subfolder = "n-suite/videos";
+	if (folder_separator > -1) {
+		subfolder = name.substring(0, folder_separator);
+		name = name.substring(folder_separator + 1);
+	}
+
+
+	let url_video = api.apiURL(`/view?filename=${encodeURIComponent(name)}&type=output&subfolder=${subfolder}${app.getPreviewFormatParam()}`);
+	videoWidget.inputEl.src = url_video
+
+	return url_video;
+}
+
+
+
+export const ExtendedComfyWidgets = {
+    ...ComfyWidgets, // Copy all the functions from ComfyWidgets
+	
+	VIDEO(node, inputName, inputData, src, app,type="input",autoplay_value=true) {
+	try {	
+		const videoWidget = node.widgets.find((w) => w.name === "video");
+		const autoplay = node.widgets.find((w) => w.name === "autoplay");
+		const defaultVal = "";
+		let res;
+		res = addVideo(node, inputName, src, app,autoplay_value);
+		
+		if (type == "input"){
+
+			const cb = node.callback;
+			videoWidget.callback = function () {
+				
+				showVideoInput(videoWidget.value, node);
+				if (cb) {
+					return cb.apply(this, arguments);
+				}
+			};
+			autoplay.callback = function () {
+				const videoWidgetz = node.widgets.find((w) => w.name === "videoWidget");
+			
+				videoWidgetz.inputEl.autoplay = autoplay.value;
+				showVideoInput(videoWidget.value, node);
+				if (cb) {
+					return cb.apply(this, arguments);
+				}
+			}
+		}
+
+		if (node.type =="LoadVideoAdvanced"){
+	
+
+	}
+
+		return res;	
+	}
+	catch (error) {
+
+		console.error("Errore in extended_widgets.js:", error);
+		throw error; 
+	
+	}
+
+},
+
+
+};

custom_nodes/ComfyUI-N-Nodes/js/gptSampler.js

@@ -0,0 +1,43 @@
+import { app } from "/scripts/app.js";
+import { ComfyWidgets } from "/scripts/widgets.js";
+
+app.registerExtension({
+	name: "n.GPTSampler",
+	async beforeRegisterNodeDef(nodeType, nodeData, app) {
+		
+		if (nodeData.name === "GPT Sampler [n-suite]") {
+			console.warn("GPTSampler detected")
+			
+			const onExecuted = nodeType.prototype.onExecuted;
+	
+			nodeType.prototype.onExecuted = function (message) {
+				onExecuted?.apply(this, arguments);
+				
+				const pos_cached = this.widgets.findIndex((w) => w.name === "cached");
+				console.warn("value:"+pos_cached)
+				
+				if (this.widgets) {
+					const pos_text = this.widgets.findIndex((w) => w.name === "text");
+					if (pos_text !== -1) {
+						for (let i = pos_text; i < this.widgets.length; i++) {
+							this.widgets[i].onRemove?.();
+						}
+						this.widgets.length = pos_text;
+					}
+				}
+
+
+				if (this.widgets[pos_cached].value === "NO") {
+					
+					const w = ComfyWidgets["STRING"](this, "text", ["STRING", { multiline: true }], app);
+					//random seed
+					var rnm = Math.floor(Math.random() * 10000)
+					w.widget.value = rnm;
+					
+				
+				}
+			
+			};
+		}
+	},
+});

custom_nodes/ComfyUI-N-Nodes/js/include_css.js

@@ -0,0 +1,22 @@
+import { $el } from "../../../scripts/ui.js";
+
+function addStylesheet(url) {
+	if (url.endsWith(".js")) {
+		url = url.substr(0, url.length - 2) + "css";
+	}
+	$el("link", {
+		parent: document.head,
+		rel: "stylesheet",
+		type: "text/css",
+		href: url.startsWith("http") ? url : getUrl(url),
+	});
+}
+function getUrl(path, baseUrl) {
+	if (baseUrl) {
+		return new URL(path, baseUrl).toString();
+	} else {
+		return new URL("../" + path, import.meta.url).toString();
+	}
+}
+
+addStylesheet(getUrl("styles.css", import.meta.url));

custom_nodes/ComfyUI-N-Nodes/js/styles.css

@@ -0,0 +1,20 @@
+textarea[placeholder="positive_prompt"] {
+    border: 1px solid #64d509;
+ 
+
+  }
+  textarea[placeholder="positive_prompt"]:focus-visible {
+    border: 1px solid #72eb0f;
+
+  }
+
+
+textarea[placeholder="negative_prompt"] {
+  border: 1px solid #a94442;
+  border-color: #a94442; 
+}
+
+textarea[placeholder="negative_prompt"]:focus-visible {
+  border: 1px solid #de5755;
+  border-color: #de5755; 
+}

custom_nodes/ComfyUI-N-Nodes/js/videoLoadAdvanced.js

@@ -0,0 +1,142 @@
+import { app } from "/scripts/app.js";
+import { api } from "/scripts/api.js"
+import { ExtendedComfyWidgets,showVideoInput } from "./extended_widgets.js";
+const MultilineSymbol = Symbol();
+const MultilineResizeSymbol = Symbol();
+
+
+async function uploadFile(file, updateNode, node, pasted = false) {
+	const videoWidget = node.widgets.find((w) => w.name === "video");
+	
+	
+	try {
+		// Wrap file in formdata so it includes filename
+		const body = new FormData();
+		body.append("image", file);
+		if (pasted) {
+			body.append("subfolder", "pasted");
+		}
+		else {
+			body.append("subfolder", "n-suite");
+		}
+	
+		const resp = await api.fetchApi("/upload/image", {
+			method: "POST",
+			body,
+		});
+
+		if (resp.status === 200) {
+			const data = await resp.json();
+			// Add the file to the dropdown list and update the widget value
+			let path = data.name;
+			
+
+			if (!videoWidget.options.values.includes(path)) {
+				videoWidget.options.values.push(path);
+			}
+			
+			if (updateNode) {
+		
+				videoWidget.value = path;
+				if (data.subfolder) path = data.subfolder + "/" + path;
+				showVideoInput(path,node);
+				
+			}
+		} else {
+			alert(resp.status + " - " + resp.statusText);
+		}
+	} catch (error) {
+		alert(error);
+	}
+}
+
+
+
+
+let uploadWidget = "";
+app.registerExtension({
+	name: "Comfy.VideoLoadAdvanced",
+	async beforeRegisterNodeDef(nodeType, nodeData, app) {
+
+		const onAdded = nodeType.prototype.onAdded;
+		if (nodeData.name === "LoadVideo [n-suite]") {
+		nodeType.prototype.onAdded = function () {
+			onAdded?.apply(this, arguments);
+			const temp_web_url = this.widgets.find((w) => w.name === "local_url");
+			const autoplay_value = this.widgets.find((w) => w.name === "autoplay");
+		
+			
+			let uploadWidget;
+			const fileInput = document.createElement("input");
+			Object.assign(fileInput, {
+				type: "file",
+				accept: "video/mp4,image/gif,video/webm",
+				style: "display: none",
+				onchange: async () => {
+					if (fileInput.files.length) {
+						await uploadFile(fileInput.files[0], true,this);
+					}
+				},
+			});
+			document.body.append(fileInput);
+			// Create the button widget for selecting the files
+			uploadWidget = this.addWidget("button", "choose file to upload", "image", () => {
+				fileInput.click();
+			},{
+				cursor: "grab",
+			},);
+			uploadWidget.serialize = false;
+
+
+		setTimeout(() => {
+			ExtendedComfyWidgets["VIDEO"](this, "videoWidget", ["STRING"], temp_web_url.value, app,"input", autoplay_value.value);
+		
+		}, 100); 
+		
+		
+		}
+	
+
+			nodeType.prototype.onDragOver = function (e) {
+				if (e.dataTransfer && e.dataTransfer.items) {
+					const image = [...e.dataTransfer.items].find((f) => f.kind === "file");
+					return !!image;
+				}
+	
+				return false;
+			};
+	
+			// On drop upload files
+			nodeType.prototype.onDragDrop = function (e) {
+				console.log("onDragDrop called");
+				let handled = false;
+				for (const file of e.dataTransfer.files) {
+					if (file.type.startsWith("video/mp4")) {
+						
+						const filePath = file.path || (file.webkitRelativePath || '').split('/').slice(1).join('/'); 
+
+
+						uploadFile(file, !handled,this ); // Dont await these, any order is fine, only update on first one
+
+						handled = true;
+					}
+				}
+	
+				return handled;
+			};
+	
+			nodeType.prototype.pasteFile = function(file) {
+				if (file.type.startsWith("video/mp4")) {
+
+					//uploadFile(file, true, is_pasted);
+
+					return true;
+				}
+				return false;
+			}
+
+
+		};
+		
+	},
+});

custom_nodes/ComfyUI-N-Nodes/js/videoSave.js

@@ -0,0 +1,87 @@
+import { app } from "/scripts/app.js";
+import { api } from "/scripts/api.js"
+import { ExtendedComfyWidgets,showVideoOutput } from "./extended_widgets.js";
+const MultilineSymbol = Symbol();
+const MultilineResizeSymbol = Symbol();
+
+
+async function uploadFile(file, updateNode, node, pasted = false) {
+	const videoWidget = node.widgets.find((w) => w.name === "video");
+	
+
+	try {
+		// Wrap file in formdata so it includes filename
+		const body = new FormData();
+		body.append("image", file);
+		if (pasted) {
+			body.append("subfolder", "pasted");
+		}
+		else {
+			body.append("subfolder", "n-suite");
+		}
+	
+		const resp = await api.fetchApi("/upload/image", {
+			method: "POST",
+			body,
+		});
+
+		if (resp.status === 200) {
+			const data = await resp.json();
+			// Add the file to the dropdown list and update the widget value
+			let path = data.name;
+			
+
+			if (!videoWidget.options.values.includes(path)) {
+				videoWidget.options.values.push(path);
+			}
+			
+			if (updateNode) {
+			//	showVideo(path,node);
+				videoWidget.value = path;
+				if (data.subfolder) path = data.subfolder + "/" + path;
+				showVideo(path,node);
+				
+			}
+		} else {
+			alert(resp.status + " - " + resp.statusText);
+		}
+	} catch (error) {
+		alert(error);
+	}
+}
+
+
+
+
+let uploadWidget = "";
+app.registerExtension({
+	name: "Comfy.VideoSave",
+	async beforeRegisterNodeDef(nodeType, nodeData, app) {
+
+		const onExecuted = nodeType.prototype.onExecuted;
+	
+			
+		const onAdded = nodeType.prototype.onAdded;
+		if (nodeData.name === "SaveVideo [n-suite]") {
+		nodeType.prototype.onAdded = function () {
+
+			ExtendedComfyWidgets["VIDEO"](this, "videoOutWidget", ["STRING"], "", app,"output");
+		
+		};
+		nodeType.prototype.onExecuted = function (message) {
+			onExecuted?.apply(this, arguments);
+			console.log(nodeData)
+
+		let full_path="";
+
+		for (const list of message.text) {
+			full_path = list;
+		}
+
+		let fullweb= showVideoOutput(full_path,this)
+		
+}
+		};
+		
+	},
+});

custom_nodes/ComfyUI-N-Nodes/libs/joytag_models.py

@@ -0,0 +1,1109 @@
+#from https://github.com/fpgaminer/joytag.git
+import json
+from pathlib import Path
+from typing import Optional
+import torch
+import torch.backends.cuda
+import torch.nn as nn
+import torch.nn.functional as F
+import torchvision
+
+from transformers.activations import QuickGELUActivation
+import math
+from einops.layers.torch import Rearrange
+import einops
+
+
+class VisionModel(nn.Module):
+	image_size: int
+	n_tags: int
+
+	def __init__(self, image_size: int, n_tags: int):
+		super().__init__()
+
+		self.image_size = image_size
+		self.n_tags = n_tags
+	
+	@staticmethod
+	def load_model(path: Path | str, device: str | None = None) -> 'VisionModel':
+		"""
+		Load a model from a directory.
+		:param path: The directory containing the model.
+		:return: The model, the image size, and the number of tags.
+		"""
+		with open(Path(path) / 'config.json', 'r') as f:
+			config = json.load(f)
+		
+		if (Path(path) / 'model.safetensors').exists():
+			from safetensors.torch import load_file
+			resume = load_file(Path(path) / 'model.safetensors', device='cpu')
+		else:
+			resume = torch.load(Path(path) / 'model.pt', map_location=torch.device('cpu'))['model']
+
+		model_classes = VisionModel.__subclasses__()
+		model_cls = next(cls for cls in model_classes if cls.__name__ == config['class'])
+
+		model = model_cls(**{k: v for k, v in config.items() if k != 'class'})
+		model.load(resume)
+		if device is not None:
+			model = model.to(device)
+
+		return model
+	
+	@staticmethod
+	def from_config(config: dict) -> 'VisionModel':
+		model_classes = VisionModel.__subclasses__()
+		model_cls = next(cls for cls in model_classes if cls.__name__ == config['class'])
+		return model_cls(**{k: v for k, v in config.items() if k != 'class'})
+	
+	def get_optimized_parameters(self, lr: float):
+		raise NotImplementedError
+	
+	def save(self):
+		raise NotImplementedError
+	
+	def load(self, state_dict):
+		raise NotImplementedError
+
+
+def basic_calculate_loss(preds: dict[str, torch.Tensor], batch: dict, pos_weight: torch.Tensor | None, loss_type: str):
+	def asl_helper(preds, target):
+		p = F.softmax(preds, dim=1)
+		xs_pos = p.clamp(min=1e-6)
+		xs_neg = (1 - p).clamp(min=1e-6)
+
+		los_pos = torch.log(torch.gather(xs_pos, 1, target.unsqueeze(1))).sum()
+		los_neg = torch.log(xs_neg)
+		los_neg = los_neg.sum() - torch.gather(los_neg, 1, target.unsqueeze(1)).sum()
+		loss = los_pos + los_neg
+
+		return -loss
+
+	if loss_type == "ce":
+		loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'])
+	elif loss_type == "weighted":
+		loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'], pos_weight=pos_weight)
+	elif loss_type == "focal":
+		gamma = 2
+		p = torch.sigmoid(preds['tags'])
+		ce_loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'], reduction='none')
+		p_t = p * batch['tags'] + (1 - p) * (1 - batch['tags'])
+		loss = ce_loss * ((1 - p_t) ** gamma)
+		loss = loss.mean()
+	elif loss_type == "focal2":
+		gamma = 2
+		p = torch.sigmoid(preds['tags'])
+		ce_loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'], reduction='none')
+		p_t = p * batch['tags'] + (1 - p) * (1 - batch['tags'])
+		loss = ce_loss * ((1 - p_t) ** gamma) * 256
+		loss = loss.mean()
+	elif loss_type == "asl":
+		p = torch.sigmoid(preds['tags'])
+		xs_pos = p
+		xs_neg = 1 - p
+
+		los_pos = batch['tags'] * torch.log(xs_pos.clamp(min=1e-6))
+		los_neg = (1 - batch['tags']) * torch.log(xs_neg.clamp(min=1e-6))
+		loss = los_pos + los_neg
+		loss = -loss.sum()
+
+		# Rating
+		loss = loss + asl_helper(preds['rating'], batch['rating'])
+
+		# Score
+		loss = loss + asl_helper(preds['score'], batch['score'])
+	elif loss_type == "asl2":
+		p = torch.sigmoid(preds['tags'])
+		xs_pos = p
+		xs_neg = 1 - p
+
+		los_pos = batch['tags'] * torch.log(xs_pos.clamp(min=1e-6))
+		los_neg = (1 - batch['tags']) * torch.log(xs_neg.clamp(min=1e-6))
+		loss = -los_pos - los_neg
+		loss = loss.sum()
+	elif loss_type == "asl3":
+		p = torch.sigmoid(preds['tags'])
+		xs_pos = p
+		xs_neg = 1 - p
+
+		los_pos = batch['tags'] * torch.log(xs_pos.clamp(min=1e-6))
+		los_neg = (1 - batch['tags']) * torch.log(xs_neg.clamp(min=1e-6))
+		loss = -los_pos - los_neg
+		loss = loss.mean()
+	elif loss_type == "asl4":
+		p = torch.sigmoid(preds['tags'])
+		xs_pos = p
+		xs_neg = 1 - p
+
+		los_pos = batch['tags'] * torch.log(xs_pos.clamp(min=1e-6))
+		los_neg = (1 - batch['tags']) * torch.log(xs_neg.clamp(min=1e-6))
+		loss = -los_pos - los_neg
+		loss = loss.mean() * 128
+	elif loss_type == "asl5":
+		loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'], pos_weight=pos_weight) * 128
+	elif loss_type == "asl6":
+		loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'], pos_weight=pos_weight) * 256
+	elif loss_type == "asl7":
+		loss = F.binary_cross_entropy_with_logits(preds['tags'], batch['tags'], pos_weight=pos_weight) * 2
+	else:
+		raise ValueError(f"Invalid loss type: {loss_type}")
+	
+	return loss
+
+
+class CLIPMlp(nn.Module):
+	def __init__(self, hidden_size: int, intermediate_size: int, activation_cls):
+		super().__init__()
+		self.activation_fn = activation_cls()
+		self.fc1 = nn.Linear(hidden_size, intermediate_size)
+		self.fc2 = nn.Linear(intermediate_size, hidden_size)
+
+	def forward(self, hidden_states: torch.Tensor):
+		hidden_states = self.fc1(hidden_states)
+		hidden_states = self.activation_fn(hidden_states)
+		hidden_states = self.fc2(hidden_states)
+		return hidden_states
+
+
+class FastCLIPAttention2(nn.Module):
+	"""Fast Attention module for CLIP-like. This is NOT a drop-in replacement for CLIPAttention, since it adds additional flexibility.  Mainly uses xformers."""
+	def __init__(self, hidden_size: int, out_dim: int, num_attention_heads: int, out_seq_len: Optional[int] = None, norm_qk: bool = False):
+		super().__init__()
+		self.out_seq_len = out_seq_len
+		self.embed_dim = hidden_size
+		self.out_dim = out_dim
+		self.norm_qk = norm_qk
+		self.num_heads = num_attention_heads
+		self.head_dim = hidden_size // num_attention_heads
+		assert self.head_dim * num_attention_heads == self.embed_dim, "embed_dim must be divisible by num_attention_heads"
+
+		self.q_proj = nn.Linear(self.embed_dim, self.embed_dim)
+		self.kv_proj = nn.Linear(self.embed_dim, self.embed_dim * 2)
+		self.out_proj = nn.Linear(self.embed_dim, self.out_dim)
+
+		if self.norm_qk:
+			self.query_norm = nn.LayerNorm(self.embed_dim)
+			self.key_norm = nn.LayerNorm(self.embed_dim)
+	
+	#def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
+	#	return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).contiguous()
+	
+	def forward(self, query_states: torch.Tensor, kv_states: torch.Tensor) -> torch.Tensor:
+		bsz, src_len, embed_dim = kv_states.size()
+		if self.out_seq_len is not None:
+			tgt_len = self.out_seq_len
+		else:
+			tgt_len = src_len
+		
+		kv_states = self.kv_proj(kv_states)  # (bsz, src_len, embed_dim * 2)
+		q_states = self.q_proj(query_states[:, :tgt_len])   # (bsz, tgt_len, embed_dim)
+
+		# NOTE: It is not clear if LayerNorm should be applied to the embed_dim, or to the head_dim
+		if self.norm_qk:
+			q_states = self.query_norm(q_states).type(q_states.dtype)
+			k_states = self.key_norm(kv_states[:, :, :embed_dim]).type(kv_states.dtype)
+			v_states = kv_states[:, :, embed_dim:]
+		else:
+			k_states = kv_states[:, :, :embed_dim]
+			v_states = kv_states[:, :, embed_dim:]
+		
+		q_states = q_states.view(bsz, tgt_len, self.num_heads, self.head_dim).transpose(1, 2)  # (bsz, num_heads, tgt_len, head_dim)
+		k_states = k_states.view(bsz, src_len, self.num_heads, self.head_dim).transpose(1, 2)  # (bsz, num_heads, src_len, head_dim)
+		v_states = v_states.view(bsz, src_len, self.num_heads, self.head_dim).transpose(1, 2)  # (bsz, num_heads, src_len, head_dim)
+
+		# Performs scale of query_states, attention, and softmax
+		with torch.backends.cuda.sdp_kernel(enable_math=False):
+			x = F.scaled_dot_product_attention(q_states, k_states, v_states)   # (bsz, num_heads, tgt_len, head_dim)
+			x = x.transpose(1, 2).contiguous().view(bsz, tgt_len, embed_dim)   # (bsz, tgt_len, embed_dim)
+		
+		# Projection
+		x = self.out_proj(x)  # (bsz, tgt_len, out_dim)
+
+		return x
+
+
+class SkipInit(nn.Module):
+	def __init__(self, hidden_size: int, channel_wise: bool, init_scale: float):
+		super().__init__()
+		self.hidden_size = hidden_size
+		self.channel_wise = channel_wise
+		self.init_scale = init_scale
+
+		if self.channel_wise:
+			self.scale = nn.Parameter(torch.ones(hidden_size) * init_scale)
+		else:
+			self.scale = nn.Parameter(torch.tensor(init_scale))
+
+	def forward(self, x: torch.Tensor) -> torch.Tensor:
+		return x * self.scale
+
+
+class FastCLIPEncoderLayer(nn.Module):
+	def __init__(
+		self,
+		hidden_size: int,
+		num_attention_heads: int,
+		out_seq_len: Optional[int],
+		activation_cls = QuickGELUActivation,
+		use_palm_alt: bool = False,
+		norm_qk: bool = False,
+		skip_init: Optional[float] = None,
+		stochastic_depth: Optional[float] = None,
+	):
+		super().__init__()
+
+		self.use_palm_alt = use_palm_alt
+		self.stochastic_depth = stochastic_depth
+		
+		self.self_attn = FastCLIPAttention2(
+			hidden_size=hidden_size,
+			out_dim=hidden_size,
+			num_attention_heads=num_attention_heads,
+			out_seq_len=out_seq_len,
+			norm_qk=norm_qk,
+		)
+		self.mlp = CLIPMlp(hidden_size, 4 * hidden_size, activation_cls)
+		self.layer_norm1 = nn.LayerNorm(hidden_size)
+		if not use_palm_alt:
+			self.layer_norm2 = nn.LayerNorm(hidden_size)
+		
+		if skip_init is not None:
+			self.attn_skip_init = SkipInit(hidden_size, channel_wise=True, init_scale=skip_init)
+			self.mlp_skip_init = SkipInit(hidden_size, channel_wise=True, init_scale=skip_init)
+		else:
+			self.attn_skip_init = nn.Identity()
+			self.mlp_skip_init = nn.Identity()
+	
+	def forward(self, hidden_states: torch.Tensor):
+		residual = hidden_states
+		hidden_states = self.layer_norm1(hidden_states)
+
+		if not self.use_palm_alt:
+			hidden_states = self.self_attn(query_states=hidden_states, kv_states=hidden_states)
+			hidden_states = self.attn_skip_init(hidden_states)
+			hidden_states = hidden_states + residual[:, :hidden_states.size(1)]
+
+			residual = hidden_states
+			hidden_states = self.layer_norm2(hidden_states)
+			hidden_states = self.mlp(hidden_states)
+			hidden_states = self.mlp_skip_init(hidden_states)
+			hidden_states = hidden_states + residual
+		else:
+			# An alternative implementation inspired by the PALM paper
+			# By performing the attention and MLP in parallel it's possible to fuse the linear projections of the attention and MLP layers
+			# We don't do that here yet, but that supposedly improves efficiency without hurting performance
+			attn = self.self_attn(query_states=hidden_states, kv_states=hidden_states)
+			attn = self.attn_skip_init(attn)
+			mlp = self.mlp(hidden_states[:, :attn.size(1)])
+			mlp = self.mlp_skip_init(mlp)
+
+			if self.stochastic_depth is not None:
+				attn = torchvision.ops.stochastic_depth(attn, self.stochastic_depth, mode='row', training=self.training)
+				mlp = torchvision.ops.stochastic_depth(mlp, self.stochastic_depth, mode='row', training=self.training)
+
+			hidden_states = residual[:, :attn.size(1)] + attn + mlp
+
+		return hidden_states
+
+
+def sinusoidal_position_embedding(width: int, height: int, depth: int, dtype, device, temperature = 10000):
+	"""
+	Sinusoidal position embedding. Returns a flat tensor of shape (h * w, d).
+	"""
+	assert depth % 4 == 0, "Embedding dimension must be divisible by 4."
+
+	y, x = torch.meshgrid(torch.arange(height, device=device), torch.arange(width, device=device), indexing="ij")
+	omega = torch.arange(depth // 4, device=device) / (depth // 4 - 1)
+	omega = 1. / (temperature ** omega)
+
+	y = y.flatten()[:, None] * omega[None, :]
+	x = x.flatten()[:, None] * omega[None, :]
+	embedding = torch.cat([x.sin(), x.cos(), y.sin(), y.cos()], dim=1)
+
+	return embedding.type(dtype)
+
+
+class CLIPEmbeddingLayer(nn.Module):
+	def __init__(self, hidden_size: int, num_channels: int, image_size: int, patch_size: int, patch_dropout: float = 0.0, good_dropout: bool = False, dpn: bool = False, sine_positional_embeddings: bool = False):
+		super().__init__()
+
+		assert image_size % patch_size == 0, "Image dimensions must be divisible by the patch size."
+
+		seq_len = (image_size // patch_size) ** 2
+		self.patch_dropout = patch_dropout
+		self.hidden_size = hidden_size
+		self.good_dropout = good_dropout
+		self.dpn = dpn
+		self.sine_positional_embeddings = sine_positional_embeddings
+		self.patch_size = patch_size
+
+		self.patch_embeddings = nn.Conv2d(
+			in_channels=num_channels,
+			out_channels=hidden_size,
+			kernel_size=patch_size,
+			stride=patch_size,
+			bias=False,
+		)
+		if not self.sine_positional_embeddings:
+			self.positional_embeddings = nn.Embedding(seq_len, hidden_size)
+		self.register_buffer("position_ids", torch.arange(seq_len))
+
+		if self.dpn:
+			self.to_patch_embeddings = nn.Sequential(
+				Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1=patch_size, p2=patch_size),
+				nn.LayerNorm(3 * patch_size * patch_size),
+				nn.Linear(3 * patch_size * patch_size, hidden_size),
+				nn.LayerNorm(hidden_size),
+			)
+		else:
+			self.to_patch_embeddings = nn.Conv2d(
+				in_channels=num_channels,
+				out_channels=hidden_size,
+				kernel_size=patch_size,
+				stride=patch_size,
+				bias=False,
+			)
+	
+	def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor:
+		B, C, H, W = pixel_values.shape
+		assert H % self.patch_size == 0, f"Input image height ({H}) needs to be divisible by the patch size ({self.patch_size})."
+		assert W % self.patch_size == 0, f"Input image width ({W}) needs to be divisible by the patch size ({self.patch_size})."
+
+		if self.dpn:
+			patches = self.to_patch_embeddings(pixel_values)
+		else:
+			patches = self.to_patch_embeddings(pixel_values)
+			patches = patches.flatten(2).transpose(1, 2)
+		
+		seq_len = patches.shape[1]
+		patch_dropout = int(math.ceil((1.0 - self.patch_dropout) * seq_len))
+		
+		if self.sine_positional_embeddings:
+			position_embeddings = sinusoidal_position_embedding(W // self.patch_size, H // self.patch_size, self.hidden_size, pixel_values.dtype, pixel_values.device)
+		else:
+			position_embeddings = self.positional_embeddings(self.position_ids)
+
+		if patch_dropout == seq_len or not self.training:
+			embeddings = patches + position_embeddings
+		elif self.good_dropout:
+			# Pick random patches to drop out
+			# The "good_dropout" variant uses random permutations for each batch item, but is slightly slower and involves more code
+
+			# The below method is a nice trick to generate a batch of random permutations.
+			# Torch (as of 1.13) doesn't have a built-in function to do this, and a for loop of torch.randperm is slow.
+			# Based on some benchmarks I measured the generation of the mask and the fetching to be only 50% slower than the non-"good_dropout" variant.
+			# And the time taken here is only a fraction of the time spent performing the embedding convolution.
+			# Generate a matrix of random numbers between 0 and 1 of shape (B, seq_len)
+			patch_mask = torch.rand(B, seq_len, device=patches.device)
+			# For each batch tensor, use argsort to convert the random numbers into a permutation of the patch indices
+			patch_mask = torch.argsort(patch_mask, dim=1)
+			# Truncate
+			patch_mask = patch_mask[:, :patch_dropout]
+
+			embeddings = patches.gather(1, patch_mask.unsqueeze(-1).expand(-1, -1, self.hidden_size)) + position_embeddings[patch_mask]
+		else:
+			# The non-"good_dropout" variant uses a single random permutation for all batch items, but is faster and uses less code
+			indices = torch.randperm(seq_len, device=pixel_values.device)[:patch_dropout]
+			embeddings = patches[:, indices, :] + position_embeddings[indices.expand(1, -1)]
+		
+		return embeddings
+
+
+class MHAPoolingHead(nn.Module):
+	def __init__(self, hidden_size: int, num_attention_heads: int, activation_cls, out_dim: int, alt_style: bool, norm_qk: bool):
+		super().__init__()
+
+		self.out_dim = out_dim if not alt_style else hidden_size
+
+		self.probe = nn.Parameter(torch.randn(hidden_size))
+
+		self.mlp = CLIPMlp(hidden_size, 4 * hidden_size, activation_cls)
+		self.layer_norm = nn.LayerNorm(hidden_size)
+		self.pooling_head = nn.Linear(hidden_size, 1)
+
+		self.self_attn = FastCLIPAttention2(
+			hidden_size=hidden_size,
+			out_dim=self.out_dim,
+			num_attention_heads=num_attention_heads,
+			out_seq_len=1,
+			norm_qk=norm_qk,
+		)
+		self.mlp = CLIPMlp(self.out_dim, 4 * self.out_dim, activation_cls)
+		self.layer_norm1 = nn.LayerNorm(hidden_size)
+		self.layer_norm2 = nn.LayerNorm(self.out_dim)
+
+		if alt_style:
+			self.final_proj = nn.Linear(hidden_size, out_dim)
+		else:
+			self.final_proj = nn.Identity()
+	
+	def forward(self, hidden_states: torch.Tensor):
+		hidden_states = self.layer_norm1(hidden_states)
+		query_states = self.probe.unsqueeze(0).unsqueeze(0).expand(hidden_states.size(0), 1, -1)
+
+		hidden_states = self.self_attn(query_states=query_states, kv_states=hidden_states)
+		# We don't use a residual connection here because the out_dim is different from the hidden_size
+
+		residual = hidden_states
+		hidden_states = self.layer_norm2(hidden_states)
+		hidden_states = self.mlp(hidden_states)
+		hidden_states = hidden_states + residual
+		hidden_states = self.final_proj(hidden_states)
+
+		return hidden_states.squeeze(1)
+
+
+class GAPHead(nn.Module):
+	def __init__(self, hidden_size: int, out_dim: int):
+		super().__init__()
+
+		self.norm = nn.LayerNorm(hidden_size)
+		self.proj = nn.Linear(hidden_size, out_dim)
+
+	def forward(self, x: torch.Tensor) -> torch.Tensor:
+		x = x.mean(dim=1)
+		x = self.norm(x)
+		x = self.proj(x)
+		return x
+
+
+class CLIPLikeModel(VisionModel):
+	def __init__(
+		self,
+		n_tags: int,
+		embedding_dim: int,
+		num_attention_heads: int,
+		activation_cls,
+		num_channels: int,
+		image_size: int,
+		patch_size: int,
+		patch_dropout: float,
+		use_palm_alt: bool,
+		num_layers: int,
+		use_mha_alt: bool,
+		loss_type: str,
+		good_dropout: bool=False,
+		dpn: bool=False,
+		sine_positional_embeddings: bool=False,
+		norm_qk: bool = False,
+		no_wd_bias: bool = False,
+		use_gap_head: bool = False,
+		skip_init: Optional[float] = None,
+		stochastic_depth: Optional[float] = None,
+	):
+		super().__init__(image_size, n_tags)
+
+		out_dim = n_tags
+		self.n_tags = n_tags
+		self.loss_type = loss_type
+		self.no_wd_bias = no_wd_bias
+		
+		stochastic_depth_space = torch.linspace(0, stochastic_depth, num_layers) if stochastic_depth is not None else None
+
+		self.embedding_layer = CLIPEmbeddingLayer(embedding_dim, num_channels, image_size, patch_size, patch_dropout, good_dropout, dpn, sine_positional_embeddings)
+		self.pre_layer_norm = nn.LayerNorm(embedding_dim)
+		self.encoder_layers = nn.ModuleList([FastCLIPEncoderLayer(
+			hidden_size=embedding_dim,
+			num_attention_heads=num_attention_heads,
+			out_seq_len=None,
+			activation_cls=activation_cls,
+			use_palm_alt=use_palm_alt,
+			norm_qk=norm_qk,
+			skip_init=skip_init,
+			stochastic_depth=stochastic_depth_space[i].item() if stochastic_depth_space is not None else None,
+		) for i in range(num_layers)])
+
+		if use_gap_head:
+			self.pooling_head = GAPHead(embedding_dim, out_dim)
+		else:
+			self.pooling_head = MHAPoolingHead(embedding_dim, num_attention_heads, activation_cls, out_dim, use_mha_alt, norm_qk=norm_qk)
+	
+	def forward(self, batch):
+		hidden_states = self.embedding_layer(batch['image'])
+		hidden_states = self.pre_layer_norm(hidden_states)
+
+		for layer in self.encoder_layers:
+			hidden_states = layer(hidden_states)
+		
+		preds = self.pooling_head(hidden_states)
+
+		result = {
+			'tags': preds,
+		}
+
+		return result
+	
+	def calculate_loss(self, preds, batch, pos_weight):
+		return basic_calculate_loss(preds, batch, pos_weight, self.loss_type)
+	
+	def get_optimized_parameters(self, lr: float):
+		if self.no_wd_bias:
+			return self.get_optimized_parameters_no_wd_bias()
+		else:
+			return self.parameters()
+	
+	def get_optimized_parameters_no_wd_bias(self):
+		decay = []
+		no_decay = []
+
+		for name, param in self.named_parameters():
+			if not param.requires_grad:
+				continue
+
+			if len(param.shape) == 1 or name.endswith(".bias"):
+				no_decay.append(param)
+				print(f'No decay: {name}')
+			else:
+				decay.append(param)
+		
+		return [
+			{'params': decay},
+			{'params': no_decay, 'weight_decay': 0.},
+		]
+	
+	def save(self):
+		return self.state_dict()
+	
+	def load(self, state_dict):
+		self.load_state_dict(state_dict)
+
+
+class MaskedAutoEncoderViT(nn.Module):
+	def __init__(
+		self,
+		n_tags: int,
+
+		embedding_dim: int,
+		num_attention_heads: int,
+		activation_cls,
+		num_channels: int,
+		image_size: int,
+		patch_size: int,
+		num_layers: int,
+		loss_type: str,
+		sine_positional_embeddings: bool=False,
+
+		decoder_embedding_dim: int = 512,
+		decoder_num_attention_heads: int = 8,
+		decoder_num_layers: int = 6,
+		decoder_force_projection: bool = False,
+
+		masking_ratio: float = 0.75,
+		mae_loss_weight: float = 1.0,
+		mae_normalize_targets: bool = False,
+		mae_post_norm: bool = False,
+	):
+		super().__init__()
+
+		self.n_tags = n_tags
+		self.seq_len = (image_size // patch_size) ** 2
+		self.embedding_dim = embedding_dim
+		self.decoder_embedding_dim = decoder_embedding_dim
+		self.sine_positional_embeddings = sine_positional_embeddings
+		self.image_size = image_size
+		self.patch_size = patch_size
+		self.masking_ratio = masking_ratio
+		self.loss_type = loss_type
+		self.mae_loss_weight = mae_loss_weight
+		self.mae_normalize_targets = mae_normalize_targets
+
+		if not self.sine_positional_embeddings:
+			self.positional_embeddings = nn.Embedding(self.seq_len, embedding_dim)
+			self.decoder_positional_embeddings = nn.Embedding(self.seq_len, decoder_embedding_dim)
+		self.register_buffer("position_ids", torch.arange(self.seq_len))
+
+		self.to_patches = Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1=patch_size, p2=patch_size)
+		self.patch_embedder = nn.Linear(num_channels * patch_size * patch_size, embedding_dim)
+
+		# Encoder
+		self.pre_layer_norm = nn.LayerNorm(embedding_dim)
+		self.encoder_layers = nn.ModuleList([FastCLIPEncoderLayer(
+			hidden_size=embedding_dim,
+			num_attention_heads=num_attention_heads,
+			out_seq_len=None,
+			activation_cls=activation_cls,
+			use_palm_alt=True,
+			norm_qk=False,
+			skip_init=None,
+		) for _ in range(num_layers)])
+
+		# Head for classification
+		self.pooling_head = GAPHead(embedding_dim, n_tags)
+
+		# Decoder
+		if embedding_dim != decoder_embedding_dim or decoder_force_projection:
+			self.encoder_to_decoder_proj = nn.Linear(embedding_dim, decoder_embedding_dim)
+		else:
+			self.encoder_to_decoder_proj = nn.Identity()
+		self.decoder_pre_layer_norm = nn.LayerNorm(decoder_embedding_dim)
+		self.decoder_layers = nn.ModuleList([FastCLIPEncoderLayer(
+			hidden_size=decoder_embedding_dim,
+			num_attention_heads=decoder_num_attention_heads,
+			out_seq_len=None,
+			activation_cls=activation_cls,
+			use_palm_alt=True,
+			norm_qk=False,
+			skip_init=None,
+		) for _ in range(decoder_num_layers)])
+
+		if mae_post_norm:
+			self.decoder_to_pixel_values = nn.Sequential(
+				nn.LayerNorm(decoder_embedding_dim),
+				nn.Linear(decoder_embedding_dim, num_channels * patch_size * patch_size)
+			)
+		else:
+			self.decoder_to_pixel_values = nn.Linear(decoder_embedding_dim, num_channels * patch_size * patch_size)
+		self.mask_token = nn.Parameter(torch.zeros(decoder_embedding_dim))
+		torch.nn.init.normal_(self.mask_token, std=0.02)
+
+	def forward(self, batch):
+		pixel_values = batch['image']
+		device = pixel_values.device
+		B, C, H, W = pixel_values.shape
+		assert H % self.patch_size == 0, f"Input image height ({H}) needs to be divisible by the patch size ({self.patch_size})."
+		assert W % self.patch_size == 0, f"Input image width ({W}) needs to be divisible by the patch size ({self.patch_size})."
+
+		# Convert image to patches (B, seq_len, C * patch_size * patch_size)
+		patches = self.to_patches(pixel_values)
+		seq_len = patches.shape[1]
+		num_masked = int(self.masking_ratio * seq_len)
+
+		# For each batch tensor, use argsort to convert the random numbers into a permutation of the patch indices
+		# From this we can get the masked and unmasked indices
+		patch_mask = torch.rand(B, seq_len, device=device)
+		patch_mask = torch.argsort(patch_mask, dim=1)
+		masked_indices, unmasked_indices = patch_mask[:, :num_masked], patch_mask[:, num_masked:]
+		batch_range = torch.arange(B, device=device)[:, None]
+
+		# Masked and unmasked patches
+		unmasked_patches = patches[batch_range, unmasked_indices]
+		masked_patches = patches[batch_range, masked_indices]
+
+		# Embed unmasked patches for the encoder (B, seq_len, embedding_dim)
+		tokens = self.patch_embedder(unmasked_patches)
+
+		if self.sine_positional_embeddings:
+			position_embeddings = sinusoidal_position_embedding(W // self.patch_size, H // self.patch_size, self.embedding_dim, pixel_values.dtype, device)
+			decoder_position_embeddings = sinusoidal_position_embedding(W // self.patch_size, H // self.patch_size, self.decoder_embedding_dim, pixel_values.dtype, device)
+		else:
+			position_embeddings = self.positional_embeddings(self.position_ids)
+			decoder_position_embeddings = self.decoder_positional_embeddings(self.position_ids)
+		
+		# Add position embeddings
+		tokens = tokens + position_embeddings[unmasked_indices]
+
+		# Run the encoder
+		encoded_tokens = self.pre_layer_norm(tokens)
+
+		for layer in self.encoder_layers:
+			encoded_tokens = layer(encoded_tokens)
+		
+		# Label predictions
+		if self.training:
+			preds = self.pooling_head(encoded_tokens)
+		else:
+			# During inference, classify using the entire image
+			# But we'll do the usual for the MAE part, just so we can see how MAE is performing during validation
+			tokens = self.patch_embedder(patches)
+			tokens = tokens + position_embeddings
+			tokens = self.pre_layer_norm(tokens)
+			for layer in self.encoder_layers:
+				tokens = layer(tokens)
+			preds = self.pooling_head(tokens)
+		
+		# Projection for the decoder and position embeddings
+		decoder_tokens = self.encoder_to_decoder_proj(encoded_tokens)
+		decoder_tokens = decoder_tokens + decoder_position_embeddings[unmasked_indices]
+
+		# Fill in the masked patches
+		mask_tokens = einops.repeat(self.mask_token, 'd -> b n d', b = B, n = num_masked)
+		mask_tokens = mask_tokens + decoder_position_embeddings[masked_indices]
+		decoder_tokens = torch.cat([decoder_tokens, mask_tokens], dim=1)
+
+		# Run the decoder
+		decoded_tokens = self.decoder_pre_layer_norm(decoder_tokens)
+		
+		for layer in self.decoder_layers:
+			decoded_tokens = layer(decoded_tokens)
+
+		# Only predict the masked patches
+		# All the masked patches are at the end of the sequence
+		decoded_tokens = decoded_tokens[:, -num_masked:]
+		pred_pixel_values = self.decoder_to_pixel_values(decoded_tokens)
+
+		# Calculate the mae loss
+		if self.mae_normalize_targets:
+			# Normalize each patch by its mean and variance. The ViCHA paper says this provides better results
+			means = masked_patches.mean(dim=-1, keepdim=True)
+			vars = masked_patches.var(dim=-1, keepdim=True)
+			target = (masked_patches - means) / (vars + 1e-6)**0.5
+			mae_loss = F.mse_loss(pred_pixel_values, target)
+		else:
+			mae_loss = F.mse_loss(pred_pixel_values, masked_patches)
+		mae_loss = mae_loss * self.mae_loss_weight
+
+		return {
+			'tags': preds,
+			'mae_loss': mae_loss,
+		}
+
+	def calculate_loss(self, preds, batch, pos_weight):
+		return basic_calculate_loss(preds, batch, pos_weight, self.loss_type) + preds['mae_loss']
+	
+	def get_optimized_parameters(self, lr: float):
+		return self.parameters()
+	
+	def save(self):
+		return self.state_dict()
+	
+	def load(self, state_dict):
+		self.load_state_dict(state_dict)
+
+
+class StochDepth(nn.Module):
+	def __init__(self, drop_rate: float, scale_by_keep: bool = False):
+		super().__init__()
+		self.drop_rate = drop_rate
+		self.scale_by_keep = scale_by_keep
+	
+	def forward(self, x):
+		if not self.training:
+			return x
+		
+		batch_size = x.shape[0]
+		r = torch.rand((batch_size, 1, 1), device=x.device)
+		keep_prob = 1 - self.drop_rate
+		binary_tensor = torch.floor(keep_prob + r)
+		if self.scale_by_keep:
+			x = x / keep_prob
+		
+		return x * binary_tensor
+
+
+class SkipInitChannelwise(nn.Module):
+	def __init__(self, channels, init_val=1e-6):
+		super().__init__()
+		self.channels = channels
+		self.init_val = init_val
+		self.skip = nn.Parameter(torch.ones(channels) * init_val)
+	
+	def forward(self, x):
+		return x * self.skip
+
+
+class PosEmbedding(nn.Module):
+	def __init__(self, d_model: int, max_len: int, use_sine: bool, patch_size: int):
+		super().__init__()
+		self.d_model = d_model
+		self.max_len = max_len
+		self.use_sine = use_sine
+		self.patch_size = patch_size
+
+		if not self.use_sine:
+			self.embedding = nn.Embedding(max_len, d_model)
+			nn.init.trunc_normal_(self.embedding.weight, std=0.02)
+			self.register_buffer("position_ids", torch.arange(max_len))
+	
+	def forward(self, x, width: int, height: int):
+		if self.use_sine:
+			position_embeddings = sinusoidal_position_embedding(width // self.patch_size, height // self.patch_size, self.d_model, x.dtype, x.device)
+		else:
+			position_embeddings = self.embedding(self.position_ids)
+		
+		return x + position_embeddings
+
+
+class MLPBlock(nn.Module):
+	def __init__(self, d_model: int, d_ff: int, stochdepth_rate: float):
+		super().__init__()
+		self.linear1 = nn.Linear(d_model, d_ff)
+		self.linear2 = nn.Linear(d_ff, d_model)
+		self.activation = nn.GELU()
+		if stochdepth_rate > 0:
+			self.stochdepth = StochDepth(stochdepth_rate, scale_by_keep=True)
+		else:
+			self.stochdepth = None
+	
+	def forward(self, x):
+		x = self.linear1(x)
+		x = self.activation(x)
+		if self.stochdepth is not None:
+			x = self.stochdepth(x)
+		x = self.linear2(x)
+		return x
+
+
+class ViTBlock(nn.Module):
+	def __init__(self, num_heads: int, d_model: int, d_ff: int, layerscale_init: float, stochdepth_rate: float):
+		super().__init__()
+		self.num_heads = num_heads
+		self.d_model = d_model
+
+		assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
+
+		# MHA
+		self.norm1 = nn.LayerNorm(d_model)
+		self.qkv_proj = nn.Linear(d_model, d_model * 3)
+		self.out_proj = nn.Linear(d_model, d_model)
+		self.skip_init1 = SkipInitChannelwise(channels=d_model, init_val=layerscale_init)
+		self.stochdepth1 = StochDepth(stochdepth_rate, scale_by_keep=True) if stochdepth_rate > 0 else None
+		
+		# MLP
+		self.norm2 = nn.LayerNorm(d_model)
+		self.mlp = MLPBlock(d_model, d_ff, stochdepth_rate)
+		self.skip_init2 = SkipInitChannelwise(channels=d_model, init_val=layerscale_init)
+		self.stochdepth2 = StochDepth(stochdepth_rate, scale_by_keep=True) if stochdepth_rate > 0 else None
+	
+	def forward(self, x):
+		bsz, src_len, embed_dim = x.shape
+
+		out = x
+		out = self.norm1(out)
+
+		# MHA
+		qkv_states = self.qkv_proj(out).split(self.d_model, dim=-1)
+		q_states = qkv_states[0].view(bsz, src_len, self.num_heads, embed_dim // self.num_heads).transpose(1, 2)  # (bsz, num_heads, src_len, embed_dim // num_heads)
+		k_states = qkv_states[1].view(bsz, src_len, self.num_heads, embed_dim // self.num_heads).transpose(1, 2)  # (bsz, num_heads, src_len, embed_dim // num_heads)
+		v_states = qkv_states[2].view(bsz, src_len, self.num_heads, embed_dim // self.num_heads).transpose(1, 2)  # (bsz, num_heads, src_len, embed_dim // num_heads)
+
+		with torch.backends.cuda.sdp_kernel(enable_math=False):
+			out = F.scaled_dot_product_attention(q_states, k_states, v_states)   # (bsz, num_heads, tgt_len, head_dim)
+			out = out.transpose(1, 2).contiguous().view(bsz, src_len, embed_dim)   # (bsz, tgt_len, embed_dim)
+		
+		out = self.out_proj(out)
+
+		out = self.skip_init1(out)
+		if self.stochdepth1 is not None:
+			out = self.stochdepth1(out)
+		x = out + x
+
+		out = self.norm2(x)
+		out = self.mlp(out)
+		out = self.skip_init2(out)
+		if self.stochdepth2 is not None:
+			out = self.stochdepth2(out)
+		
+		out = out + x
+
+		return out
+
+
+def CaiT_LayerScale_init(network_depth):
+	if network_depth <= 18:
+		return 1e-1
+	elif network_depth <= 24:
+		return 1e-5
+	else:
+		return 1e-6
+
+
+class CNNLayerNorm(nn.Module):
+	def __init__(self, d_model: int):
+		super().__init__()
+		self.norm = nn.LayerNorm(d_model)
+	
+	def forward(self, x: torch.Tensor) -> torch.Tensor:
+		x = x.transpose(1, 3)
+		x = self.norm(x)
+		x = x.transpose(1, 3)
+		return x
+
+
+class CNNStem(nn.Module):
+	def __init__(self, config: str):
+		super().__init__()
+		self.config = config
+
+		layers = []
+		channels = 3
+
+		for line in config.split(";"):
+			ty, line = line.split(":") if ":" in line else (line, "")
+			options = line.split(",")
+			options = [o.split("=") for o in options] if line else []
+			options = {k: v for k, v in options}
+
+			if ty == 'conv':
+				layers.append(nn.Conv2d(
+					in_channels=channels,
+					out_channels=int(options['c']),
+					kernel_size=int(options['k'] if 'k' in options else 3),
+					stride=int(options['s'] if 's' in options else 2),
+					bias=True,
+					padding=int(options['p'] if 'p' in options else 1),
+				))
+				channels = int(options['c'])
+			elif ty == 'bn':
+				layers.append(nn.BatchNorm2d(channels))
+			elif ty == 'ln':
+				layers.append(CNNLayerNorm(channels))
+			elif ty == 'relu':
+				layers.append(nn.ReLU())
+			elif ty == 'gelu':
+				layers.append(nn.GELU())
+
+		self.conv = nn.Sequential(*layers)
+	
+	def forward(self, x: torch.Tensor) -> torch.Tensor:
+		return self.conv(x)
+
+
+class ViT(VisionModel):
+	def __init__(self,
+		n_tags: int,
+		image_size: int,
+		num_blocks: int,
+		patch_size: int,
+		d_model: int,
+		mlp_dim: int,
+		num_heads: int,
+		stochdepth_rate: float,
+		use_sine: bool,
+		loss_type: str,
+		layerscale_init: Optional[float] = None,
+		head_mean_after: bool = False,
+		cnn_stem: str | None = None,
+		patch_dropout: float = 0.0,
+	):
+		super().__init__(image_size, n_tags)
+
+		#assert image_size % patch_size == 0, "image_size must be divisible by patch_size"
+		assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
+
+		out_dim = n_tags
+		self.n_tags = n_tags
+		self.loss_type = loss_type
+		self.patch_size = patch_size
+		self.head_mean_after = head_mean_after
+		self.patch_dropout = patch_dropout
+
+		layerscale_init = CaiT_LayerScale_init(num_blocks) if layerscale_init is None else layerscale_init
+		self.patch_embeddings = nn.Conv2d(
+			in_channels=3,
+			out_channels=d_model,
+			kernel_size=patch_size,
+			stride=patch_size,
+			bias=True,
+		) if cnn_stem is None else CNNStem(cnn_stem)
+		self.pos_embedding = PosEmbedding(d_model, (image_size // patch_size) ** 2, use_sine=use_sine, patch_size=patch_size)
+
+		self.blocks = nn.ModuleList([
+			ViTBlock(num_heads, d_model, mlp_dim, layerscale_init, stochdepth_rate)
+			for _ in range(num_blocks)
+		])
+
+		self.norm = nn.LayerNorm(d_model)
+		self.head = nn.Linear(d_model, out_dim)
+
+	def forward(self, batch, return_embeddings=False, return_loss: bool = False, pos_weight = None):
+		B, C, H, W = batch['image'].shape
+		assert H % self.patch_size == 0, f"Input image height ({H}) needs to be divisible by the patch size ({self.patch_size})."
+		assert W % self.patch_size == 0, f"Input image width ({W}) needs to be divisible by the patch size ({self.patch_size})."
+
+		x = self.patch_embeddings(batch['image'])  # (bsz, d_model, patch_num, patch_num)
+		x = x.flatten(2).transpose(1, 2)  # (bsz, patch_num ** 2, d_model)
+		x = self.pos_embedding(x, W, H)   # (bsz, patch_num ** 2, d_model)
+
+		# Patch dropout
+		seq_len = x.shape[1]
+		patch_dropout = int(math.ceil((1.0 - self.patch_dropout) * seq_len))
+
+		if patch_dropout != seq_len:
+			# Generate a matrix of random numbers between 0 and 1 of shape (B, seq_len)
+			patch_mask = torch.rand(B, seq_len, device=x.device)
+			# For each batch tensor, use argsort to convert the random numbers into a permutation of the patch indices
+			patch_mask = torch.argsort(patch_mask, dim=1)
+			# Truncate
+			patch_mask = patch_mask[:, :patch_dropout]
+
+			x = x.gather(1, patch_mask.unsqueeze(-1).expand(-1, -1, x.shape[-1]))
+
+			#indices = torch.randperm(seq_len, device=x.device)[:patch_dropout]
+			#x = x[:, indices, :]
+
+		# Transformer
+		for block in self.blocks:
+			x = block(x)
+		
+		# Head
+		result = {}
+
+		x = self.norm(x)
+		if self.head_mean_after:
+			x = self.head(x)
+			x = x.mean(dim=1)
+		else:
+			x = x.mean(dim=1)
+			if return_embeddings:
+				result['embeddings'] = x
+			x = self.head(x)
+
+		result['tags'] = x
+
+		if return_loss:
+			result['loss'] = self.calculate_loss(result, batch, pos_weight)
+
+		return result
+	
+	def calculate_loss(self, preds, batch, pos_weight):
+		return basic_calculate_loss(preds, batch, pos_weight, self.loss_type)
+	
+	def get_optimized_parameters(self, lr: float):
+		return self.parameters()
+	
+	def save(self):
+		return self.state_dict()
+	
+	def load(self, state_dict):
+		if 'head.weight' in state_dict and 'head.bias' in state_dict and state_dict['head.weight'].shape[0] == (self.n_tags + 9):
+			# Support old models which included 3 rating and 6 score dimensions
+			state_dict['head.weight'] = state_dict['head.weight'][:self.n_tags]
+			state_dict['head.bias'] = state_dict['head.bias'][:self.n_tags]
+
+		self.load_state_dict(state_dict)
+
+
+from PIL import Image
+import torchvision.transforms.functional as TVF
+from torchvision import transforms
+
+def prepare_image(image: Image.Image, target_size: int) -> torch.Tensor:
+	# Pad image to square
+	image_shape = image.size
+	max_dim = max(image_shape)
+	pad_left = (max_dim - image_shape[0]) // 2
+	pad_top = (max_dim - image_shape[1]) // 2
+
+	padded_image = Image.new('RGB', (max_dim, max_dim), (255, 255, 255))
+	padded_image.paste(image, (pad_left, pad_top))
+
+	# Resize image
+	if max_dim != target_size:
+		padded_image = padded_image.resize((target_size, target_size), Image.BICUBIC)
+	
+	# Convert to tensor
+	image_tensor = TVF.pil_to_tensor(padded_image) / 255.0
+
+	# Normalize
+	image_tensor = TVF.normalize(image_tensor, mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
+
+	return image_tensor
+
+
+@torch.no_grad()
+def predict(image: Image.Image, model, top_tags, THRESHOLD=0.4):
+	image = transforms.ToPILImage()(image.permute(2, 0, 1))
+	image_tensor = prepare_image(image, model.image_size)
+	batch = {
+		'image': image_tensor.unsqueeze(0).to('cuda'),
+	}
+
+	with torch.amp.autocast_mode.autocast('cuda', enabled=True):
+		preds = model(batch)
+		tag_preds = preds['tags'].sigmoid().cpu()
+	
+	scores = {top_tags[i]: tag_preds[0][i] for i in range(len(top_tags))}
+	predicted_tags = [tag for tag, score in scores.items() if score > THRESHOLD]
+	tag_string = ', '.join(predicted_tags)
+
+	return tag_string, scores

custom_nodes/ComfyUI-N-Nodes/libs/migrate.py

@@ -0,0 +1,43 @@
+import sys
+import os
+
+def migrate_workflow(input_file_path):
+    try:
+        file_name, file_extension = os.path.splitext(input_file_path)
+        
+        output_file_path = f"{file_name}_migrated.json"
+
+        pre_list = ('LoadVideo', 'SaveVideo','FrameInterpolator', 'LoadFramesFromFolder','SetMetadataForSaveVideo','GPT Loader Simple','GPTSampler','String Variable','Integer Variable','Float Variable','DynamicPrompt')
+        post_list= ('LoadVideo [n-suite]', 'SaveVideo [n-suite]','FrameInterpolator [n-suite]', 'LoadFramesFromFolder [n-suite]','SetMetadataForSaveVideo [n-suite]','GPT Loader Simple [n-suite]','GPT Sampler [n-suite]','String Variable [n-suite]','Integer Variable [n-suite]','Float Variable [n-suite]','DynamicPrompt [n-suite]')
+        replacements = list(zip(pre_list, post_list))
+
+        with open(input_file_path, 'r') as input_file:
+            content = input_file.read()
+              
+            # s&r  
+            for old, new in replacements:
+                content = content.replace(f'"Node name for S&R": "{old}"', f'"Node name for S&R": "{new}"')
+            #type
+            for old, new in replacements:
+                content = content.replace(f'"type": "{old}"', f'"type": "{new}"')
+
+        with open(output_file_path, 'w') as output_file:
+            output_file.write(content)
+        
+        print("Replacement completed successfully.")
+    except Exception as e:
+        print(f"An error occurred: {str(e)}")
+
+if __name__ == "__main__":
+    print(len(sys.argv))
+    if len(sys.argv) != 2:
+        print("Error: Provide the path of the text file to migrate.")
+        sys.exit(1)
+    
+    file_path = sys.argv[1]
+    
+    if not os.path.isfile(file_path):
+        print(f"Error: The file {file_path} does not exist.")
+        sys.exit(1)
+    
+    migrate_workflow(file_path)

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/.gitignore

@@ -0,0 +1,2 @@
+.venv
+__pycache__

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/README.md

@@ -0,0 +1,68 @@
+## commit 38af98596e59f2a6c25c6b52b2bd5a672dab4144
+
+# 🌔 moondream
+
+a tiny vision language model that kicks ass and runs anywhere
+
+## moondream1
+
+1.6B parameter model built using SigLIP, Phi-1.5 and the LLaVA training dataset.
+Weights are licensed under CC-BY-SA due to using the LLaVA dataset. Try it out
+on [Huggingface Spaces](https://huggingface.co/spaces/vikhyatk/moondream1)!
+
+**Benchmarks**
+
+| Model | Parameters | VQAv2 | GQA | VizWiz | TextVQA |
+| --- | --- | --- | --- | --- | --- |
+| LLaVA-1.5 | 13.3B | 80.0 | 63.3 | 53.6 | 61.3 |
+| LLaVA-1.5 | 7.3B | 78.5 | 62.0 | 50.0 | 58.2 |
+| [MC-LLaVA-3B](https://huggingface.co/visheratin/MC-LLaVA-3b) | 3B | 64.2 | 49.6 | 24.9 | 38.6 |
+| [LLaVA-Phi](https://arxiv.org/pdf/2401.02330.pdf) | 3B | 71.4 | - | 35.9 | 48.6 |
+| **moondream1** | 1.6B | 74.3 | 56.3 | 30.3 | 39.8 |
+
+**Examples**
+
+| Image | Examples |
+| --- | --- |
+| ![](assets/demo-1.jpg) | **What is the title of this book?**<br>The Little Book of Deep Learning<br><br>**What can you tell me about this book?**<br>The book appears to be a white booklet titled "The Little Book of Deep Learning." It is held in a person's hand, and it seems to be a personal possession. The book's content focuses on the basics of deep learning, which is a field of artificial intelligence that uses neural networks to process and analyze data. It is likely that the book provides an introduction to the concepts and techniques involved in deep learning, making it accessible for beginners and helping them understand the fundamentals of this advanced machine learning approach.<br><br>**Where is the person standing?**<br>The person is standing on a balcony or a deck, which is located outside the house. |
+| ![](assets/demo-2.jpg) | **What type of food is the girl holding?**<br>The girl is holding a large hamburger or burger, which is a type of sandwich made from ground meat, typically consisting of a beef patty, and is usually served between two slices of bread.<br><br>**What color is the woman's hair?**<br>The woman's hair is white.<br><br>**What is the girl doing in the image?**<br>The girl in the image is eating a hamburger. |
+| ![](assets/demo-3.jpg) | **What color is the train?**<br>The train is red.<br><br>**What is the bus doing in the image?**<br>The bus is driving down a street, passing through an intersection, and traveling on a train track.<br><br>**What kind of public transportation is in the image?**<br>The image features a red trolley or commuter train on a city street, which is a form of public transportation. |
+| ![](assets/demo-4.jpg) | **What is the girl looking at?**<br>The girl is looking at her reflection in the mirror while adjusting her uniform.<br><br>**What color is the woman's hair?**<br>The woman's hair is brown.<br><br>**What is the woman doing in the mirror?**<br>The woman is adjusting her hair in the mirror. |
+| ![](assets/demo-5.jpg) | **What might be the reason for the dog's aggressive behavior?**<br>The dog's aggressive behavior, with its teeth bared and growling, could be due to several reasons. It is possible that the dog is feeling threatened, scared, or defensive in its current environment, such as a room with a person it doesn't know well or a situation that provokes it. Alternatively, the dog might be reacting to a perceived threat or discomfort from the person holding it. It is essential to assess the situation and the dog's body language to determine the exact cause of its aggressive behavior and respond accordingly to ensure the safety and well-being of both the dog and the person involved.<br><br>**What kind of dog is in the picture?**<br>The picture features a small dog, possibly a Chihuahua, with red eyes and a mean, hungry-looking expression.<br><br>**What color is the dog?**<br>The dog is black and white. |
+
+**Usage**
+
+Clone this repository and install the dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+Use the `sample.py` script to run the model on CPU:
+
+```bash
+python sample.py --image [IMAGE_PATH] --prompt [PROMPT]
+```
+
+When the `--prompt` argument is not provided, the script will allow you to ask
+questions interactively.
+
+**Gradio demo**
+
+Use the `gradio_demo.py` script to run the gradio app:
+
+```python
+python gradio_demo.py
+```
+
+
+**Limitations**
+
+* The model may generate inaccurate statements.
+* It may struggle to adhere to intricate or nuanced instructions.
+* It is primarily designed to understand English. Informal English, slang, and
+  non-English languages may not work well.
+* The model may not be free from societal biases. Users should be aware of this
+  and exercise caution and critical thinking when using the model.
+* The model may generate offensive, inappropriate, or hurtful content if it is
+  prompted to do so.

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/_sample.py

@@ -0,0 +1,26 @@
+import argparse
+from PIL import Image
+from moondream import VisionEncoder, TextModel
+from huggingface_hub import snapshot_download
+
+def main(image_path, prompt):
+    model_path = snapshot_download("vikhyatk/moondream1")
+    vision_encoder = VisionEncoder(model_path)
+    text_model = TextModel(model_path)
+    image = Image.open(image_path)
+    image_embeds = vision_encoder(image)
+
+    if prompt is None:
+        while True:
+            question = input("> ")
+            print(text_model.answer_question(image_embeds, question))
+    else:
+        print(">", prompt)
+        print(text_model.answer_question(image_embeds, prompt))
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--image", type=str, required=True)
+    parser.add_argument("--prompt", type=str, required=False)
+    args = parser.parse_args()
+    main(args.image, args.prompt)

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/gradio_demo.py

@@ -0,0 +1,36 @@
+import re
+import gradio as gr
+from moondream import VisionEncoder, TextModel
+from huggingface_hub import snapshot_download
+from threading import Thread
+from transformers import TextIteratorStreamer
+
+model_path = snapshot_download("vikhyatk/moondream1")
+vision_encoder = VisionEncoder(model_path)
+text_model = TextModel(model_path)
+
+def moondream(img, prompt):
+    image_embeds = vision_encoder(img)
+    streamer = TextIteratorStreamer(text_model.tokenizer, skip_special_tokens=True)
+    thread = Thread(target=text_model.answer_question, kwargs={
+        "image_embeds": image_embeds, "question": prompt, "streamer": streamer})
+    thread.start()
+
+    buffer = ""
+    for new_text in streamer:
+        clean_text = re.sub("<$|END$", "", new_text)
+        buffer += clean_text
+        yield buffer.strip("<END")
+
+with gr.Blocks() as demo:
+    gr.Markdown("# 🌔 moondream \n ### A tiny vision language model. [GitHub](https://github.com/vikhyat/moondream)")
+    with gr.Row():
+        prompt = gr.Textbox(label='Input Prompt', placeholder='Type here...', scale=4)
+        submit = gr.Button('Submit')
+    with gr.Row():
+        img = gr.Image(type='pil', label='Upload an Image')
+        output = gr.TextArea(label="Response", info='Please wait for a few seconds..')
+    submit.click(moondream, [img, prompt], output)
+    prompt.submit(moondream, [img, prompt], output)
+
+demo.queue().launch(debug=True)

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/moondream/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/moondream/README.md

@@ -0,0 +1,6 @@
+---
+language:
+- en
+---
+
+Check out the [moondream repository on GitHub](https://github.com/vikhyat/moondream) for inference code and other details.

custom_nodes/ComfyUI-N-Nodes/libs/moondream_repo/moondream/__init__.py

@@ -0,0 +1,2 @@
+from .vision_encoder import VisionEncoder
+from .text_model import TextModel