initial model

README.md

@@ -1,3 +1,141 @@
 ---
 license: mit
 ---
+
+# Tau LLM Unity ML Agents Project
+
+Welcome to the Tau LLM Unity ML Agents Project repository! This project focuses on training reinforcement learning agents using Unity ML-Agents and the PPO algorithm. Our goal is to optimize the performance of the agents through various configurations and training runs.
+
+## Project Overview
+
+This repository contains the code and configurations for training agents in a Unity environment using the Proximal Policy Optimization (PPO) algorithm. The agents are designed to learn and adapt to their environment, improving their performance over time.
+
+### Key Features
+
+- **Reinforcement Learning**: Utilizes the PPO algorithm for training agents.
+- **Unity ML-Agents**: Integrates with Unity ML-Agents for a seamless training experience.
+- **Custom Reward Functions**: Implements gradient-based reward functions for nuanced feedback.
+- **Memory Networks**: Incorporates memory networks to handle temporal dependencies.
+- **TensorBoard Integration**: Monitors training progress and performance using TensorBoard.
+
+## Configuration
+
+Below is the configuration used for training the agents:
+
+```yaml
+behaviors:
+  TauAgent:
+    trainer_type: ppo
+    hyperparameters:
+      batch_size: 256
+      buffer_size: 4096
+      learning_rate: 0.00003
+      beta: 0.005
+      epsilon: 0.2
+      lambd: 0.95
+      num_epoch: 10
+      learning_rate_schedule: linear
+    network_settings:
+      normalize: true
+      hidden_units: 256
+      num_layers: 4
+      vis_encode_type: simple
+      memory:
+        memory_size: 256
+        sequence_length: 256
+        num_layers: 4
+    reward_signals:
+      extrinsic:
+        gamma: 0.99
+        strength: 1.0
+      curiosity:
+        gamma: 0.995
+        strength: 0.1
+        network_settings:
+          normalize: true
+          hidden_units: 256
+          num_layers: 4
+          learning_rate: 0.00003
+    keep_checkpoints: 10
+    checkpoint_interval: 100000
+    threaded: true
+    max_steps: 3000000
+    time_horizon: 256
+    summary_freq: 10000
+```
+
+## Model Naming Convention
+
+The models in this repository follow the naming convention `Tau_<series>_<max_steps>`. This helps in easily identifying the series and the number of training steps for each model.
+
+## Getting Started
+
+### Prerequisites
+
+- Unity 6
+- Unity ML-Agents Toolkit
+- Python 3.10.11
+- PyTorch
+- Transformers
+
+### Installation
+
+1. Clone the repository:
+   ```bash
+   git clone https://github.com/yourusername/tau-llm-unity-ml-agents.git
+   cd tau-llm-unity-ml-agents
+   ```
+
+2. Install the required Python packages:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+3. Open the Unity project:
+   - Launch Unity Hub and open the project folder.
+
+### Training the Agent
+
+To start training the agent, run the following command:
+```bash
+mlagents-learn config/trainer_config.yaml --run-id=run1
+```
+
+### Monitoring Training
+
+You can monitor the training progress using TensorBoard:
+```bash
+tensorboard --logdir=results --port=6006
+```
+
+## Results
+
+The training results, including the average reward and cumulative reward, can be visualized using TensorBoard. The graphs below show the performance of the agent over time:
+
+![Average Reward](path/to/average_reward.png)
+![Cumulative Reward](path/to/cumulative_reward.png)
+
+## Citation
+
+If you use this project in your research, please cite it as follows:
+
+```bibtex
+@misc{Tau,
+  author = {K. Rawson},
+  title = {Tau LLM Unity ML Agents Project},
+  year = {2024},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/p3nGu1nZz/Tau}},
+}
+```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- Unity ML-Agents Toolkit
+- TensorFlow and PyTorch communities
+- Hugging Face for hosting the model repository

results/tau_agent_A1_2M/Tau-A1-2M.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08931e19cffa93c14fed86e9bb88278424715303928d4761bf3dcc257fdde73d
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1199744.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b24d7a70f3f708362ccd3b35ccbf309d81696c379a5c2111810676ffda6c9c3d
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1199744.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f556a11f0fea58e2cc679cf2f9ad6e86403425ced6496f25188080f8f29bc8e
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1299958.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb6f5d1ee696b00963d7cb00a10b924fdf123f5eb46b618ba006117c7d843919
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1299958.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c0244046f46126c14c82f60ef46b799325fb0f35205cec04ccec9141784a93c
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1399744.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d0b419212303e89f05b6d735d04bb392166df4dd491fd0036ea2fce40a3abd6
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1399744.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c120a1fa8108b6b9558d9667fe949808b3e16394d499693e20392d2ea1f6c28e
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1499776.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f424ea2d9e633119050d04d95e1079bee5e8c3a1a9fee31282ca95855bd7d885
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1499776.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:912696d11a6837fc71783e311bed38195e1dda57fe4123a64141db5e96083ba3
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1599808.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56f777fafa9cc0919950a0231834f75954a17c4e07b9bcf7c6b2b3dbc5426c41
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1599808.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f311e89bfe1d1fa8a578efe91d9ceafece8fa50a349a0721013634ff0e664ef9
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1699840.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90533537979d9abeb815e499a777474c4c0c66e3068c3e9de39c17512f6cd35c
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1699840.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09f5ead3bb039e49e211a2ca8d7afa788223c1ed2b9883342efc46ef66799982
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1799808.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de1ea3ba5c8d90ce7be467ee2871441c2dbb220e8761d14b8c3d70439bc9ad7b
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1799808.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32def0f45b71e638ddd1ad302b620df2526e4099b0bc65c9f4b1ec7a2737b092
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1899840.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9677e46c8e368b0e5f5a3aa982ca3949bf1f4489fa58ae55cea8801e56563aba
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1899840.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9cacd7cdf76e019c35c7618a719b7c58d5b468f7a47d136dc4d1dcea7ede6b7
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-1999872.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5eb6959372271405f646cec449cddcc6d19f604a7d02b5422b02aa7035aa9906
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-1999872.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc7b6a20afe2601acf9523584f01e56a6a62f274dff5066b5b53ae4621953aa
+size 15534256

results/tau_agent_A1_2M/checkpoints/TauAgent-2005504.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08931e19cffa93c14fed86e9bb88278424715303928d4761bf3dcc257fdde73d
+size 2186395

results/tau_agent_A1_2M/checkpoints/TauAgent-2005504.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:260196c3491aca9156c57f3d29bba9cb40b9655acd53407f09075f191df035ae
+size 15534256

results/tau_agent_A1_2M/checkpoints/checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37080da1574efbcf39b5938261f36738c44437f2ceb72501058d86a7ffe8d386
+size 15533332

results/tau_agent_A1_2M/configuration.yaml

@@ -0,0 +1,93 @@
+default_settings: null
+behaviors:
+  TauAgent:
+    trainer_type: ppo
+    hyperparameters:
+      batch_size: 256
+      buffer_size: 4096
+      learning_rate: 3.0e-05
+      beta: 0.005
+      epsilon: 0.2
+      lambd: 0.95
+      num_epoch: 7
+      shared_critic: false
+      learning_rate_schedule: linear
+      beta_schedule: linear
+      epsilon_schedule: linear
+    checkpoint_interval: 100000
+    network_settings:
+      normalize: true
+      hidden_units: 256
+      num_layers: 4
+      vis_encode_type: simple
+      memory:
+        sequence_length: 256
+        memory_size: 256
+      goal_conditioning_type: hyper
+      deterministic: false
+    reward_signals:
+      extrinsic:
+        gamma: 0.99
+        strength: 1.0
+        network_settings:
+          normalize: false
+          hidden_units: 128
+          num_layers: 2
+          vis_encode_type: simple
+          memory: null
+          goal_conditioning_type: hyper
+          deterministic: false
+      curiosity:
+        gamma: 0.995
+        strength: 0.1
+        network_settings:
+          normalize: true
+          hidden_units: 256
+          num_layers: 4
+          vis_encode_type: simple
+          memory: null
+          goal_conditioning_type: hyper
+          deterministic: false
+        learning_rate: 0.0003
+        encoding_size: null
+    init_path: null
+    keep_checkpoints: 10
+    even_checkpoints: false
+    max_steps: 2000000
+    time_horizon: 256
+    summary_freq: 10000
+    threaded: true
+    self_play: null
+    behavioral_cloning: null
+env_settings:
+  env_path: .\Build
+  env_args: null
+  base_port: 5005
+  num_envs: 1
+  num_areas: 1
+  timeout_wait: 300
+  seed: -1
+  max_lifetime_restarts: 10
+  restarts_rate_limit_n: 1
+  restarts_rate_limit_period_s: 60
+engine_settings:
+  width: 84
+  height: 84
+  quality_level: 5
+  time_scale: 20
+  target_frame_rate: -1
+  capture_frame_rate: 60
+  no_graphics: false
+environment_parameters: null
+checkpoint_settings:
+  run_id: tau_agent_ppo_A1
+  initialize_from: null
+  load_model: false
+  resume: false
+  force: true
+  train_model: false
+  inference: false
+  results_dir: results
+torch_settings:
+  device: cuda
+debug: false

results/tau_agent_A3_1M/Tau-A3-1M.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5d20133e25c7b1f17c9fe045373f12448adfba0a341d2ce0ab683dc0a505e9
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-1001575.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5d20133e25c7b1f17c9fe045373f12448adfba0a341d2ce0ab683dc0a505e9
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-1001575.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c1e3f84038d1be138512d0c2b168f8fed07c9e123765c992dbb88ce19ad9729
+size 23269214

results/tau_agent_A3_1M/checkpoints/TauAgent-12324.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaf958d6b87edcf7ebcfbfe28296a04bbdb8f8a5fad6aa1b8f23bcf747cd89d1
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-12324.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c98eb2737900135b3e52175a3b2c94a6d3bd3c88d1b85667f32af788be5b6075
+size 23268710

results/tau_agent_A3_1M/checkpoints/TauAgent-199903.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f024ccd7748fde0b0b7e8fa59d06951bf15fcefb555457746ee03d2f7a90bbc
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-199903.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b77bbf92946f07c4ea7dba982ce78ad57c00ee5a9bca7edd7271d7918bbbda7
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-28282.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd48d742cff7e006aa71e8dcb2c31c31c74bc9ae03d54b76559c3b3dd8745c61
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-28282.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8513da1f1353113fd99c51dd85e61f57e4dfb4b8d31b644d0f7af73ac4c3b47e
+size 23268710

results/tau_agent_A3_1M/checkpoints/TauAgent-299879.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ff77dadbb9a1fafc12a4946bdd5c63f4e6f43fa1763cff4dd0929fa3d499a2b
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-299879.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b03039f2bdfbaa4625a6f13b8580c72c43bef5bfd9025e9f915e2bf3a51b8dfa
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-399831.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0f215196437e7d6a5e1757fb59a21b4ea1cda1a3b7af937a427659a375b9a0d
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-399831.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0bd14a4a2ab99d335f034d318f7c372770c62ea615a4e2804bd8994de6b8050
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-499989.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:187b76bb76c34cba0dac85f49f58a7ea5bde3e4a23f9bfc899e2023cdbba3e70
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-499989.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:782a3ce94817b5f93f82f245128eabd0e5944a6f1ce158921c7cb500cc53c1d2
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-599755.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:545ee682b7953e6ed53cb351c7e17d16fcd665bf5d9a9f3cb1e32452bceaa760
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-599755.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53a8c5e1897e7aa448878fc936a0b0968addbffe7e0b73cfa17999fd75de4aa6
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-699907.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f5612c9190f8d25f0b2e1ae27d72aca029f7962cd1d1c9c2605296389979b4
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-699907.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01a48a1e97204e86b79e57dde69e65a2e156df2159595e7583f69355a2726e41
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-799975.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3946bf8a6a03014256bedd19b4965ec552903ba24a711ac5b3ae5bfd96e18f82
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-799975.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50854837ea697627700f4b23582acd5083cbc41be015f5177bec6fa417a75141
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-899787.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3834f38a6da9d23a037c64da280ad1b87c77d9228ffc461ccc7c312a0f147c38
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-899787.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c4867e2a04b63b7412b26745df14b83b2a4465dfcd36b37018158fa2b685661
+size 23268962

results/tau_agent_A3_1M/checkpoints/TauAgent-999987.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f599e796aa05601ba2a7a5c54f4c702594601fa5c5d3861f60964129a4d4109
+size 1983173

results/tau_agent_A3_1M/checkpoints/TauAgent-999987.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d7607c7b79859970d0162c13f77a89730ac548341ae0438001612c2ab0745ab
+size 23268962

results/tau_agent_A3_1M/checkpoints/checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94b764243f51b496724081db5f3d7a11d270dc706ec1c5635abfc81dc20cfb0b
+size 23267702