WHAM Demonstrator and README

.gitattributes

@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 fonts/arial.ttf filter=lfs diff=lfs merge=lfs -text
 *.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,240 @@
+---
+datasets:
+- MSRTestOrg/bleeding-edge-gameplay-sample
+tags:
+- wham
+- microsoft
+language:
+- en
+license_link: LICENSE.md
+---
+# World and Human Action Model (WHAM)
+📄 [Paper](https://www.nature.com/articles/s41586-025-08600-3) • 🔗 [Sample Data](https://huggingface.co/datasets/microsoft/bleeding-edge-gameplay-sample)
+<div align="center">
+Anssi Kanervisto, Dave Bignell, Linda Yilin Wen, Martin Grayson, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao,
+Abdelhak Lemkhenter, Chentian Jiang, Gavin Costello, Gunshi Gupta, Marko Tot, Shu Ishida, Tarun Gupta, Udit Arora,
+Ryen W. White, Sam Devlin, Cecily Morrison, Katja Hofmann
+</div><br>
+<div align='center'>
+Dynamic Generated Gameplay Sequence using WHAM. Showcasing diverse characters and actions across intricate maps.
+  <div style="display: flex; flex-wrap: wrap;">
+    <img style="width: calc(33.33%); margin-bottom: -35px;" src="assets/Readme/wham_gen_1.gif">
+    <img style="width: calc(33.33%); margin-bottom: -35px;" src="assets/Readme/wham_gen_2.gif">
+    <img style="width: calc(33.33%); margin-bottom: -35px;" src="assets/Readme/wham_gen_3.gif">
+    <img style="width: calc(33.33%); margin-bottom: -35px;" src="assets/Readme/wham_gen_4.gif">
+    <img style="width: calc(33.33%); margin-bottom: -35px;" src="assets/Readme/wham_gen_5.gif">
+    <img style="width: calc(33.33%); margin-bottom: -35px;" src="assets/Readme/wham_gen_6.gif">
+    <img style="width: calc(33.33%);" src="assets/Readme/wham_gen_7.gif">
+    <img style="width: calc(33.33%);" src="assets/Readme/wham_gen_8.gif">
+    <img style="width: calc(33.33%);" src="assets/Readme/wham_gen_9.gif">
+  </div>
+</div><br>
+<div align='center'>
+WHAM is capable of generating consistent, diverse, and persistent outputs, enabling various use cases for creative iteration.
+<img style="width: 100%;" src="assets/Readme/model_capabilities.gif">
+</div>
+
+Muse is powered by a World and Human Action Model (WHAM), which is a generative model of gameplay (visuals and/or controller actions) trained on gameplay data of Ninja Theory’s Xbox game Bleeding Edge. Model development was informed by requirements of game creatives that we identified through a user study. Our goal is to explore the capabilities that generative AI models need to support human creative exploration. WHAM is developed by the [Game Intelligence group](https://www.microsoft.com/en-us/research/group/game-intelligence/) at [Microsoft Research](https://www.microsoft.com/en-us/research/), in collaboration with [TaiX](https://www.microsoft.com/en-us/research/project/taix/) and [Ninja Theory](https://ninjatheory.com/).
+
+# Model Card
+
+WHAM is an autoregressive model that has been trained to predict (tokenized) game visuals and controller actions given a prompt. Prompts here can be either visual (one or more initial game visuals) and / or controller actions. This allows the user to run the model in (a) world modelling mode (generate visuals given controller actions), (b) behavior policy (generate controller actions given past visuals), or (c) generate both visuals and behavior.
+
+WHAM consists of two components, an encoder-decoder [VQ-GAN](https://compvis.github.io/taming-transformers/) trained to encode game visuals to a discrete representation, and a transformer backbone trained to perform next-token prediction. We train both components from scratch. The resulting model can generate consistent game sequences, and shows evidence of capturing the 3D structure of the game environment, the effects of controller actions, and the temporal structure of the game (up to the model’s context length).
+
+WHAM was trained on  human gameplay data to predict game visuals and players’ controller actions. We worked with the game studio Ninja Theory and their game [Bleeding Edge](https://www.bleedingedge.com/) – a 3D, 4v4 multiplayer video game. From the resulting data we extracted one year’s worth of anonymized gameplay from 27,990 players, capturing a wide range of behaviors and interactions. A sample of this data is provided [here](https://huggingface.co/datasets/microsoft/bleeding-edge-gameplay-sample)
+
+## Model Details
+
+### Trained Models
+
+In this release we provide the weights of two WHAM instances: 200M WHAM and 1.6B WHAM. Both have been trained from scratch on the same data set. 1.6B WHAM is evaluated in [our paper](https://www.nature.com/articles/s41586-025-08600-3). We additionally provide 200M WHAM as a more lightweight option for faster explorations.
+-   [WHAM with 200M parameters](models/WHAM_200M.ckpt), 1M training steps, model size: 3.7GB
+-   [WHAM with 1.6B parameters](models/WHAM_1.6B_v1.ckpt), 200k training steps, model size: 18.9GB
+
+## Usage
+
+### System Requirements
+
+The steps below have been tested on the following setup:
+- Linux workstation with Ubuntu 20.04.4 LTS
+- Windows 11 workstation running WSL2 with Ubuntu 20.04.6 LTS
+
+The current setup assumes that a CUDA-supported GPU is available for model inference. This has been tested on systems with `NVIDIA RTX A6000` and `NVIDIA GeForce GTX 1080` respectively. In addition, approximately `15GB` of free hard disk space is required for dowmloading the models.
+
+The steps under Installation assume a python 3.9 installation that can be 
+called using the command `python3.9` and the venv package for creating virtual environments. If either of these is not present, you can install this version of python under Ubuntu using:
+
+```bash
+sudo apt install python3.9
+sudo apt install python3.9-venv
+```
+
+If you are using the WHAM Demonstrator, please ensure that you have the required [.NET Core Runtime](https://dotnet.microsoft.com/en-us/download/dotnet/7.0). If this is not yet installed, an error message will pop up from which you can follow a link to download and install this package.
+
+### Installation
+
+1. Clone this repository. We recommend starting without the large model files, using `GIT_LFS_SKIP_SMUDGE=1 git clone [email protected]:MSRTestOrg/WHAM`
+2. `cd WHAM`
+3. `./setup_local.sh`
+
+This will set up a `python3.9` virtual environment and install the required packages (this includes packages required for the model server). The typical install time should be approximately 5 minutes.
+
+3. Run `source venv/bin/activate` whenever you want to run model inference or the model server 
+
+4. Download model from this HuggingFace repository (See note below): 
+   1. Go to Files and versions and navigate to the `models` folder.
+   2. Download the model checkpoint. The instructions below assume that the model checkpoints have been downloaded to your local `models` folder.
+
+**Note:** On Linux systems, you can use `git clone` to clone the enire repository, including large files. Due to a limitation of `git lfs` on Windows, only files up to `4GB` are supported and we recommend downloading the model files manually from the `models` folder.
+
+
+### Local Model Inference
+
+This section assumes that you have followed the installation steps above.
+
+(Optional) Download [sample data](https://huggingface.co/datasets/microsoft/bleeding-edge-gameplay-sample). For the local inference examples below, we recommend that you start with the `tiny-sample` set of only 4 trajectories for your initial exploration.
+
+You can now run model inference to generate gameplay sequences as follows:
+
+```python
+python run_dreaming.py --model_path <path_to_checkpoint.ckpt> --data_path <path_to_sample_data_folder>
+```
+
+To run the 200M parameter (small) model (if you copied the tiny-sample folder to the root directory):
+
+```bash
+python run_dreaming.py --model_path models/WHAM_200M.ckpt --data_path tiny-sample
+```
+
+This uses the data in `data_path` as initial prompt sequences. The script will create a `dreaming_output` directory which will create two files per ground truth data file:
+- An `.npz` file that contains a number of entries, most important of which are:
+  - `encoded_decoded_ground_truth_images`: the original context images, encoded and decoded with the VQGAN.
+  - `dreamt_images`: the sequence of all dreamt images.
+- An `.mp4` file of the context data + dreamt images for easier viewing.
+
+This requires approximately 4.5GB of VRAM on a single A6000, but only uses batch size of one. To speed up the process, increase batch size with `--batch_size` argument. With a single A6000 and `--batch_size 12` this uses approximately 30GB of VRAM. Generating gameplay sequences from the full 512 video dataset takes around 24 hours.
+
+Please note that the first output from the script is generated when the first gameplay sequence has been generated. This may take several minutes when using an `A6000` GPU, or longer for older generation GPUs.
+
+See python `run_dreaming.py --help` for different settings.
+
+### WHAM Demonstrator
+
+#### Setting up the Model Server
+
+We have tested the server code as provided on a single Linux machine with four `A6000 GPUs` (large model) as well as on a Windows machine running Ubuntu under `WSL2`, equipped with a single `GeForce GTX 1080` (small model). Model inferences can be run on lower spec NVIDIA GPUs by reducing the batchsize.
+
+The steps below assume that the installation steps above have been followed and that the model files have been downloaded to your local machine.
+
+In your terminal, activate the newly installed virtual environment (if it isn't already):
+
+```bash
+source venv/bin/activate
+```
+
+Start the server, pointing it to the model:
+
+```bash
+python run_server.py --model <path_to_model_file>
+```
+
+To run the 200M parameter (small) model:
+
+```bash
+python run_server.py --model models/WHAM_200M.ckpt
+```
+
+To run the 1.6B parameter (large) model:
+
+```bash
+python run_server.py --model models/WHAM_1.6B_v1.ckpt
+```
+
+
+The server will start and by default listen on localhost port 5000 (this can be configured with `--port <port>`).
+
+**Note:** If you run out of VRAM when running the server, you can reduce the `MAX_BATCH_SIZE` variable in `run_server.py`.
+
+
+#### Install the WHAM Demonstrator App (Windows only)
+
+After cloning or downloading this repository, navigate to the folder `WHAM/wham_demonstrator`, and start the Windows application `WHAMDemonstrator.exe` within that folder.
+
+Follow the instructions in the provided README.md within WHAM Demonstrator to connect to your model server and get an overview of supported functionality.
+
+
+## Intended Uses
+
+This model and accompanying code are intended for academic research purposes only. WHAM has been trained on gameplay data from a single game, Bleeding Edge, and is intended to be used to generate plausible gameplay sequences resembling this game.
+
+The model is not intended to be used to generate imagery outside of the game Bleeding Edge. Generated images include watermark and provenance metadata. Do not remove the watermark or provenance metadata..
+
+WHAM can be used in multiple scenarios. The following list illustrates the types of tasks that WHAM can be used for:
+- World Model: Visuals are predicted, given a real starting state and action sequence.
+- Behaviour Policy: Given visuals, the model predicts the next controller action.
+- Full Generation: The model generates both the visuals and the controller actions a human player might take in the game.
+
+## Training
+
+### Model
+
+- Architecture: A decoder-only transformer that predicts the next token corresponding to an interleaved sequence of observations and actions. The image tokenizer is a VQ-GAN.
+- Context length: 10 (observation, action) pairs / 5560 tokens
+- Dataset size: The model was trained on data from approximately `500,000` Bleeding Edge games from all seven game maps (over 1 billion observation, action pairs 10Hz, equivalent to over 7 years of continuous human gameplay). A data sample is provided in [bleeding-edge-gameplay-sample](https://huggingface.co/datasets/microsoft/bleeding-edge-gameplay-sample). This is the test data used for our evaluation results, and has the same format as the training data.
+- GPUs: 98xH100 GPUs
+- Training time: 5 days
+
+### Software
+
+- [PyTorch Lightning](https://github.com/pytorch/pytorch)
+- [Flash-Attention](https://github.com/HazyResearch/flash-attention)
+- [ffmpeg](https://github.com/FFmpeg/FFmpeg)
+- [exiftool](https://github.com/exiftool/exiftool)
+
+## Bias, Risks and Limitations
+
+- The training data represents gameplay recordings from a variety of skilled and unskilled gameplayers, representing diverse demographic characteristics. Not all possible player characteristics are represented and model performance may therefore vary.
+- The model, as it is, can only be used to generate visuals and controller inputs. Users should not manipulate images and attempt to generate offensive scenes.   
+
+### Technical limitations, operational factors, and ranges
+
+Model: 
+-	Trained on a single game, very specialized, not intended for image prompts that are out of context or from other domains
+-	Limited context length (10s)
+-	Limited image resolution (300px x 180px), the model can only generate images at this fixed resolution.
+-	Generated images and controls can incorrect or unrecognizable.
+-	Inference time is currently too slow for real-time use.
+
+WHAM Demonstrator:
+-	Developed as a way to explore potential interactions. This is not intended as a fully-fledged user experience or demo.
+
+Models trained using game data may potentially behave in ways that are unfair, unreliable, or offensive, in turn causing harms. We emphasize that these types of harms are not mutually exclusive. A single model can exhibit more than one type of harm, potentially relating to multiple different groups of people. For example, the output of the model can be nonsensical or might look reasonable but is inaccurate with respect to external validation sources.
+Although users can input any image as a starting point, the model is only trained to generate images and controller actions based on the structure of the Bleeding Edge game environment that it has learned from the training data.    Out of domain inputs lead to unpredictable results. For example, this could include a sequence of images that dissolve into unrecognizable blobs   .  
+Model generations when “out of scope” image elements are introduced will either:
+- Dissolve into unrecognizable blobs of color.
+- Morphed into game-relevant items such as game characters. 
+
+## Evaluating WHAM
+WHAM is evaluated based on its consistency, diversity, and persistency. Consistency is measured using Fréchet Video Distance (FVD), while diversity is assessed by comparing the marginal distribution of real human actions to those generated by the model using the Wasserstein distance. Persistency is tested using two scenarios: by adding a static power-up object to a game visual and by adding another player character to a game visual used for prompting the model. For detailed evaluation results, see the paper that [introduces the model](https://www.nature.com/articles/s41586-025-08600-3). 
+
+### Responsible AI testing
+WHAM has been tested with out of context prompt images to evaluate the risk of outputting harmful or nonsensical images. The generated image sequences did not retain the initial image, but rather dissolved into either unrecognizable blobs or to scenes resembling the training environment. 
+
+
+## License
+
+The model is licensed under the [Microsoft Research License](LICENSE.md)
+
+this work has been funded by Microsoft Research
+
+## Privacy & Ethics Statement
+
+[Microsoft Privacy Statement](https://go.microsoft.com/fwlink/?LinkId=521839)
+
+## Trademark Notice
+
+**Trademarks** This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft’s Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.
+
+## Contact Information  
+For questions please email to [email protected]

WHAM_Demonstrator.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d19ef23d22081044202e464e48dd4f5f6f232215a4cc797ab1a75dd3eb0e0d9
+size 3565673

wham_demonstrator/README.md

@@ -3,22 +3,31 @@
 
 # WHAM Demonstrator Instructions
 
-## Entering a Server IP Address
-The model and API should be hosted on a server and port that is accessible to the app, and the machine running the app. When opening the app, enter the IP address with port into the text box at the top left of the window, in the form `http://127.0.0.1:5000`. You might need to click on Options for the Settings to be visible
+## Selecting the type of endpoint
+Select if you will be running the server from AI Foundry (Azure AI Endpoint) or with run_server.py (Local Server).
+You might need to click on Options for the Settings to be visible.
 ![Figure 1](../assets/Demonstrator/Fig_01.png)
 
+## Entering Server IP Address and API Key (Azure AI Foundry)
+If you have deployed your model in Azure AI Foundry. Enter the endpoint address as well as the API Key.
+![Figure 2](../assets/Demonstrator/Fig_02.png)
+
+## Entering a Server IP Address (Local Deployment)
+If you have deployed your own model using run_server.py. The model and API should be hosted on a server and port that is accessible to the app. Enter the IP address with port into the text box at the top left of the window, in the form `http://127.0.0.1:5000`.
+![Figure 3](../assets/Demonstrator/Fig_03.png)
+
 ## Opening Starting Frames
 Creating sequences requires at least one start frame to generate gameplay sequence from. The WHAM Demonstrator contains some example starting frames. You can open the example frames using the “Open Example” button.
-![Figure 2](../assets/Demonstrator/Fig_02.png)
+![Figure 4](../assets/Demonstrator/Fig_04.png)
 
 You can also start with either one or more images that you choose, or you can open a previously saved generated gameplay sequence timeline.
-![Figure 3](../assets/Demonstrator/Fig_03.png)
+![Figure 5](../assets/Demonstrator/Fig_05.png)
 To open one or more images to seed a generated gameplay sequence, select `File -> Open Image(s)…`, and select one or more images. The images will appear as the first images on the timeline at the bottom of the window. To open a previously saved timeline, select `File -> Open Timeline JSON…` and select the JSON file saved with the timeline. The seed images should now appear in the timeline at the bottom of the view.
-![Figure 4](../assets/Demonstrator/Fig_04.png)
+![Figure 6](../assets/Demonstrator/Fig_06.png)
 
 ## Prediction Parameters
 When creating generated gameplay sequence, there are a number of parameters you can set:
-![Figure 5](../assets/Demonstrator/Fig_05.png)
+![Figure 7](../assets/Demonstrator/Fig_07.png)
 - **In**: The number of input frames into the next generated gameplay sequence. The maximum is 10.
 - **Out**: The number of frames you would like to be returned per generated sequence.
 - **Reps**: The number of branches for each generated gameplay sequence initiated.
@@ -26,7 +35,7 @@ When creating generated gameplay sequence, there are a number of parameters you
 
 ## The Timeline User Interface
 The timeline can get quite big. Here are a few controls to help manage the user interface when the timeline is large:
-![Figure 6](../assets/Demonstrator/Fig_06.png)
+![Figure 8](../assets/Demonstrator/Fig_08.png)
 - **Resize bar**: Click and drag this to change the vertical size of the timeline.
 - **Frame selection**: If you want to focus in on a particular pathway, you can select the end frame, and all frames leading up to it will also be selected.
 - **Selected frames only**: Toggle this to flatten the timeline to only the selected frames.
@@ -34,16 +43,16 @@ The timeline can get quite big. Here are a few controls to help manage the user
 
 There is also a “snapshot” button that allows you to take a single picture of your entire timeline, even if it has scrolled out of view. 
 When you select a frame, you will see an outline around the frame in the timeline, and the frame will appear in the main area above.
-![Figure 7](../assets/Demonstrator/Fig_07.png)
+![Figure 9](../assets/Demonstrator/Fig_09.png)
 
 **Note**: When making a frame the selected frame (and have the image appear above), you need to click on the image part of the frame thumbnail on the timeline, not the selection box.
 
 ## Generating gameplay sequence
 To create a generated gameplay sequence (or set of generated gamepla sequences), you must first click on the frame you would like to generate from. The WHAM Demonstrator allows creating new gameplay branches from any frame. So please ensure the frame you wish to generate from has been selected. If the parameters are correct and the server address has been entered, you can click “Predict” in the bar above the timeline. You should see the selected number of generated gameplay sequence branches, and new frames start to appear.
-![Figure 8](../assets/Demonstrator/Fig_08.png)
+![Figure 10](../assets/Demonstrator/Fig_10.png)
 
 The frames may take a number of seconds to appear, depending on the hardware used to host the model. You can cancel one of the branches at any moment by clicking on the ‘X’ in the last frame square. When the frame generation is complete, you can select another frame anywhere in the timeline and select “Predict” again.
-![Figure 9](../assets/Demonstrator/Fig_09.png)
+![Figure 11](../assets/Demonstrator/Fig_11.png)
 
 ## Manipulating Images
 New image layers can be added to any gameplay frame to introduce new content or characters to include in future generated frames. You could add a power-up or an NPC to a frame and have those included as the gameplay sequence evolves. 
@@ -51,16 +60,16 @@ New image layers can be added to any gameplay frame to introduce new content or
 **Note**: This feature is not fully supported yet in the models shared with the WHAM Demonstrator, so performance can be unpredictable. To help added elements really “stick” in the generated sequence, we recommend creating a sequence of 5 frames with your added element in before continuing to generate sequences.
 
 To add a new element, select the frame you wish to manipulate so it appears in the main frame area and click the `+` button in the layers panel. 
-![Figure 10](../assets/Demonstrator/Fig_10.png)
+![Figure 12](../assets/Demonstrator/Fig_12.png)
 
 You will then be prompted to select an image. We recommend a transparent PNG, like the example supplied within the Examples folder called "Layer_Image". The selected image will then appear both as a layer in the layers panel, and on the currently selected frame.
-![Figure 11](../assets/Demonstrator/Fig_11.png)
+![Figure 13](../assets/Demonstrator/Fig_13.png)
 
 To move and resize the added element, click on the element, either in the layers panel, or on the image directly, and you can use your mouse wheel to resize, or drag it around to move. Here, the character has been placed on the right of the frame. 
-![Figure 12](../assets/Demonstrator/Fig_12.png)
+![Figure 14](../assets/Demonstrator/Fig_14.png)
 
 To easily create a sequence, click the “copy to next frame” button, and the layers will be added to the next frame for easier sequencing. Here are a further 4 frames showing the character movement as they enter from the right. 
-![Figure 13](../assets/Demonstrator/Fig_13.png)
+![Figure 15](../assets/Demonstrator/Fig_15.png)
 
 Predictions can now happen from the manipulated frame.
 
@@ -68,12 +77,12 @@ Predictions can now happen from the manipulated frame.
 
 ## Controller Input
 The WHAM Demonstrator can also be controlled using an Xbox controller. 
-![Figure 14](../assets/Demonstrator/Fig_14.png)
+![Figure 16](../assets/Demonstrator/Fig_16.png)
 
 With a controller connected and a frame selected, you can hold down the buttons you wish to use on the controller for around a second and the desired input will be passed to the model for the next frame. When you hold buttons, a blue progress bar will appear below the controller, when this disappears, you must release all of the controller buttons. The WHAM Demonstrator does not yet support holding the buttons for multiple frame predictions. Also, in this mode, only one frame will be produced at a time. When you hold the buttons down on the controller, you will also notice that whenever a frame is selected, this controller image will show the action state of the controller too.
 
 ## Saving Generated Gameplay Sequence
 If you have created a gameplay sequence timeline that you want to save, or even continue later, you can save either a flat, selected sequence of frames, or the entire timeline. 
-![Figure 15](../assets/Demonstrator/Fig_15.png)
+![Figure 17](../assets/Demonstrator/Fig_17.png)
 
 Both of these options will ask for a folder and all of the required images and timeline information will be saved to that folder.