remove old template and use https://github.com/ashleve/lightning-hydra-template instead

model/.dockerignore

@@ -1,4 +0,0 @@
-.env
-in/
-out/
-new_proect.py

model/Dockerfile

@@ -1,13 +0,0 @@
-FROM python:3.7
-
-WORKDIR /usr/src/app
-
-RUN apt-get update 
-RUN apt-get install libgl1 -y
-
-COPY requirements.txt ./
-RUN pip install --no-cache-dir -r requirements.txt
-
-COPY . .
-
-CMD sh -c "python train.py --config config.json"

model/LICENSE

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2018 Victor Huang
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

model/README.md

@@ -1,378 +0,0 @@
-# PyTorch Template Project
-PyTorch deep learning project made easy.
-
-<!-- @import "[TOC]" {cmd="toc" depthFrom=1 depthTo=6 orderedList=false} -->
-
-<!-- code_chunk_output -->
-
-* [PyTorch Template Project](#pytorch-template-project)
- * [Requirements](#requirements)
- * [Features](#features)
- * [Folder Structure](#folder-structure)
- * [Usage](#usage)
- * [Config file format](#config-file-format)
- * [Using config files](#using-config-files)
- * [Resuming from checkpoints](#resuming-from-checkpoints)
- * [Using Multiple GPU](#using-multiple-gpu)
- * [Customization](#customization)
- * [Custom CLI options](#custom-cli-options)
- * [Data Loader](#data-loader)
- * [Trainer](#trainer)
- * [Model](#model)
- * [Loss](#loss)
- * [metrics](#metrics)
- * [Additional logging](#additional-logging)
- * [Validation data](#validation-data)
- * [Checkpoints](#checkpoints)
- * [Tensorboard Visualization](#tensorboard-visualization)
- * [Contribution](#contribution)
- * [TODOs](#todos)
- * [License](#license)
- * [Acknowledgements](#acknowledgements)
-
-<!-- /code_chunk_output -->
-
-## Requirements
-* Python >= 3.5 (3.6 recommended)
-* PyTorch >= 0.4 (1.2 recommended)
-* tqdm (Optional for `test.py`)
-* tensorboard >= 1.14 (see [Tensorboard Visualization](#tensorboard-visualization))
-
-## Features
-* Clear folder structure which is suitable for many deep learning projects.
-* `.json` config file support for convenient parameter tuning.
-* Customizable command line options for more convenient parameter tuning.
-* Checkpoint saving and resuming.
-* Abstract base classes for faster development:
- * `BaseTrainer` handles checkpoint saving/resuming, training process logging, and more.
- * `BaseDataLoader` handles batch generation, data shuffling, and validation data splitting.
- * `BaseModel` provides basic model summary.
-
-## Folder Structure
- ```
- pytorch-template/
- │
- ├── train.py - main script to start training
- ├── test.py - evaluation of trained model
- │
- ├── config.json - holds configuration for training
- ├── parse_config.py - class to handle config file and cli options
- │
- ├── new_project.py - initialize new project with template files
- │
- ├── base/ - abstract base classes
- │ ├── base_data_loader.py
- │ ├── base_model.py
- │ └── base_trainer.py
- │
- ├── data_loader/ - anything about data loading goes here
- │ └── data_loaders.py
- │
- ├── data/ - default directory for storing input data
- │
- ├── model/ - models, losses, and metrics
- │ ├── model.py
- │ ├── metric.py
- │ └── loss.py
- │
- ├── saved/
- │ ├── models/ - trained models are saved here
- │ └── log/ - default logdir for tensorboard and logging output
- │
- ├── trainer/ - trainers
- │ └── trainer.py
- │
- ├── logger/ - module for tensorboard visualization and logging
- │ ├── visualization.py
- │ ├── logger.py
- │ └── logger_config.json
- │ 
- └── utils/ - small utility functions
- ├── util.py
- └── ...
- ```
-
-## Usage
-The code in this repo is an MNIST example of the template.
-Try `python train.py -c config.json` to run code.
-
-### Config file format
-Config files are in `.json` format:
-```javascript
-{
- "name": "Mnist_LeNet", // training session name
- "n_gpu": 1, // number of GPUs to use for training.
- 
- "arch": {
- "type": "MnistModel", // name of model architecture to train
- "args": {
-
- } 
- },
- "data_loader": {
- "type": "MnistDataLoader", // selecting data loader
- "args":{
- "data_dir": "data/", // dataset path
- "batch_size": 64, // batch size
- "shuffle": true, // shuffle training data before splitting
- "validation_split": 0.1 // size of validation dataset. float(portion) or int(number of samples)
- "num_workers": 2, // number of cpu processes to be used for data loading
- }
- },
- "optimizer": {
- "type": "Adam",
- "args":{
- "lr": 0.001, // learning rate
- "weight_decay": 0, // (optional) weight decay
- "amsgrad": true
- }
- },
- "loss": "nll_loss", // loss
- "metrics": [
- "accuracy", "top_k_acc" // list of metrics to evaluate
- ], 
- "lr_scheduler": {
- "type": "StepLR", // learning rate scheduler
- "args":{
- "step_size": 50, 
- "gamma": 0.1
- }
- },
- "trainer": {
- "epochs": 100, // number of training epochs
- "save_dir": "saved/", // checkpoints are saved in save_dir/models/name
- "save_freq": 1, // save checkpoints every save_freq epochs
- "verbosity": 2, // 0: quiet, 1: per epoch, 2: full
- 
- "monitor": "min val_loss" // mode and metric for model performance monitoring. set 'off' to disable.
- "early_stop": 10 // number of epochs to wait before early stop. set 0 to disable.
- 
- "tensorboard": true, // enable tensorboard visualization
- }
-}
-```
-
-Add addional configurations if you need.
-
-### Using config files
-Modify the configurations in `.json` config files, then run:
-
- ```
- python train.py --config config.json
- ```
-
-### Resuming from checkpoints
-You can resume from a previously saved checkpoint by:
-
- ```
- python train.py --resume path/to/checkpoint
- ```
-
-### Using Multiple GPU
-You can enable multi-GPU training by setting `n_gpu` argument of the config file to larger number.
-If configured to use smaller number of gpu than available, first n devices will be used by default.
-Specify indices of available GPUs by cuda environmental variable.
- ```
- python train.py --device 2,3 -c config.json
- ```
- This is equivalent to
- ```
- CUDA_VISIBLE_DEVICES=2,3 python train.py -c config.py
- ```
-
-## Customization
-
-### Project initialization
-Use the `new_project.py` script to make your new project directory with template files.
-`python new_project.py ../NewProject` then a new project folder named 'NewProject' will be made.
-This script will filter out unneccessary files like cache, git files or readme file. 
-
-### Custom CLI options
-
-Changing values of config file is a clean, safe and easy way of tuning hyperparameters. However, sometimes
-it is better to have command line options if some values need to be changed too often or quickly.
-
-This template uses the configurations stored in the json file by default, but by registering custom options as follows
-you can change some of them using CLI flags.
-
- ```python
- # simple class-like object having 3 attributes, `flags`, `type`, `target`.
- CustomArgs = collections.namedtuple('CustomArgs', 'flags type target')
- options = [
- CustomArgs(['--lr', '--learning_rate'], type=float, target=('optimizer', 'args', 'lr')),
- CustomArgs(['--bs', '--batch_size'], type=int, target=('data_loader', 'args', 'batch_size'))
- # options added here can be modified by command line flags.
- ]
- ```
-`target` argument should be sequence of keys, which are used to access that option in the config dict. In this example, `target` 
-for the learning rate option is `('optimizer', 'args', 'lr')` because `config['optimizer']['args']['lr']` points to the learning rate.
-`python train.py -c config.json --bs 256` runs training with options given in `config.json` except for the `batch size`
-which is increased to 256 by command line options.
-
-
-### Data Loader
-* **Writing your own data loader**
-
-1. **Inherit ```BaseDataLoader```**
-
- `BaseDataLoader` is a subclass of `torch.utils.data.DataLoader`, you can use either of them.
-
- `BaseDataLoader` handles:
- * Generating next batch
- * Data shuffling
- * Generating validation data loader by calling
- `BaseDataLoader.split_validation()`
-
-* **DataLoader Usage**
-
- `BaseDataLoader` is an iterator, to iterate through batches:
- ```python
- for batch_idx, (x_batch, y_batch) in data_loader:
- pass
- ```
-* **Example**
-
- Please refer to `data_loader/data_loaders.py` for an MNIST data loading example.
-
-### Trainer
-* **Writing your own trainer**
-
-1. **Inherit ```BaseTrainer```**
-
- `BaseTrainer` handles:
- * Training process logging
- * Checkpoint saving
- * Checkpoint resuming
- * Reconfigurable performance monitoring for saving current best model, and early stop training.
- * If config `monitor` is set to `max val_accuracy`, which means then the trainer will save a checkpoint `model_best.pth` when `validation accuracy` of epoch replaces current `maximum`.
- * If config `early_stop` is set, training will be automatically terminated when model performance does not improve for given number of epochs. This feature can be turned off by passing 0 to the `early_stop` option, or just deleting the line of config.
-
-2. **Implementing abstract methods**
-
- You need to implement `_train_epoch()` for your training process, if you need validation then you can implement `_valid_epoch()` as in `trainer/trainer.py`
-
-* **Example**
-
- Please refer to `trainer/trainer.py` for MNIST training.
-
-* **Iteration-based training**
-
- `Trainer.__init__` takes an optional argument, `len_epoch` which controls number of batches(steps) in each epoch.
-
-### Model
-* **Writing your own model**
-
-1. **Inherit `BaseModel`**
-
- `BaseModel` handles:
- * Inherited from `torch.nn.Module`
- * `__str__`: Modify native `print` function to prints the number of trainable parameters.
-
-2. **Implementing abstract methods**
-
- Implement the foward pass method `forward()`
-
-* **Example**
-
- Please refer to `model/model.py` for a LeNet example.
-
-### Loss
-Custom loss functions can be implemented in 'model/loss.py'. Use them by changing the name given in "loss" in config file, to corresponding name.
-
-### Metrics
-Metric functions are located in 'model/metric.py'.
-
-You can monitor multiple metrics by providing a list in the configuration file, e.g.:
- ```json
- "metrics": ["accuracy", "top_k_acc"],
- ```
-
-### Additional logging
-If you have additional information to be logged, in `_train_epoch()` of your trainer class, merge them with `log` as shown below before returning:
-
- ```python
- additional_log = {"gradient_norm": g, "sensitivity": s}
- log.update(additional_log)
- return log
- ```
-
-### Testing
-You can test trained model by running `test.py` passing path to the trained checkpoint by `--resume` argument.
-
-### Validation data
-To split validation data from a data loader, call `BaseDataLoader.split_validation()`, then it will return a data loader for validation of size specified in your config file.
-The `validation_split` can be a ratio of validation set per total data(0.0 <= float < 1.0), or the number of samples (0 <= int < `n_total_samples`).
-
-**Note**: the `split_validation()` method will modify the original data loader
-**Note**: `split_validation()` will return `None` if `"validation_split"` is set to `0`
-
-### Checkpoints
-You can specify the name of the training session in config files:
- ```json
- "name": "MNIST_LeNet",
- ```
-
-The checkpoints will be saved in `save_dir/name/timestamp/checkpoint_epoch_n`, with timestamp in mmdd_HHMMSS format.
-
-A copy of config file will be saved in the same folder.
-
-**Note**: checkpoints contain:
- ```python
- {
- 'arch': arch,
- 'epoch': epoch,
- 'state_dict': self.model.state_dict(),
- 'optimizer': self.optimizer.state_dict(),
- 'monitor_best': self.mnt_best,
- 'config': self.config
- }
- ```
-
-### Tensorboard Visualization
-This template supports Tensorboard visualization by using either `torch.utils.tensorboard` or [TensorboardX](https://github.com/lanpa/tensorboardX).
-
-1. **Install**
-
- If you are using pytorch 1.1 or higher, install tensorboard by 'pip install tensorboard>=1.14.0'.
-
- Otherwise, you should install tensorboardx. Follow installation guide in [TensorboardX](https://github.com/lanpa/tensorboardX).
-
-2. **Run training** 
-
- Make sure that `tensorboard` option in the config file is turned on.
-
- ```
- "tensorboard" : true
- ```
-
-3. **Open Tensorboard server** 
-
- Type `tensorboard --logdir saved/log/` at the project root, then server will open at `http://localhost:6006`
-
-By default, values of loss and metrics specified in config file, input images, and histogram of model parameters will be logged.
-If you need more visualizations, use `add_scalar('tag', data)`, `add_image('tag', image)`, etc in the `trainer._train_epoch` method.
-`add_something()` methods in this template are basically wrappers for those of `tensorboardX.SummaryWriter` and `torch.utils.tensorboard.SummaryWriter` modules. 
-
-**Note**: You don't have to specify current steps, since `WriterTensorboard` class defined at `logger/visualization.py` will track current steps.
-
-## Contribution
-Feel free to contribute any kind of function or enhancement, here the coding style follows PEP8
-
-Code should pass the [Flake8](http://flake8.pycqa.org/en/latest/) check before committing.
-
-## TODOs
-
-- [ ] Multiple optimizers
-- [ ] Support more tensorboard functions
-- [x] Using fixed random seed
-- [x] Support pytorch native tensorboard
-- [x] `tensorboardX` logger support
-- [x] Configurable logging layout, checkpoint naming
-- [x] Iteration-based training (instead of epoch-based)
-- [x] Adding command line option for fine-tuning
-
-## License
-This project is licensed under the MIT License. See LICENSE for more details
-
-## Acknowledgements
-This project is inspired by the project [Tensorflow-Project-Template](https://github.com/MrGemy95/Tensorflow-Project-Template) by [Mahmoud Gemy](https://github.com/MrGemy95)

model/base/__init__.py

@@ -1,3 +0,0 @@
-from .base_data_loader import *
-from .base_model import *
-from .base_trainer import *

model/base/base_data_loader.py

@@ -1,61 +0,0 @@
-import numpy as np
-from torch.utils.data import DataLoader
-from torch.utils.data.dataloader import default_collate
-from torch.utils.data.sampler import SubsetRandomSampler
-
-
-class BaseDataLoader(DataLoader):
- """
- Base class for all data loaders
- """
- def __init__(self, dataset, batch_size, shuffle, validation_split, num_workers, collate_fn=default_collate):
- self.validation_split = validation_split
- self.shuffle = shuffle
-
- self.batch_idx = 0
- self.n_samples = len(dataset)
-
- self.sampler, self.valid_sampler = self._split_sampler(self.validation_split)
-
- self.init_kwargs = {
- 'dataset': dataset,
- 'batch_size': batch_size,
- 'shuffle': self.shuffle,
- 'collate_fn': collate_fn,
- 'num_workers': num_workers
- }
- super().__init__(sampler=self.sampler, **self.init_kwargs)
-
- def _split_sampler(self, split):
- if split == 0.0:
- return None, None
-
- idx_full = np.arange(self.n_samples)
-
- np.random.seed(0)
- np.random.shuffle(idx_full)
-
- if isinstance(split, int):
- assert split > 0
- assert split < self.n_samples, "validation set size is configured to be larger than entire dataset."
- len_valid = split
- else:
- len_valid = int(self.n_samples * split)
-
- valid_idx = idx_full[0:len_valid]
- train_idx = np.delete(idx_full, np.arange(0, len_valid))
-
- train_sampler = SubsetRandomSampler(train_idx)
- valid_sampler = SubsetRandomSampler(valid_idx)
-
- # turn off shuffle option which is mutually exclusive with sampler
- self.shuffle = False
- self.n_samples = len(train_idx)
-
- return train_sampler, valid_sampler
-
- def split_validation(self):
- if self.valid_sampler is None:
- return None
- else:
- return DataLoader(sampler=self.valid_sampler, **self.init_kwargs)

model/base/base_model.py

@@ -1,25 +0,0 @@
-import torch.nn as nn
-import numpy as np
-from abc import abstractmethod
-
-
-class BaseModel(nn.Module):
- """
- Base class for all models
- """
- @abstractmethod
- def forward(self, *inputs):
- """
- Forward pass logic
-
- :return: Model output
- """
- raise NotImplementedError
-
- def __str__(self):
- """
- Model prints with number of trainable parameters
- """
- model_parameters = filter(lambda p: p.requires_grad, self.parameters())
- params = sum([np.prod(p.size()) for p in model_parameters])
- return super().__str__() + '\nTrainable parameters: {}'.format(params)

model/base/base_trainer.py

@@ -1,151 +0,0 @@
-import torch
-from abc import abstractmethod
-from numpy import inf
-from logger import TensorboardWriter
-
-
-class BaseTrainer:
- """
- Base class for all trainers
- """
- def __init__(self, model, criterion, metric_ftns, optimizer, config):
- self.config = config
- self.logger = config.get_logger('trainer', config['trainer']['verbosity'])
-
- self.model = model
- self.criterion = criterion
- self.metric_ftns = metric_ftns
- self.optimizer = optimizer
-
- cfg_trainer = config['trainer']
- self.epochs = cfg_trainer['epochs']
- self.save_period = cfg_trainer['save_period']
- self.monitor = cfg_trainer.get('monitor', 'off')
-
- # configuration to monitor model performance and save best
- if self.monitor == 'off':
- self.mnt_mode = 'off'
- self.mnt_best = 0
- else:
- self.mnt_mode, self.mnt_metric = self.monitor.split()
- assert self.mnt_mode in ['min', 'max']
-
- self.mnt_best = inf if self.mnt_mode == 'min' else -inf
- self.early_stop = cfg_trainer.get('early_stop', inf)
- if self.early_stop <= 0:
- self.early_stop = inf
-
- self.start_epoch = 1
-
- self.checkpoint_dir = config.save_dir
-
- # setup visualization writer instance 
- self.writer = TensorboardWriter(config.log_dir, self.logger, cfg_trainer['tensorboard'])
-
- if config.resume is not None:
- self._resume_checkpoint(config.resume)
-
- @abstractmethod
- def _train_epoch(self, epoch):
- """
- Training logic for an epoch
-
- :param epoch: Current epoch number
- """
- raise NotImplementedError
-
- def train(self):
- """
- Full training logic
- """
- not_improved_count = 0
- for epoch in range(self.start_epoch, self.epochs + 1):
- result = self._train_epoch(epoch)
-
- # save logged informations into log dict
- log = {'epoch': epoch}
- log.update(result)
-
- # print logged informations to the screen
- for key, value in log.items():
- self.logger.info(' {:15s}: {}'.format(str(key), value))
-
- # evaluate model performance according to configured metric, save best checkpoint as model_best
- best = False
- if self.mnt_mode != 'off':
- try:
- # check whether model performance improved or not, according to specified metric(mnt_metric)
- improved = (self.mnt_mode == 'min' and log[self.mnt_metric] <= self.mnt_best) or \
- (self.mnt_mode == 'max' and log[self.mnt_metric] >= self.mnt_best)
- except KeyError:
- self.logger.warning("Warning: Metric '{}' is not found. "
- "Model performance monitoring is disabled.".format(self.mnt_metric))
- self.mnt_mode = 'off'
- improved = False
-
- if improved:
- self.mnt_best = log[self.mnt_metric]
- not_improved_count = 0
- best = True
- else:
- not_improved_count += 1
-
- if not_improved_count > self.early_stop:
- self.logger.info("Validation performance didn\'t improve for {} epochs. "
- "Training stops.".format(self.early_stop))
- break
-
- if epoch % self.save_period == 0:
- self._save_checkpoint(epoch, save_best=best)
-
- def _save_checkpoint(self, epoch, save_best=False):
- """
- Saving checkpoints
-
- :param epoch: current epoch number
- :param log: logging information of the epoch
- :param save_best: if True, rename the saved checkpoint to 'model_best.pth'
- """
- arch = type(self.model).__name__
- state = {
- 'arch': arch,
- 'epoch': epoch,
- 'state_dict': self.model.state_dict(),
- 'optimizer': self.optimizer.state_dict(),
- 'monitor_best': self.mnt_best,
- 'config': self.config
- }
- filename = str(self.checkpoint_dir / 'checkpoint-epoch{}.pth'.format(epoch))
- torch.save(state, filename)
- self.logger.info("Saving checkpoint: {} ...".format(filename))
- if save_best:
- best_path = str(self.checkpoint_dir / 'model_best.pth')
- torch.save(state, best_path)
- self.logger.info("Saving current best: model_best.pth ...")
-
- def _resume_checkpoint(self, resume_path):
- """
- Resume from saved checkpoints
-
- :param resume_path: Checkpoint path to be resumed
- """
- resume_path = str(resume_path)
- self.logger.info("Loading checkpoint: {} ...".format(resume_path))
- checkpoint = torch.load(resume_path)
- self.start_epoch = checkpoint['epoch'] + 1
- self.mnt_best = checkpoint['monitor_best']
-
- # load architecture params from checkpoint.
- if checkpoint['config']['arch'] != self.config['arch']:
- self.logger.warning("Warning: Architecture configuration given in config file is different from that of "
- "checkpoint. This may yield an exception while state_dict is being loaded.")
- self.model.load_state_dict(checkpoint['state_dict'])
-
- # load optimizer state from checkpoint only when optimizer type is not changed.
- if checkpoint['config']['optimizer']['type'] != self.config['optimizer']['type']:
- self.logger.warning("Warning: Optimizer type given in config file is different from that of checkpoint. "
- "Optimizer parameters not being resumed.")
- else:
- self.optimizer.load_state_dict(checkpoint['optimizer'])
-
- self.logger.info("Checkpoint loaded. Resume training from epoch {}".format(self.start_epoch))

model/config.json

@@ -1,50 +0,0 @@
-{
- "name": "Mnist_LeNet",
- "n_gpu": 1,
-
- "arch": {
- "type": "MnistModel",
- "args": {}
- },
- "data_loader": {
- "type": "MnistDataLoader",
- "args":{
- "data_dir": "data/",
- "batch_size": 128,
- "shuffle": true,
- "validation_split": 0.1,
- "num_workers": 2
- }
- },
- "optimizer": {
- "type": "Adam",
- "args":{
- "lr": 0.001,
- "weight_decay": 0,
- "amsgrad": true
- }
- },
- "loss": "nll_loss",
- "metrics": [
- "accuracy", "top_k_acc"
- ],
- "lr_scheduler": {
- "type": "StepLR",
- "args": {
- "step_size": 50,
- "gamma": 0.1
- }
- },
- "trainer": {
- "epochs": 100,
-
- "save_dir": "saved/",
- "save_period": 1,
- "verbosity": 2,
- 
- "monitor": "min val_loss",
- "early_stop": 10,
-
- "tensorboard": true
- }
-}

model/data_loader/data_loaders.py

@@ -1,16 +0,0 @@
-from torchvision import datasets, transforms
-from base import BaseDataLoader
-
-
-class MnistDataLoader(BaseDataLoader):
- """
- MNIST data loading demo using BaseDataLoader
- """
- def __init__(self, data_dir, batch_size, shuffle=True, validation_split=0.0, num_workers=1, training=True):
- trsfm = transforms.Compose([
- transforms.ToTensor(),
- transforms.Normalize((0.1307,), (0.3081,))
- ])
- self.data_dir = data_dir
- self.dataset = datasets.MNIST(self.data_dir, train=training, download=True, transform=trsfm)
- super().__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)

model/docker-compose.yml

@@ -1,12 +0,0 @@
-version: "3" # optional since v1.27.0
-services:
- model:
- build: .
- ports:
- - 6006:6006
- volumes:
- - ./saved:/usr/src/app/saved:z
- # - ./out/:/usr/src/app/out:z
- # - ./in/:/usr/src/app/in:z
- #env_file:
- # - .env

model/logger/__init__.py

@@ -1,2 +0,0 @@
-from .logger import *
-from .visualization import *

model/logger/logger.py

@@ -1,22 +0,0 @@
-import logging
-import logging.config
-from pathlib import Path
-from utils import read_json
-
-
-def setup_logging(save_dir, log_config='logger/logger_config.json', default_level=logging.INFO):
- """
- Setup logging configuration
- """
- log_config = Path(log_config)
- if log_config.is_file():
- config = read_json(log_config)
- # modify logging paths based on run config
- for _, handler in config['handlers'].items():
- if 'filename' in handler:
- handler['filename'] = str(save_dir / handler['filename'])
-
- logging.config.dictConfig(config)
- else:
- print("Warning: logging configuration file is not found in {}.".format(log_config))
- logging.basicConfig(level=default_level)

model/logger/logger_config.json

@@ -1,32 +0,0 @@
-
-{
- "version": 1, 
- "disable_existing_loggers": false, 
- "formatters": {
- "simple": {"format": "%(message)s"}, 
- "datetime": {"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"}
- }, 
- "handlers": {
- "console": {
- "class": "logging.StreamHandler", 
- "level": "DEBUG", 
- "formatter": "simple", 
- "stream": "ext://sys.stdout"
- }, 
- "info_file_handler": {
- "class": "logging.handlers.RotatingFileHandler", 
- "level": "INFO", 
- "formatter": "datetime", 
- "filename": "info.log", 
- "maxBytes": 10485760, 
- "backupCount": 20, "encoding": "utf8"
- }
- }, 
- "root": {
- "level": "INFO", 
- "handlers": [
- "console", 
- "info_file_handler"
- ]
- }
-}

model/logger/visualization.py

@@ -1,73 +0,0 @@
-import importlib
-from datetime import datetime
-
-
-class TensorboardWriter():
- def __init__(self, log_dir, logger, enabled):
- self.writer = None
- self.selected_module = ""
-
- if enabled:
- log_dir = str(log_dir)
-
- # Retrieve vizualization writer.
- succeeded = False
- for module in ["torch.utils.tensorboard", "tensorboardX"]:
- try:
- self.writer = importlib.import_module(module).SummaryWriter(log_dir)
- succeeded = True
- break
- except ImportError:
- succeeded = False
- self.selected_module = module
-
- if not succeeded:
- message = "Warning: visualization (Tensorboard) is configured to use, but currently not installed on " \
- "this machine. Please install TensorboardX with 'pip install tensorboardx', upgrade PyTorch to " \
- "version >= 1.1 to use 'torch.utils.tensorboard' or turn off the option in the 'config.json' file."
- logger.warning(message)
-
- self.step = 0
- self.mode = ''
-
- self.tb_writer_ftns = {
- 'add_scalar', 'add_scalars', 'add_image', 'add_images', 'add_audio',
- 'add_text', 'add_histogram', 'add_pr_curve', 'add_embedding'
- }
- self.tag_mode_exceptions = {'add_histogram', 'add_embedding'}
- self.timer = datetime.now()
-
- def set_step(self, step, mode='train'):
- self.mode = mode
- self.step = step
- if step == 0:
- self.timer = datetime.now()
- else:
- duration = datetime.now() - self.timer
- self.add_scalar('steps_per_sec', 1 / duration.total_seconds())
- self.timer = datetime.now()
-
- def __getattr__(self, name):
- """
- If visualization is configured to use:
- return add_data() methods of tensorboard with additional information (step, tag) added.
- Otherwise:
- return a blank function handle that does nothing
- """
- if name in self.tb_writer_ftns:
- add_data = getattr(self.writer, name, None)
-
- def wrapper(tag, data, *args, **kwargs):
- if add_data is not None:
- # add mode(train/valid) tag
- if name not in self.tag_mode_exceptions:
- tag = '{}/{}'.format(tag, self.mode)
- add_data(tag, data, self.step, *args, **kwargs)
- return wrapper
- else:
- # default action for returning methods defined in this class, set_step() for instance.
- try:
- attr = object.__getattr__(name)
- except AttributeError:
- raise AttributeError("type object '{}' has no attribute '{}'".format(self.selected_module, name))
- return attr

model/model/loss.py

@@ -1,5 +0,0 @@
-import torch.nn.functional as F
-
-
-def nll_loss(output, target):
- return F.nll_loss(output, target)

model/model/metric.py

@@ -1,20 +0,0 @@
-import torch
-
-
-def accuracy(output, target):
- with torch.no_grad():
- pred = torch.argmax(output, dim=1)
- assert pred.shape[0] == len(target)
- correct = 0
- correct += torch.sum(pred == target).item()
- return correct / len(target)
-
-
-def top_k_acc(output, target, k=3):
- with torch.no_grad():
- pred = torch.topk(output, k, dim=1)[1]
- assert pred.shape[0] == len(target)
- correct = 0
- for i in range(k):
- correct += torch.sum(pred[:, i] == target).item()
- return correct / len(target)

model/model/model.py

@@ -1,22 +0,0 @@
-import torch.nn as nn
-import torch.nn.functional as F
-from base import BaseModel
-
-
-class MnistModel(BaseModel):
- def __init__(self, num_classes=10):
- super().__init__()
- self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
- self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
- self.conv2_drop = nn.Dropout2d()
- self.fc1 = nn.Linear(320, 50)
- self.fc2 = nn.Linear(50, num_classes)
-
- def forward(self, x):
- x = F.relu(F.max_pool2d(self.conv1(x), 2))
- x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
- x = x.view(-1, 320)
- x = F.relu(self.fc1(x))
- x = F.dropout(x, training=self.training)
- x = self.fc2(x)
- return F.log_softmax(x, dim=1)

model/new_project.py

@@ -1,18 +0,0 @@
-import sys
-from pathlib import Path
-from shutil import copytree, ignore_patterns
-
-
-# This script initializes new pytorch project with the template files.
-# Run `python3 new_project.py ../MyNewProject` then new project named 
-# MyNewProject will be made
-current_dir = Path()
-assert (current_dir / 'new_project.py').is_file(), 'Script should be executed in the pytorch-template directory'
-assert len(sys.argv) == 2, 'Specify a name for the new project. Example: python3 new_project.py MyNewProject'
-
-project_name = Path(sys.argv[1])
-target_dir = current_dir / project_name
-
-ignore = [".git", "data", "saved", "new_project.py", "LICENSE", ".flake8", "README.md", "__pycache__"]
-copytree(current_dir, target_dir, ignore=ignore_patterns(*ignore))
-print('New project initialized at', target_dir.absolute().resolve())

model/parse_config.py

@@ -1,157 +0,0 @@
-import os
-import logging
-from pathlib import Path
-from functools import reduce, partial
-from operator import getitem
-from datetime import datetime
-from logger import setup_logging
-from utils import read_json, write_json
-
-
-class ConfigParser:
- def __init__(self, config, resume=None, modification=None, run_id=None):
- """
- class to parse configuration json file. Handles hyperparameters for training, initializations of modules, checkpoint saving
- and logging module.
- :param config: Dict containing configurations, hyperparameters for training. contents of `config.json` file for example.
- :param resume: String, path to the checkpoint being loaded.
- :param modification: Dict keychain:value, specifying position values to be replaced from config dict.
- :param run_id: Unique Identifier for training processes. Used to save checkpoints and training log. Timestamp is being used as default
- """
- # load config file and apply modification
- self._config = _update_config(config, modification)
- self.resume = resume
-
- # set save_dir where trained model and log will be saved.
- save_dir = Path(self.config['trainer']['save_dir'])
-
- exper_name = self.config['name']
- if run_id is None: # use timestamp as default run-id
- run_id = datetime.now().strftime(r'%m%d_%H%M%S')
- self._save_dir = save_dir / 'models' / exper_name / run_id
- self._log_dir = save_dir / 'log' / exper_name / run_id
-
- # make directory for saving checkpoints and log.
- exist_ok = run_id == ''
- self.save_dir.mkdir(parents=True, exist_ok=exist_ok)
- self.log_dir.mkdir(parents=True, exist_ok=exist_ok)
-
- # save updated config file to the checkpoint dir
- write_json(self.config, self.save_dir / 'config.json')
-
- # configure logging module
- setup_logging(self.log_dir)
- self.log_levels = {
- 0: logging.WARNING,
- 1: logging.INFO,
- 2: logging.DEBUG
- }
-
- @classmethod
- def from_args(cls, args, options=''):
- """
- Initialize this class from some cli arguments. Used in train, test.
- """
- for opt in options:
- args.add_argument(*opt.flags, default=None, type=opt.type)
- if not isinstance(args, tuple):
- args = args.parse_args()
-
- if args.device is not None:
- os.environ["CUDA_VISIBLE_DEVICES"] = args.device
- if args.resume is not None:
- resume = Path(args.resume)
- cfg_fname = resume.parent / 'config.json'
- else:
- msg_no_cfg = "Configuration file need to be specified. Add '-c config.json', for example."
- assert args.config is not None, msg_no_cfg
- resume = None
- cfg_fname = Path(args.config)
- 
- config = read_json(cfg_fname)
- if args.config and resume:
- # update new config for fine-tuning
- config.update(read_json(args.config))
-
- # parse custom cli options into dictionary
- modification = {opt.target : getattr(args, _get_opt_name(opt.flags)) for opt in options}
- return cls(config, resume, modification)
-
- def init_obj(self, name, module, *args, **kwargs):
- """
- Finds a function handle with the name given as 'type' in config, and returns the
- instance initialized with corresponding arguments given.
-
- `object = config.init_obj('name', module, a, b=1)`
- is equivalent to
- `object = module.name(a, b=1)`
- """
- module_name = self[name]['type']
- module_args = dict(self[name]['args'])
- assert all([k not in module_args for k in kwargs]), 'Overwriting kwargs given in config file is not allowed'
- module_args.update(kwargs)
- return getattr(module, module_name)(*args, **module_args)
-
- def init_ftn(self, name, module, *args, **kwargs):
- """
- Finds a function handle with the name given as 'type' in config, and returns the
- function with given arguments fixed with functools.partial.
-
- `function = config.init_ftn('name', module, a, b=1)`
- is equivalent to
- `function = lambda *args, **kwargs: module.name(a, *args, b=1, **kwargs)`.
- """
- module_name = self[name]['type']
- module_args = dict(self[name]['args'])
- assert all([k not in module_args for k in kwargs]), 'Overwriting kwargs given in config file is not allowed'
- module_args.update(kwargs)
- return partial(getattr(module, module_name), *args, **module_args)
-
- def __getitem__(self, name):
- """Access items like ordinary dict."""
- return self.config[name]
-
- def get_logger(self, name, verbosity=2):
- msg_verbosity = 'verbosity option {} is invalid. Valid options are {}.'.format(verbosity, self.log_levels.keys())
- assert verbosity in self.log_levels, msg_verbosity
- logger = logging.getLogger(name)
- logger.setLevel(self.log_levels[verbosity])
- return logger
-
- # setting read-only attributes
- @property
- def config(self):
- return self._config
-
- @property
- def save_dir(self):
- return self._save_dir
-
- @property
- def log_dir(self):
- return self._log_dir
-
-# helper functions to update config dict with custom cli options
-def _update_config(config, modification):
- if modification is None:
- return config
-
- for k, v in modification.items():
- if v is not None:
- _set_by_path(config, k, v)
- return config
-
-def _get_opt_name(flags):
- for flg in flags:
- if flg.startswith('--'):
- return flg.replace('--', '')
- return flags[0].replace('--', '')
-
-def _set_by_path(tree, keys, value):
- """Set a value in a nested object in tree by sequence of keys."""
- keys = keys.split(';')
- _get_by_path(tree, keys[:-1])[keys[-1]] = value
-
-def _get_by_path(tree, keys):
- """Access a nested object in tree by sequence of keys."""
- return reduce(getitem, keys, tree)

model/requirements.txt

@@ -1,6 +0,0 @@
-torch>=1.1
-torchvision
-numpy
-tqdm
-tensorboardx
-pandas

model/test.py

@@ -1,81 +0,0 @@
-import argparse
-import torch
-from tqdm import tqdm
-import data_loader.data_loaders as module_data
-import model.loss as module_loss
-import model.metric as module_metric
-import model.model as module_arch
-from parse_config import ConfigParser
-
-
-def main(config):
- logger = config.get_logger('test')
-
- # setup data_loader instances
- data_loader = getattr(module_data, config['data_loader']['type'])(
- config['data_loader']['args']['data_dir'],
- batch_size=512,
- shuffle=False,
- validation_split=0.0,
- training=False,
- num_workers=2
- )
-
- # build model architecture
- model = config.init_obj('arch', module_arch)
- logger.info(model)
-
- # get function handles of loss and metrics
- loss_fn = getattr(module_loss, config['loss'])
- metric_fns = [getattr(module_metric, met) for met in config['metrics']]
-
- logger.info('Loading checkpoint: {} ...'.format(config.resume))
- checkpoint = torch.load(config.resume)
- state_dict = checkpoint['state_dict']
- if config['n_gpu'] > 1:
- model = torch.nn.DataParallel(model)
- model.load_state_dict(state_dict)
-
- # prepare model for testing
- device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
- model = model.to(device)
- model.eval()
-
- total_loss = 0.0
- total_metrics = torch.zeros(len(metric_fns))
-
- with torch.no_grad():
- for i, (data, target) in enumerate(tqdm(data_loader)):
- data, target = data.to(device), target.to(device)
- output = model(data)
-
- #
- # save sample images, or do something with output here
- #
-
- # computing loss, metrics on test set
- loss = loss_fn(output, target)
- batch_size = data.shape[0]
- total_loss += loss.item() * batch_size
- for i, metric in enumerate(metric_fns):
- total_metrics[i] += metric(output, target) * batch_size
-
- n_samples = len(data_loader.sampler)
- log = {'loss': total_loss / n_samples}
- log.update({
- met.__name__: total_metrics[i].item() / n_samples for i, met in enumerate(metric_fns)
- })
- logger.info(log)
-
-
-if __name__ == '__main__':
- args = argparse.ArgumentParser(description='PyTorch Template')
- args.add_argument('-c', '--config', default=None, type=str,
- help='config file path (default: None)')
- args.add_argument('-r', '--resume', default=None, type=str,
- help='path to latest checkpoint (default: None)')
- args.add_argument('-d', '--device', default=None, type=str,
- help='indices of GPUs to enable (default: all)')
-
- config = ConfigParser.from_args(args)
- main(config)

model/train.py

@@ -1,73 +0,0 @@
-import argparse
-import collections
-import torch
-import numpy as np
-import data_loader.data_loaders as module_data
-import model.loss as module_loss
-import model.metric as module_metric
-import model.model as module_arch
-from parse_config import ConfigParser
-from trainer import Trainer
-from utils import prepare_device
-
-
-# fix random seeds for reproducibility
-SEED = 123
-torch.manual_seed(SEED)
-torch.backends.cudnn.deterministic = True
-torch.backends.cudnn.benchmark = False
-np.random.seed(SEED)
-
-def main(config):
- logger = config.get_logger('train')
-
- # setup data_loader instances
- data_loader = config.init_obj('data_loader', module_data)
- valid_data_loader = data_loader.split_validation()
-
- # build model architecture, then print to console
- model = config.init_obj('arch', module_arch)
- logger.info(model)
-
- # prepare for (multi-device) GPU training
- device, device_ids = prepare_device(config['n_gpu'])
- model = model.to(device)
- if len(device_ids) > 1:
- model = torch.nn.DataParallel(model, device_ids=device_ids)
-
- # get function handles of loss and metrics
- criterion = getattr(module_loss, config['loss'])
- metrics = [getattr(module_metric, met) for met in config['metrics']]
-
- # build optimizer, learning rate scheduler. delete every lines containing lr_scheduler for disabling scheduler
- trainable_params = filter(lambda p: p.requires_grad, model.parameters())
- optimizer = config.init_obj('optimizer', torch.optim, trainable_params)
- lr_scheduler = config.init_obj('lr_scheduler', torch.optim.lr_scheduler, optimizer)
-
- trainer = Trainer(model, criterion, metrics, optimizer,
- config=config,
- device=device,
- data_loader=data_loader,
- valid_data_loader=valid_data_loader,
- lr_scheduler=lr_scheduler)
-
- trainer.train()
-
-
-if __name__ == '__main__':
- args = argparse.ArgumentParser(description='PyTorch Template')
- args.add_argument('-c', '--config', default=None, type=str,
- help='config file path (default: None)')
- args.add_argument('-r', '--resume', default=None, type=str,
- help='path to latest checkpoint (default: None)')
- args.add_argument('-d', '--device', default=None, type=str,
- help='indices of GPUs to enable (default: all)')
-
- # custom cli options to modify configuration from default values given in json file.
- CustomArgs = collections.namedtuple('CustomArgs', 'flags type target')
- options = [
- CustomArgs(['--lr', '--learning_rate'], type=float, target='optimizer;args;lr'),
- CustomArgs(['--bs', '--batch_size'], type=int, target='data_loader;args;batch_size')
- ]
- config = ConfigParser.from_args(args, options)
- main(config)

model/trainer/__init__.py

@@ -1 +0,0 @@
-from .trainer import *

model/trainer/trainer.py

@@ -1,110 +0,0 @@
-import numpy as np
-import torch
-from torchvision.utils import make_grid
-from base import BaseTrainer
-from utils import inf_loop, MetricTracker
-
-
-class Trainer(BaseTrainer):
- """
- Trainer class
- """
- def __init__(self, model, criterion, metric_ftns, optimizer, config, device,
- data_loader, valid_data_loader=None, lr_scheduler=None, len_epoch=None):
- super().__init__(model, criterion, metric_ftns, optimizer, config)
- self.config = config
- self.device = device
- self.data_loader = data_loader
- if len_epoch is None:
- # epoch-based training
- self.len_epoch = len(self.data_loader)
- else:
- # iteration-based training
- self.data_loader = inf_loop(data_loader)
- self.len_epoch = len_epoch
- self.valid_data_loader = valid_data_loader
- self.do_validation = self.valid_data_loader is not None
- self.lr_scheduler = lr_scheduler
- self.log_step = int(np.sqrt(data_loader.batch_size))
-
- self.train_metrics = MetricTracker('loss', *[m.__name__ for m in self.metric_ftns], writer=self.writer)
- self.valid_metrics = MetricTracker('loss', *[m.__name__ for m in self.metric_ftns], writer=self.writer)
-
- def _train_epoch(self, epoch):
- """
- Training logic for an epoch
-
- :param epoch: Integer, current training epoch.
- :return: A log that contains average loss and metric in this epoch.
- """
- self.model.train()
- self.train_metrics.reset()
- for batch_idx, (data, target) in enumerate(self.data_loader):
- data, target = data.to(self.device), target.to(self.device)
-
- self.optimizer.zero_grad()
- output = self.model(data)
- loss = self.criterion(output, target)
- loss.backward()
- self.optimizer.step()
-
- self.writer.set_step((epoch - 1) * self.len_epoch + batch_idx)
- self.train_metrics.update('loss', loss.item())
- for met in self.metric_ftns:
- self.train_metrics.update(met.__name__, met(output, target))
-
- if batch_idx % self.log_step == 0:
- self.logger.debug('Train Epoch: {} {} Loss: {:.6f}'.format(
- epoch,
- self._progress(batch_idx),
- loss.item()))
- self.writer.add_image('input', make_grid(data.cpu(), nrow=8, normalize=True))
-
- if batch_idx == self.len_epoch:
- break
- log = self.train_metrics.result()
-
- if self.do_validation:
- val_log = self._valid_epoch(epoch)
- log.update(**{'val_'+k : v for k, v in val_log.items()})
-
- if self.lr_scheduler is not None:
- self.lr_scheduler.step()
- return log
-
- def _valid_epoch(self, epoch):
- """
- Validate after training an epoch
-
- :param epoch: Integer, current training epoch.
- :return: A log that contains information about validation
- """
- self.model.eval()
- self.valid_metrics.reset()
- with torch.no_grad():
- for batch_idx, (data, target) in enumerate(self.valid_data_loader):
- data, target = data.to(self.device), target.to(self.device)
-
- output = self.model(data)
- loss = self.criterion(output, target)
-
- self.writer.set_step((epoch - 1) * len(self.valid_data_loader) + batch_idx, 'valid')
- self.valid_metrics.update('loss', loss.item())
- for met in self.metric_ftns:
- self.valid_metrics.update(met.__name__, met(output, target))
- self.writer.add_image('input', make_grid(data.cpu(), nrow=8, normalize=True))
-
- # add histogram of model parameters to the tensorboard
- for name, p in self.model.named_parameters():
- self.writer.add_histogram(name, p, bins='auto')
- return self.valid_metrics.result()
-
- def _progress(self, batch_idx):
- base = '[{}/{} ({:.0f}%)]'
- if hasattr(self.data_loader, 'n_samples'):
- current = batch_idx * self.data_loader.batch_size
- total = self.data_loader.n_samples
- else:
- current = batch_idx
- total = self.len_epoch
- return base.format(current, total, 100.0 * current / total)

model/utils/__init__.py

@@ -1 +0,0 @@
-from .util import *

model/utils/util.py

@@ -1,67 +0,0 @@
-import json
-import torch
-import pandas as pd
-from pathlib import Path
-from itertools import repeat
-from collections import OrderedDict
-
-
-def ensure_dir(dirname):
- dirname = Path(dirname)
- if not dirname.is_dir():
- dirname.mkdir(parents=True, exist_ok=False)
-
-def read_json(fname):
- fname = Path(fname)
- with fname.open('rt') as handle:
- return json.load(handle, object_hook=OrderedDict)
-
-def write_json(content, fname):
- fname = Path(fname)
- with fname.open('wt') as handle:
- json.dump(content, handle, indent=4, sort_keys=False)
-
-def inf_loop(data_loader):
- ''' wrapper function for endless data loader. '''
- for loader in repeat(data_loader):
- yield from loader
-
-def prepare_device(n_gpu_use):
- """
- setup GPU device if available. get gpu device indices which are used for DataParallel
- """
- n_gpu = torch.cuda.device_count()
- if n_gpu_use > 0 and n_gpu == 0:
- print("Warning: There\'s no GPU available on this machine,"
- "training will be performed on CPU.")
- n_gpu_use = 0
- if n_gpu_use > n_gpu:
- print(f"Warning: The number of GPU\'s configured to use is {n_gpu_use}, but only {n_gpu} are "
- "available on this machine.")
- n_gpu_use = n_gpu
- device = torch.device('cuda:0' if n_gpu_use > 0 else 'cpu')
- list_ids = list(range(n_gpu_use))
- return device, list_ids
-
-class MetricTracker:
- def __init__(self, *keys, writer=None):
- self.writer = writer
- self._data = pd.DataFrame(index=keys, columns=['total', 'counts', 'average'])
- self.reset()
-
- def reset(self):
- for col in self._data.columns:
- self._data[col].values[:] = 0
-
- def update(self, key, value, n=1):
- if self.writer is not None:
- self.writer.add_scalar(key, value)
- self._data.total[key] += value * n
- self._data.counts[key] += n
- self._data.average[key] = self._data.total[key] / self._data.counts[key]
-
- def avg(self, key):
- return self._data.average[key]
-
- def result(self):
- return dict(self._data.average)

models/.env.example

@@ -0,0 +1,7 @@
+# this is example of the file that can be used for storing private and user specific environment variables, like keys or system paths
+# create a file named .env (by default .env will be excluded from version control)
+# the variables declared in .env are loaded in train.py automatically
+# hydra allows you to reference variables in .yaml configs with special syntax: ${oc.env:MY_VAR}
+
+MY_VAR="/home/user/my/system/path"
+MY_KEY="asdgjhawi8y23ihsghsueity23ihwd"

models/.gitignore

@@ -0,0 +1,148 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+# However, in case of collaboration, if having platform-specific dependencies or dependencies
+# having no cross-platform support, pipenv may install dependencies that don't work, or not
+# install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+### VisualStudioCode
+.vscode/*
+!.vscode/settings.json
+!.vscode/tasks.json
+!.vscode/launch.json
+!.vscode/extensions.json
+*.code-workspace
+**/.vscode
+
+# JetBrains
+.idea/
+
+# Lightning-Hydra-Template
+configs/local/default.yaml
+data/
+logs/
+wandb/
+.env
+.autoenv

models/.pre-commit-config.yaml

@@ -0,0 +1,44 @@
+repos:
+ - repo: https://github.com/pre-commit/pre-commit-hooks
+ rev: v4.1.0
+ hooks:
+ # list of supported hooks: https://pre-commit.com/hooks.html
+ - id: trailing-whitespace
+ - id: end-of-file-fixer
+ - id: check-yaml
+ - id: check-added-large-files
+ - id: debug-statements
+ - id: detect-private-key
+
+ # python code formatting
+ - repo: https://github.com/psf/black
+ rev: 22.1.0
+ hooks:
+ - id: black
+ args: [--line-length, "99"]
+
+ # python import sorting
+ - repo: https://github.com/PyCQA/isort
+ rev: 5.10.1
+ hooks:
+ - id: isort
+ args: ["--profile", "black", "--filter-files"]
+
+ # yaml formatting
+ - repo: https://github.com/pre-commit/mirrors-prettier
+ rev: v2.5.1
+ hooks:
+ - id: prettier
+ types: [yaml]
+
+ # python code analysis
+ - repo: https://github.com/PyCQA/flake8
+ rev: 4.0.1
+ hooks:
+ - id: flake8
+
+ # jupyter notebook cell output clearing
+ - repo: https://github.com/kynan/nbstripout
+ rev: 0.5.0
+ hooks:
+ - id: nbstripout

models/README.md

@@ -0,0 +1,1445 @@
+<div align="center">
+
+# Lightning-Hydra-Template
+
+<a href="https://www.python.org/"><img alt="Python" src="https://img.shields.io/badge/-Python 3.7+-blue?style=for-the-badge&logo=python&logoColor=white"></a>
+<a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/-PyTorch 1.8+-ee4c2c?style=for-the-badge&logo=pytorch&logoColor=white"></a>
+<a href="https://pytorchlightning.ai/"><img alt="Lightning" src="https://img.shields.io/badge/-Lightning 1.5+-792ee5?style=for-the-badge&logo=pytorchlightning&logoColor=white"></a>
+<a href="https://hydra.cc/"><img alt="Config: hydra" src="https://img.shields.io/badge/config-hydra 1.1-89b8cd?style=for-the-badge&labelColor=gray"></a>
+<a href="https://black.readthedocs.io/en/stable/"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-black.svg?style=for-the-badge&labelColor=gray"></a>
+
+A clean and scalable template to kickstart your deep learning project 🚀⚡🔥<br>
+Click on [<kbd>Use this template</kbd>](https://github.com/ashleve/lightning-hydra-template/generate) to initialize new repository.
+
+_Suggestions are always welcome!_
+
+</div>
+
+<br><br>
+
+## 📌&nbsp;&nbsp;Introduction
+
+This template tries to be as general as possible. It integrates many different MLOps tools.
+
+> Effective usage of this template requires learning of a couple of technologies: [PyTorch](https://pytorch.org), [PyTorch Lightning](https://www.pytorchlightning.ai) and [Hydra](https://hydra.cc). Knowledge of some experiment logging framework like [Weights&Biases](https://wandb.com), [Neptune](https://neptune.ai) or [MLFlow](https://mlflow.org) is also recommended.
+
+**Why you should use it:** it allows you to rapidly iterate over new models/datasets and scale your projects from small single experiments to hyperparameter searches on computing clusters, without writing any boilerplate code. To my knowledge, it's one of the most convenient all-in-one technology stack for Deep Learning research. Good starting point for reproducing papers, kaggle competitions or small-team research projects. It's also a collection of best practices for efficient workflow and reproducibility.
+
+**Why you shouldn't use it:** this template is not fitted to be a production environment, should be used more as a fast experimentation tool. Apart from that, Lightning and Hydra are still evolving and integrate many libraries, which means sometimes things break - for the list of currently known bugs, visit [this page](https://github.com/ashleve/lightning-hydra-template/labels/bug). Also, even though Lightning is very flexible, it's not well suited for every possible deep learning task. See [#Limitations](#limitations) for more.
+
+### Why PyTorch Lightning?
+
+[PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) is a lightweight PyTorch wrapper for high-performance AI research.
+It makes your code neatly organized and provides lots of useful features, like ability to run model on CPU, GPU, multi-GPU cluster and TPU.
+
+### Why Hydra?
+
+[Hydra](https://github.com/facebookresearch/hydra) is an open-source Python framework that simplifies the development of research and other complex applications. The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line. It allows you to conveniently manage experiments and provides many useful plugins, like [Optuna Sweeper](https://hydra.cc/docs/next/plugins/optuna_sweeper) for hyperparameter search, or [Ray Launcher](https://hydra.cc/docs/next/plugins/ray_launcher) for running jobs on a cluster.
+
+<br>
+
+## Main Ideas Of This Template
+
+- **Predefined Structure**: clean and scalable so that work can easily be extended and replicated | [#Project Structure](#project-structure)
+- **Rapid Experimentation**: thanks to automating pipeline with config files and hydra command line superpowers | [#Your Superpowers](#your-superpowers)
+- **Reproducibility**: obtaining similar results is supported in multiple ways | [#Reproducibility](#reproducibility)
+- **Little Boilerplate**: so pipeline can be easily modified | [#How It Works](#how-it-works)
+- **Main Configuration**: main config file specifies default training configuration | [#Main Project Configuration](#main-project-configuration)
+- **Experiment Configurations**: can be composed out of smaller configs and override chosen hyperparameters | [#Experiment Configuration](#experiment-configuration)
+- **Workflow**: comes down to 4 simple steps | [#Workflow](#workflow)
+- **Experiment Tracking**: many logging frameworks can be easily integrated, like Tensorboard, MLFlow or W&B | [#Experiment Tracking](#experiment-tracking)
+- **Logs**: all logs (checkpoints, data from loggers, hparams, etc.) are stored in a convenient folder structure imposed by Hydra | [#Logs](#logs)
+- **Hyperparameter Search**: made easier with Hydra built-in plugins like [Optuna Sweeper](https://hydra.cc/docs/next/plugins/optuna_sweeper) | [#Hyperparameter Search](#hyperparameter-search)
+- **Tests**: unit tests and shell/command based tests for speeding up the development | [#Tests](#tests)
+- **Best Practices**: a couple of recommended tools, practices and standards for efficient workflow and reproducibility | [#Best Practices](#best-practices)
+
+<br>
+
+## Project Structure
+
+The directory structure of new project looks like this:
+
+```
+├── configs <- Hydra configuration files
+│ ├── callbacks <- Callbacks configs
+│ ├── datamodule <- Datamodule configs
+│ ├── debug <- Debugging configs
+│ ├── experiment <- Experiment configs
+│ ├── hparams_search <- Hyperparameter search configs
+│ ├── local <- Local configs
+│ ├── log_dir <- Logging directory configs
+│ ├── logger <- Logger configs
+│ ├── model <- Model configs
+│ ├── trainer <- Trainer configs
+│ │
+│ ├── test.yaml <- Main config for testing
+│ └── train.yaml <- Main config for training
+│
+├── data <- Project data
+│
+├── logs <- Logs generated by Hydra and PyTorch Lightning loggers
+│
+├── notebooks <- Jupyter notebooks. Naming convention is a number (for ordering),
+│ the creator's initials, and a short `-` delimited description,
+│ e.g. `1.0-jqp-initial-data-exploration.ipynb`.
+│
+├── scripts <- Shell scripts
+│
+├── src <- Source code
+│ ├── datamodules <- Lightning datamodules
+│ ├── models <- Lightning models
+│ ├── utils <- Utility scripts
+│ ├── vendor <- Third party code that cannot be installed using PIP/Conda
+│ │
+│ ├── testing_pipeline.py
+│ └── training_pipeline.py
+│
+├── tests <- Tests of any kind
+│ ├── helpers <- A couple of testing utilities
+│ ├── shell <- Shell/command based tests
+│ └── unit <- Unit tests
+│
+├── test.py <- Run testing
+├── train.py <- Run training
+│
+├── .env.example <- Template of the file for storing private environment variables
+├── .gitignore <- List of files/folders ignored by git
+├── .pre-commit-config.yaml <- Configuration of pre-commit hooks for code formatting
+├── requirements.txt <- File for installing python dependencies
+├── setup.cfg <- Configuration of linters and pytest
+└── README.md
+```
+
+<br>
+
+## 🚀&nbsp;&nbsp;Quickstart
+
+```bash
+# clone project
+git clone https://github.com/ashleve/lightning-hydra-template
+cd lightning-hydra-template
+
+# [OPTIONAL] create conda environment
+conda create -n myenv python=3.8
+conda activate myenv
+
+# install pytorch according to instructions
+# https://pytorch.org/get-started/
+
+# install requirements
+pip install -r requirements.txt
+```
+
+Template contains example with MNIST classification.<br>
+When running `python train.py` you should see something like this:
+
+<div align="center">
+
+![](https://github.com/ashleve/lightning-hydra-template/blob/resources/terminal.png)
+
+</div>
+
+### ⚡&nbsp;&nbsp;Your Superpowers
+
+<details>
+<summary><b>Override any config parameter from command line</b></summary>
+
+> Hydra allows you to easily overwrite any parameter defined in your config.
+
+```bash
+python train.py trainer.max_epochs=20 model.lr=1e-4
+```
+
+> You can also add new parameters with `+` sign.
+
+```bash
+python train.py +model.new_param="uwu"
+```
+
+</details>
+
+<details>
+<summary><b>Train on CPU, GPU, multi-GPU and TPU</b></summary>
+
+> PyTorch Lightning makes it easy to train your models on different hardware.
+
+```bash
+# train on CPU
+python train.py trainer.gpus=0
+
+# train on 1 GPU
+python train.py trainer.gpus=1
+
+# train on TPU
+python train.py +trainer.tpu_cores=8
+
+# train with DDP (Distributed Data Parallel) (4 GPUs)
+python train.py trainer.gpus=4 +trainer.strategy=ddp
+
+# train with DDP (Distributed Data Parallel) (8 GPUs, 2 nodes)
+python train.py trainer.gpus=4 +trainer.num_nodes=2 +trainer.strategy=ddp
+```
+
+</details>
+
+<details>
+<summary><b>Train with mixed precision</b></summary>
+
+```bash
+# train with pytorch native automatic mixed precision (AMP)
+python train.py trainer.gpus=1 +trainer.precision=16
+```
+
+</details>
+
+<!-- deepspeed support still in beta
+<details>
+<summary><b>Optimize large scale models on multiple GPUs with Deepspeed</b></summary>
+
+```bash
+python train.py +trainer.
+```
+
+</details>
+ -->
+
+<details>
+<summary><b>Train model with any logger available in PyTorch Lightning, like Weights&Biases or Tensorboard</b></summary>
+
+> PyTorch Lightning provides convenient integrations with most popular logging frameworks, like Tensorboard, Neptune or simple csv files. Read more [here](#experiment-tracking). Using wandb requires you to [setup account](https://www.wandb.com/) first. After that just complete the config as below.<br> > **Click [here](https://wandb.ai/hobglob/template-dashboard/) to see example wandb dashboard generated with this template.**
+
+```bash
+# set project and entity names in `configs/logger/wandb`
+wandb:
+ project: "your_project_name"
+ entity: "your_wandb_team_name"
+```
+
+```bash
+# train model with Weights&Biases (link to wandb dashboard should appear in the terminal)
+python train.py logger=wandb
+```
+
+</details>
+
+<details>
+<summary><b>Train model with chosen experiment config</b></summary>
+
+> Experiment configurations are placed in [configs/experiment/](configs/experiment/).
+
+```bash
+python train.py experiment=example
+```
+
+</details>
+
+<details>
+<summary><b>Attach some callbacks to run</b></summary>
+
+> Callbacks can be used for things such as as model checkpointing, early stopping and [many more](https://pytorch-lightning.readthedocs.io/en/latest/extensions/callbacks.html#built-in-callbacks).<br>
+> Callbacks configurations are placed in [configs/callbacks/](configs/callbacks/).
+
+```bash
+python train.py callbacks=default
+```
+
+</details>
+
+<details>
+<summary><b>Use different tricks available in Pytorch Lightning</b></summary>
+
+> PyTorch Lightning provides about [40+ useful trainer flags](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#trainer-flags).
+
+```yaml
+# gradient clipping may be enabled to avoid exploding gradients
+python train.py +trainer.gradient_clip_val=0.5
+
+# stochastic weight averaging can make your models generalize better
+python train.py +trainer.stochastic_weight_avg=true
+
+# run validation loop 4 times during a training epoch
+python train.py +trainer.val_check_interval=0.25
+
+# accumulate gradients
+python train.py +trainer.accumulate_grad_batches=10
+
+# terminate training after 12 hours
+python train.py +trainer.max_time="00:12:00:00"
+```
+
+</details>
+
+<details>
+<summary><b>Easily debug</b></summary>
+
+> Visit [configs/debug/](configs/debug/) for different debugging configs.
+
+```bash
+# runs 1 epoch in default debugging mode
+# changes logging directory to `logs/debugs/...`
+# sets level of all command line loggers to 'DEBUG'
+# enables extra trainer flags like tracking gradient norm
+# enforces debug-friendly configuration
+python train.py debug=default
+
+# runs test epoch without training
+python train.py debug=test_only
+
+# run 1 train, val and test loop, using only 1 batch
+python train.py +trainer.fast_dev_run=true
+
+# raise exception if there are any numerical anomalies in tensors, like NaN or +/-inf
+python train.py +trainer.detect_anomaly=true
+
+# print execution time profiling after training ends
+python train.py +trainer.profiler="simple"
+
+# try overfitting to 1 batch
+python train.py +trainer.overfit_batches=1 trainer.max_epochs=20
+
+# use only 20% of the data
+python train.py +trainer.limit_train_batches=0.2 \
++trainer.limit_val_batches=0.2 +trainer.limit_test_batches=0.2
+
+# log second gradient norm of the model
+python train.py +trainer.track_grad_norm=2
+```
+
+</details>
+
+<details>
+<summary><b>Resume training from checkpoint</b></summary>
+
+> Checkpoint can be either path or URL.
+
+```yaml
+python train.py trainer.resume_from_checkpoint="/path/to/ckpt/name.ckpt"
+```
+
+> ⚠️ Currently loading ckpt in Lightning doesn't resume logger experiment, but it will be supported in future Lightning release.
+
+</details>
+
+<details>
+<summary><b>Execute evaluation for a given checkpoint</b></summary>
+
+> Checkpoint can be either path or URL.
+
+```yaml
+python test.py ckpt_path="/path/to/ckpt/name.ckpt"
+```
+
+</details>
+
+<details>
+<summary><b>Create a sweep over hyperparameters</b></summary>
+
+```bash
+# this will run 6 experiments one after the other,
+# each with different combination of batch_size and learning rate
+python train.py -m datamodule.batch_size=32,64,128 model.lr=0.001,0.0005
+```
+
+> ⚠️ This sweep is not failure resistant (if one job crashes than the whole sweep crashes).
+
+</details>
+
+<details>
+<summary><b>Create a sweep over hyperparameters with Optuna</b></summary>
+
+> Using [Optuna Sweeper](https://hydra.cc/docs/next/plugins/optuna_sweeper) plugin doesn't require you to code any boilerplate into your pipeline, everything is defined in a [single config file](configs/hparams_search/mnist_optuna.yaml)!
+
+```bash
+# this will run hyperparameter search defined in `configs/hparams_search/mnist_optuna.yaml`
+# over chosen experiment config
+python train.py -m hparams_search=mnist_optuna experiment=example_simple
+```
+
+> ⚠️ Currently this sweep is not failure resistant (if one job crashes than the whole sweep crashes). Might be supported in future Hydra release.
+
+</details>
+
+<details>
+<summary><b>Execute all experiments from folder</b></summary>
+
+> Hydra provides special syntax for controlling behavior of multiruns. Learn more [here](https://hydra.cc/docs/next/tutorials/basic/running_your_app/multi-run). The command below executes all experiments from folder [configs/experiment/](configs/experiment/).
+
+```bash
+python train.py -m 'experiment=glob(*)'
+```
+
+</details>
+
+<details>
+<summary><b>Execute sweep on a remote AWS cluster</b></summary>
+
+> This should be achievable with simple config using [Ray AWS launcher for Hydra](https://hydra.cc/docs/next/plugins/ray_launcher). Example is not yet implemented in this template.
+
+</details>
+
+<!-- <details>
+<summary><b>Execute sweep on a SLURM cluster</b></summary>
+
+> This should be achievable with either [the right lightning trainer flags](https://pytorch-lightning.readthedocs.io/en/latest/clouds/cluster.html?highlight=SLURM#slurm-managed-cluster) or simple config using [Submitit launcher for Hydra](https://hydra.cc/docs/plugins/submitit_launcher). Example is not yet implemented in this template.
+
+</details> -->
+
+<details>
+<summary><b>Use Hydra tab completion</b></summary>
+
+> Hydra allows you to autocomplete config argument overrides in shell as you write them, by pressing `tab` key. Learn more [here](https://hydra.cc/docs/tutorials/basic/running_your_app/tab_completion).
+
+</details>
+
+<details>
+<summary><b>Apply pre-commit hooks</b></summary>
+
+> Apply pre-commit hooks to automatically format your code and configs, perform code analysis and remove output from jupyter notebooks. See [# Best Practices](#best-practices) for more.
+
+```bash
+pre-commit run -a
+```
+
+</details>
+
+<br>
+
+## ❤️&nbsp;&nbsp;Contributions
+
+Have a question? Found a bug? Missing a specific feature? Have an idea for improving documentation? Feel free to file a new issue, discussion or PR with respective title and description. If you already found a solution to your problem, don't hesitate to share it. Suggestions for new best practices are always welcome!
+
+<br>
+
+## ℹ️&nbsp;&nbsp;Guide
+
+### How To Get Started
+
+- First, you should probably get familiar with [PyTorch Lightning](https://www.pytorchlightning.ai)
+- Next, go through [Hydra quick start guide](https://hydra.cc/docs/intro/) and [basic Hydra tutorial](https://hydra.cc/docs/tutorials/basic/your_first_app/simple_cli/)
+
+<br>
+
+### How It Works
+
+All PyTorch Lightning modules are dynamically instantiated from module paths specified in config. Example model config:
+
+```yaml
+_target_: src.models.mnist_model.MNISTLitModule
+input_size: 784
+lin1_size: 256
+lin2_size: 256
+lin3_size: 256
+output_size: 10
+lr: 0.001
+```
+
+Using this config we can instantiate the object with the following line:
+
+```python
+model = hydra.utils.instantiate(config.model)
+```
+
+This allows you to easily iterate over new models! Every time you create a new one, just specify its module path and parameters in appropriate config file. <br>
+
+Switch between models and datamodules with command line arguments:
+
+```bash
+python train.py model=mnist
+```
+
+The whole pipeline managing the instantiation logic is placed in [src/training_pipeline.py](src/training_pipeline.py).
+
+<br>
+
+### Main Project Configuration
+
+Location: [configs/train.yaml](configs/train.yaml) <br>
+Main project config contains default training configuration.<br>
+It determines how config is composed when simply executing command `python train.py`.<br>
+
+<details>
+<summary><b>Show main project config</b></summary>
+
+```yaml
+# specify here default training configuration
+defaults:
+ - _self_
+ - datamodule: mnist.yaml
+ - model: mnist.yaml
+ - callbacks: default.yaml
+ - logger: null # set logger here or use command line (e.g. `python train.py logger=tensorboard`)
+ - trainer: default.yaml
+ - log_dir: default.yaml
+
+ # experiment configs allow for version control of specific configurations
+ # e.g. best hyperparameters for each combination of model and datamodule
+ - experiment: null
+
+ # debugging config (enable through command line, e.g. `python train.py debug=default)
+ - debug: null
+
+ # config for hyperparameter optimization
+ - hparams_search: null
+
+ # optional local config for machine/user specific settings
+ # it's optional since it doesn't need to exist and is excluded from version control
+ - optional local: default.yaml
+
+ # enable color logging
+ - override hydra/hydra_logging: colorlog
+ - override hydra/job_logging: colorlog
+
+# path to original working directory
+# hydra hijacks working directory by changing it to the new log directory
+# https://hydra.cc/docs/next/tutorials/basic/running_your_app/working_directory
+original_work_dir: ${hydra:runtime.cwd}
+
+# path to folder with data
+data_dir: ${original_work_dir}/data/
+
+# pretty print config at the start of the run using Rich library
+print_config: True
+
+# disable python warnings if they annoy you
+ignore_warnings: True
+
+# set False to skip model training
+train: True
+
+# evaluate on test set, using best model weights achieved during training
+# lightning chooses best weights based on the metric specified in checkpoint callback
+test: True
+
+# seed for random number generators in pytorch, numpy and python.random
+seed: null
+
+# default name for the experiment, determines logging folder path
+# (you can overwrite this name in experiment configs)
+name: "default"
+```
+
+</details>
+
+<br>
+
+### Experiment Configuration
+
+Location: [configs/experiment](configs/experiment)<br>
+Experiment configs allow you to overwrite parameters from main project configuration.<br>
+For example, you can use them to version control best hyperparameters for each combination of model and dataset.
+
+<details>
+<summary><b>Show example experiment config</b></summary>
+
+```yaml
+# to execute this experiment run:
+# python train.py experiment=example
+
+defaults:
+ - override /datamodule: mnist.yaml
+ - override /model: mnist.yaml
+ - override /callbacks: default.yaml
+ - override /logger: null
+ - override /trainer: default.yaml
+
+# all parameters below will be merged with parameters from default configurations set above
+# this allows you to overwrite only specified parameters
+
+# name of the run determines folder name in logs
+name: "simple_dense_net"
+
+seed: 12345
+
+trainer:
+ min_epochs: 10
+ max_epochs: 10
+ gradient_clip_val: 0.5
+
+model:
+ lin1_size: 128
+ lin2_size: 256
+ lin3_size: 64
+ lr: 0.002
+
+datamodule:
+ batch_size: 64
+
+logger:
+ wandb:
+ tags: ["mnist", "${name}"]
+```
+
+</details>
+
+<br>
+
+### Local Configuration
+
+Location: [configs/local](configs/local) <br>
+Some configurations are user/machine/installation specific (e.g. configuration of local cluster, or harddrive paths on a specific machine). For such scenarios, a file `configs/local/default.yaml` can be created which is automatically loaded but not tracked by Git.
+
+<details>
+<summary><b>Show example local Slurm cluster config</b></summary>
+
+```yaml
+# @package _global_
+
+defaults:
+ - override /hydra/launcher@_here_: submitit_slurm
+
+data_dir: /mnt/scratch/data/
+
+hydra:
+ launcher:
+ timeout_min: 1440
+ gpus_per_task: 1
+ gres: gpu:1
+ job:
+ env_set:
+ MY_VAR: /home/user/my/system/path
+ MY_KEY: asdgjhawi8y23ihsghsueity23ihwd
+```
+
+</details>
+
+<br>
+
+### Workflow
+
+1. Write your PyTorch Lightning module (see [models/mnist_module.py](src/models/mnist_module.py) for example)
+2. Write your PyTorch Lightning datamodule (see [datamodules/mnist_datamodule.py](src/datamodules/mnist_datamodule.py) for example)
+3. Write your experiment config, containing paths to your model and datamodule
+4. Run training with chosen experiment config: `python train.py experiment=experiment_name`
+
+<br>
+
+### Logs
+
+**Hydra creates new working directory for every executed run.** By default, logs have the following structure:
+
+```
+├── logs
+│ ├── experiments # Folder for the logs generated by experiments
+│ │ ├── runs # Folder for single runs
+│ │ │ ├── experiment_name # Experiment name
+│ │ │ │ ├── YYYY-MM-DD_HH-MM-SS # Datetime of the run
+│ │ │ │ │ ├── .hydra # Hydra logs
+│ │ │ │ │ ├── csv # Csv logs
+│ │ │ │ │ ├── wandb # Weights&Biases logs
+│ │ │ │ │ ├── checkpoints # Training checkpoints
+│ │ │ │ │ └── ... # Any other thing saved during training
+│ │ │ │ └── ...
+│ │ │ └── ...
+│ │ │
+│ │ └── multiruns # Folder for multiruns
+│ │ ├── experiment_name # Experiment name
+│ │ │ ├── YYYY-MM-DD_HH-MM-SS # Datetime of the multirun
+│ │ │ │ ├──1 # Multirun job number
+│ │ │ │ ├──2
+│ │ │ │ └── ...
+│ │ │ └── ...
+│ │ └── ...
+│ │
+│ ├── evaluations # Folder for the logs generated during testing
+│ │ └── ...
+│ │
+│ └── debugs # Folder for the logs generated during debugging
+│ └── ...
+```
+
+You can change this structure by modifying paths in [hydra configuration](configs/log_dir).
+
+<br>
+
+### Experiment Tracking
+
+PyTorch Lightning supports many popular logging frameworks:<br>
+**[Weights&Biases](https://www.wandb.com/) · [Neptune](https://neptune.ai/) · [Comet](https://www.comet.ml/) · [MLFlow](https://mlflow.org) · [Tensorboard](https://www.tensorflow.org/tensorboard/)**
+
+These tools help you keep track of hyperparameters and output metrics and allow you to compare and visualize results. To use one of them simply complete its configuration in [configs/logger](configs/logger) and run:
+
+```bash
+python train.py logger=logger_name
+```
+
+You can use many of them at once (see [configs/logger/many_loggers.yaml](configs/logger/many_loggers.yaml) for example).
+
+You can also write your own logger.
+
+Lightning provides convenient method for logging custom metrics from inside LightningModule. Read the docs [here](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html#automatic-logging) or take a look at [MNIST example](src/models/mnist_module.py).
+
+<br>
+
+### Hyperparameter Search
+
+Defining hyperparameter optimization is as easy as adding new config file to [configs/hparams_search](configs/hparams_search).
+
+<details>
+<summary><b>Show example</b></summary>
+
+```yaml
+defaults:
+ - override /hydra/sweeper: optuna
+
+# choose metric which will be optimized by Optuna
+optimized_metric: "val/acc_best"
+
+hydra:
+ # here we define Optuna hyperparameter search
+ # it optimizes for value returned from function with @hydra.main decorator
+ # learn more here: https://hydra.cc/docs/next/plugins/optuna_sweeper
+ sweeper:
+ _target_: hydra_plugins.hydra_optuna_sweeper.optuna_sweeper.OptunaSweeper
+ storage: null
+ study_name: null
+ n_jobs: 1
+
+ # 'minimize' or 'maximize' the objective
+ direction: maximize
+
+ # number of experiments that will be executed
+ n_trials: 20
+
+ # choose Optuna hyperparameter sampler
+ # learn more here: https://optuna.readthedocs.io/en/stable/reference/samplers.html
+ sampler:
+ _target_: optuna.samplers.TPESampler
+ seed: 12345
+ consider_prior: true
+ prior_weight: 1.0
+ consider_magic_clip: true
+ consider_endpoints: false
+ n_startup_trials: 10
+ n_ei_candidates: 24
+ multivariate: false
+ warn_independent_sampling: true
+
+ # define range of hyperparameters
+ search_space:
+ datamodule.batch_size:
+ type: categorical
+ choices: [32, 64, 128]
+ model.lr:
+ type: float
+ low: 0.0001
+ high: 0.2
+ model.lin1_size:
+ type: categorical
+ choices: [32, 64, 128, 256, 512]
+ model.lin2_size:
+ type: categorical
+ choices: [32, 64, 128, 256, 512]
+ model.lin3_size:
+ type: categorical
+ choices: [32, 64, 128, 256, 512]
+```
+
+</details>
+
+Next, you can execute it with: `python train.py -m hparams_search=mnist_optuna`
+
+Using this approach doesn't require you to add any boilerplate into your pipeline, everything is defined in a single config file.
+
+You can use different optimization frameworks integrated with Hydra, like Optuna, Ax or Nevergrad.
+
+The `optimization_results.yaml` will be available under `logs/multirun` folder.
+
+This approach doesn't support advanced technics like prunning - for more sophisticated search, you probably shouldn't use hydra multirun feature and instead write your own optimization pipeline.
+
+<br>
+
+### Inference
+
+The following code is an example of loading model from checkpoint and running predictions.<br>
+
+<details>
+<summary><b>Show example</b></summary>
+
+```python
+from PIL import Image
+from torchvision import transforms
+
+from src.models.mnist_module import MNISTLitModule
+
+
+def predict():
+ """Example of inference with trained model.
+ It loads trained image classification model from checkpoint.
+ Then it loads example image and predicts its label.
+ """
+
+ # ckpt can be also a URL!
+ CKPT_PATH = "last.ckpt"
+
+ # load model from checkpoint
+ # model __init__ parameters will be loaded from ckpt automatically
+ # you can also pass some parameter explicitly to override it
+ trained_model = MNISTLitModule.load_from_checkpoint(checkpoint_path=CKPT_PATH)
+
+ # print model hyperparameters
+ print(trained_model.hparams)
+
+ # switch to evaluation mode
+ trained_model.eval()
+ trained_model.freeze()
+
+ # load data
+ img = Image.open("data/example_img.png").convert("L") # convert to black and white
+ # img = Image.open("data/example_img.png").convert("RGB") # convert to RGB
+
+ # preprocess
+ mnist_transforms = transforms.Compose(
+ [
+ transforms.ToTensor(),
+ transforms.Resize((28, 28)),
+ transforms.Normalize((0.1307,), (0.3081,)),
+ ]
+ )
+ img = mnist_transforms(img)
+ img = img.reshape((1, *img.size())) # reshape to form batch of size 1
+
+ # inference
+ output = trained_model(img)
+ print(output)
+
+
+if __name__ == "__main__":
+ predict()
+
+```
+
+</details>
+
+<br>
+
+### Tests
+
+Template comes with example tests implemented with pytest library. To execute them simply run:
+
+```bash
+# run all tests
+pytest
+
+# run tests from specific file
+pytest tests/shell/test_basic_commands.py
+
+# run all tests except the ones marked as slow
+pytest -k "not slow"
+```
+
+To speed up the development, you can once in a while execute tests that run a couple of quick experiments, like training 1 epoch on 25% of data, executing single train/val/test step, etc. Those kind of tests don't check for any specific output - they exist to simply verify that executing some bash commands doesn't end up in throwing exceptions. You can find them implemented in [tests/shell](tests/shell) folder.
+
+You can easily modify the commands in the scripts for your use case. If 1 epoch is too much for your model, then make it run for a couple of batches instead (by using the right trainer flags).
+
+<br>
+
+### Callbacks
+
+The branch [`wandb-callbacks`](https://github.com/ashleve/lightning-hydra-template/tree/wandb-callbacks) contains example callbacks enabling better Weights&Biases integration, which you can use as a reference for writing your own callbacks (see [wandb_callbacks.py](https://github.com/ashleve/lightning-hydra-template/tree/wandb-callbacks/src/callbacks/wandb_callbacks.py)).
+
+Callbacks which support reproducibility:
+
+- **WatchModel**
+- **UploadCodeAsArtifact**
+- **UploadCheckpointsAsArtifact**
+
+Callbacks which provide examples of logging custom visualisations:
+
+- **LogConfusionMatrix**
+- **LogF1PrecRecHeatmap**
+- **LogImagePredictions**
+
+To try all of the callbacks at once, switch to the right branch:
+
+```bash
+git checkout wandb-callbacks
+```
+
+And then run the following command:
+
+```bash
+python train.py logger=wandb callbacks=wandb
+```
+
+To see the result of all the callbacks attached, take a look at [this experiment dashboard](https://wandb.ai/hobglob/template-tests/runs/3rw7q70h).
+
+<br>
+
+### Multi-GPU Training
+
+Lightning supports multiple ways of doing distributed training. The most common one is DDP, which spawns separate process for each GPU and averages gradients between them. To learn about other approaches read the [lightning docs](https://pytorch-lightning.readthedocs.io/en/latest/advanced/multi_gpu.html).
+
+You can run DDP on mnist example with 4 GPUs like this:
+
+```bash
+python train.py trainer.gpus=4 +trainer.strategy=ddp
+```
+
+⚠️ When using DDP you have to be careful how you write your models - learn more [here](https://pytorch-lightning.readthedocs.io/en/latest/advanced/multi_gpu.html).
+
+<br>
+
+### Docker
+
+First you will need to [install Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) to enable GPU support.
+
+The template Dockerfile is provided on branch [`dockerfiles`](https://github.com/ashleve/lightning-hydra-template/tree/dockerfiles). Copy it to the template root folder.
+
+To build the container use:
+
+```bash
+docker build -t <project_name> .
+```
+
+To mount the project to the container use:
+
+```bash
+docker run -v $(pwd):/workspace/project --gpus all -it --rm <project_name>
+```
+
+<br>
+
+### Reproducibility
+
+What provides reproducibility:
+
+- Hydra manages your configs
+- Hydra manages your logging paths and makes every executed run store its hyperparameters and config overrides in a separate file in logs
+- Single seed for random number generators in pytorch, numpy and python.random
+- LightningDataModule allows you to encapsulate data split, transformations and default parameters in a single, clean abstraction
+- LightningModule separates your research code from engineering code in a clean way
+- Experiment tracking frameworks take care of logging metrics and hparams, some can also store results and artifacts in cloud
+- Pytorch Lightning takes care of creating training checkpoints
+- Example callbacks for wandb show how you can save and upload a snapshot of codebase every time the run is executed, as well as upload ckpts and track model gradients
+
+<!--
+You can load the config of previous run using:
+
+```bash
+python train.py --config-path /logs/runs/.../.hydra/ --config-name config.yaml
+```
+
+The `config.yaml` from `.hydra` folder contains all overriden parameters and sections. This approach however is not officially supported by Hydra and doesn't override the `hydra/` part of the config, meaning logging paths will revert to default!
+ -->
+<br>
+
+### Limitations
+
+- Currently, template doesn't support k-fold cross validation, but it's possible to achieve it with Lightning Loop interface. See the [official example](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/loop_examples/kfold.py). Implementing it requires rewriting the training pipeline.
+- Pytorch Lightning might not be the best choice for scalable reinforcement learning, it's probably better to use something like [Ray](https://github.com/ray-project/ray).
+- Currently hyperparameter search with Hydra Optuna Plugin doesn't support prunning.
+- Hydra changes working directory to new logging folder for every executed run, which might not be compatible with the way some libraries work.
+
+<br>
+
+## Useful Tricks
+
+<details>
+<summary><b>Accessing datamodule attributes in model</b></summary>
+
+1. The simplest way is to pass datamodule attribute directly to model on initialization:
+
+ ```python
+ # ./src/training_pipeline.py
+ datamodule = hydra.utils.instantiate(config.datamodule)
+ model = hydra.utils.instantiate(config.model, some_param=datamodule.some_param)
+ ```
+
+ This is not a very robust solution, since it assumes all your datamodules have `some_param` attribute available (otherwise the run will crash).
+
+2. If you only want to access datamodule config, you can simply pass it as an init parameter:
+
+ ```python
+ # ./src/training_pipeline.py
+ model = hydra.utils.instantiate(config.model, dm_conf=config.datamodule, _recursive_=False)
+ ```
+
+ Now you can access any datamodule config part like this:
+
+ ```python
+ # ./src/models/my_model.py
+ class MyLitModel(LightningModule):
+ def __init__(self, dm_conf, param1, param2):
+ super().__init__()
+
+ batch_size = dm_conf.batch_size
+ ```
+
+3. If you need to access the datamodule object attributes, a little hacky solution is to add Omegaconf resolver to your datamodule:
+
+ ```python
+ # ./src/datamodules/my_datamodule.py
+ from omegaconf import OmegaConf
+
+ class MyDataModule(LightningDataModule):
+ def __init__(self, param1, param2):
+ super().__init__()
+
+ self.param1 = param1
+
+ resolver_name = "datamodule"
+ OmegaConf.register_new_resolver(
+ resolver_name,
+ lambda name: getattr(self, name),
+ use_cache=False
+ )
+ ```
+
+ This way you can reference any datamodule attribute from your config like this:
+
+ ```yaml
+ # this will return attribute 'param1' from datamodule object
+ param1: ${datamodule: param1}
+ ```
+
+ When later accessing this field, say in your lightning model, it will get automatically resolved based on all resolvers that are registered. Remember not to access this field before datamodule is initialized or it will crash. **You also need to set `resolve=False` in `print_config()` in [train.py](train.py) or it will throw errors:**
+
+ ```python
+ # ./src/train.py
+ utils.print_config(config, resolve=False)
+ ```
+
+</details>
+
+<details>
+<summary><b>Automatic activation of virtual environment and tab completion when entering folder</b></summary>
+
+1. Create a new file called `.autoenv` (this name is excluded from version control in `.gitignore`). <br>
+ You can use it to automatically execute shell commands when entering folder. Add some commands to your `.autoenv` file, like in the example below:
+
+ ```bash
+ # activate conda environment
+ conda activate myenv
+
+ # activate hydra tab completion for bash
+ eval "$(python train.py -sc install=bash)"
+ ```
+
+ (these commands will be executed whenever you're openning or switching terminal to folder containing `.autoenv` file)
+
+2. To setup this automation for bash, execute the following line (it will append your `.bashrc` file):
+
+ ```bash
+ echo "autoenv() { if [ -x .autoenv ]; then source .autoenv ; echo '.autoenv executed' ; fi } ; cd() { builtin cd \"\$@\" ; autoenv ; } ; autoenv" >> ~/.bashrc
+ ```
+
+3. Lastly add execution previliges to your `.autoenv` file:
+
+ ```
+ chmod +x .autoenv
+ ```
+
+ (for safety, only `.autoenv` with previligies will be executed)
+
+**Explanation**
+
+The mentioned line appends your `.bashrc` file with 2 commands:
+
+1. `autoenv() { if [ -x .autoenv ]; then source .autoenv ; echo '.autoenv executed' ; fi }` - this declares the `autoenv()` function, which executes `.autoenv` file if it exists in current work dir and has execution previligies
+2. `cd() { builtin cd \"\$@\" ; autoenv ; } ; autoenv` - this extends behaviour of `cd` command, to make it execute `autoenv()` function each time you change folder in terminal or open new terminal
+
+</details>
+
+<!--
+<details>
+<summary><b>Making sweeps failure resistant</b></summary>
+
+TODO
+
+</details>
+ -->
+
+<br>
+
+## Best Practices
+
+<details>
+<summary><b>Use Miniconda for GPU environments</b></summary>
+
+Use miniconda for your python environments (it's usually unnecessary to install full anaconda environment, miniconda should be enough).
+It makes it easier to install some dependencies, like cudatoolkit for GPU support. It also allows you to acccess your environments globally.
+
+Example installation:
+
+```bash
+wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh
+```
+
+Create new conda environment:
+
+```bash
+conda create -n myenv python=3.8
+conda activate myenv
+```
+
+</details>
+
+<details>
+<summary><b>Use automatic code formatting</b></summary>
+
+Use pre-commit hooks to standardize code formatting of your project and save mental energy.<br>
+Simply install pre-commit package with:
+
+```bash
+pip install pre-commit
+```
+
+Next, install hooks from [.pre-commit-config.yaml](.pre-commit-config.yaml):
+
+```bash
+pre-commit install
+```
+
+After that your code will be automatically reformatted on every new commit.<br>
+Currently template contains configurations of **black** (python code formatting), **isort** (python import sorting), **flake8** (python code analysis), **prettier** (yaml formating) and **nbstripout** (clearing output from jupyter notebooks). <br>
+
+To reformat all files in the project use command:
+
+```bash
+pre-commit run -a
+```
+
+</details>
+
+<details>
+<summary><b>Set private environment variables in .env file</b></summary>
+
+System specific variables (e.g. absolute paths to datasets) should not be under version control or it will result in conflict between different users. Your private keys also shouldn't be versioned since you don't want them to be leaked.<br>
+
+Template contains `.env.example` file, which serves as an example. Create a new file called `.env` (this name is excluded from version control in .gitignore).
+You should use it for storing environment variables like this:
+
+```
+MY_VAR=/home/user/my_system_path
+```
+
+All variables from `.env` are loaded in `train.py` automatically.
+
+Hydra allows you to reference any env variable in `.yaml` configs like this:
+
+```yaml
+path_to_data: ${oc.env:MY_VAR}
+```
+
+</details>
+
+<details>
+<summary><b>Name metrics using '/' character</b></summary>
+
+Depending on which logger you're using, it's often useful to define metric name with `/` character:
+
+```python
+self.log("train/loss", loss)
+```
+
+This way loggers will treat your metrics as belonging to different sections, which helps to get them organised in UI.
+
+</details>
+
+<details>
+<summary><b>Use torchmetrics</b></summary>
+
+Use official [torchmetrics](https://github.com/PytorchLightning/metrics) library to ensure proper calculation of metrics. This is especially important for multi-GPU training!
+
+For example, instead of calculating accuracy by yourself, you should use the provided `Accuracy` class like this:
+
+```python
+from torchmetrics.classification.accuracy import Accuracy
+
+
+class LitModel(LightningModule):
+ def __init__(self)
+ self.train_acc = Accuracy()
+ self.val_acc = Accuracy()
+
+ def training_step(self, batch, batch_idx):
+ ...
+ acc = self.train_acc(predictions, targets)
+ self.log("train/acc", acc)
+ ...
+
+ def validation_step(self, batch, batch_idx):
+ ...
+ acc = self.val_acc(predictions, targets)
+ self.log("val/acc", acc)
+ ...
+```
+
+Make sure to use different metric instance for each step to ensure proper value reduction over all GPU processes.
+
+Torchmetrics provides metrics for most use cases, like F1 score or confusion matrix. Read [documentation](https://torchmetrics.readthedocs.io/en/latest/#more-reading) for more.
+
+</details>
+
+<details>
+<summary><b>Follow PyTorch Lightning style guide</b></summary>
+
+The style guide is available [here](https://pytorch-lightning.readthedocs.io/en/latest/starter/style_guide.html).<br>
+
+1. Be explicit in your init. Try to define all the relevant defaults so that the user doesn’t have to guess. Provide type hints. This way your module is reusable across projects!
+
+ ```python
+ class LitModel(LightningModule):
+ def __init__(self, layer_size: int = 256, lr: float = 0.001):
+ ```
+
+2. Preserve the recommended method order.
+
+ ```python
+ class LitModel(LightningModule):
+
+ def __init__():
+ ...
+
+ def forward():
+ ...
+
+ def training_step():
+ ...
+
+ def training_step_end():
+ ...
+
+ def training_epoch_end():
+ ...
+
+ def validation_step():
+ ...
+
+ def validation_step_end():
+ ...
+
+ def validation_epoch_end():
+ ...
+
+ def test_step():
+ ...
+
+ def test_step_end():
+ ...
+
+ def test_epoch_end():
+ ...
+
+ def configure_optimizers():
+ ...
+
+ def any_extra_hook():
+ ...
+ ```
+
+</details>
+
+<details>
+<summary><b>Version control your data and models with DVC</b></summary>
+
+Use [DVC](https://dvc.org) to version control big files, like your data or trained ML models.<br>
+To initialize the dvc repository:
+
+```bash
+dvc init
+```
+
+To start tracking a file or directory, use `dvc add`:
+
+```bash
+dvc add data/MNIST
+```
+
+DVC stores information about the added file (or a directory) in a special .dvc file named data/MNIST.dvc, a small text file with a human-readable format. This file can be easily versioned like source code with Git, as a placeholder for the original data:
+
+```bash
+git add data/MNIST.dvc data/.gitignore
+git commit -m "Add raw data"
+```
+
+</details>
+
+<details>
+<summary><b>Support installing project as a package</b></summary>
+
+It allows other people to easily use your modules in their own projects.
+Change name of the `src` folder to your project name and add `setup.py` file:
+
+```python
+from setuptools import find_packages, setup
+
+
+setup(
+ name="src", # change "src" folder name to your project name
+ version="0.0.0",
+ description="Describe Your Cool Project",
+ author="...",
+ author_email="...",
+ url="https://github.com/ashleve/lightning-hydra-template", # replace with your own github project link
+ install_requires=[
+ "pytorch>=1.10.0",
+ "pytorch-lightning>=1.4.0",
+ "hydra-core>=1.1.0",
+ ],
+ packages=find_packages(),
+)
+```
+
+Now your project can be installed from local files:
+
+```bash
+pip install -e .
+```
+
+Or directly from git repository:
+
+```bash
+pip install git+git://github.com/YourGithubName/your-repo-name.git --upgrade
+```
+
+So any file can be easily imported into any other file like so:
+
+```python
+from project_name.models.mnist_module import MNISTLitModule
+from project_name.datamodules.mnist_datamodule import MNISTDataModule
+```
+
+</details>
+
+<!-- <details>
+<summary><b>Make notebooks independent from other files</b></summary>
+
+It's a good practice for jupyter notebooks to be portable. Try to make them independent from src files. If you need to access external code, try to embed it inside the notebook.
+
+</details> -->
+
+<!--<details>
+<summary><b>Use Docker</b></summary>
+
+Docker makes it easy to initialize the whole training environment, e.g. when you want to execute experiments in cloud or on some private computing cluster. You can extend [dockerfiles](https://github.com/ashleve/lightning-hydra-template/tree/dockerfiles) provided in the template with your own instructions for building the image.<br>
+
+</details> -->
+
+<br>
+
+## Other Repositories
+
+<details>
+<summary><b>Inspirations</b></summary>
+
+This template was inspired by:
+[PyTorchLightning/deep-learninig-project-template](https://github.com/PyTorchLightning/deep-learning-project-template),
+[drivendata/cookiecutter-data-science](https://github.com/drivendata/cookiecutter-data-science),
+[tchaton/lightning-hydra-seed](https://github.com/tchaton/lightning-hydra-seed),
+[Erlemar/pytorch_tempest](https://github.com/Erlemar/pytorch_tempest),
+[lucmos/nn-template](https://github.com/lucmos/nn-template).
+
+</details>
+
+<details>
+<summary><b>Useful repositories</b></summary>
+
+- [pytorch/hydra-torch](https://github.com/pytorch/hydra-torch) - resources for configuring PyTorch classes with Hydra,
+- [romesco/hydra-lightning](https://github.com/romesco/hydra-lightning) - resources for configuring PyTorch Lightning classes with Hydra
+- [lucmos/nn-template](https://github.com/lucmos/nn-template) - similar template
+- [PyTorchLightning/lightning-transformers](https://github.com/PyTorchLightning/lightning-transformers) - official Lightning Transformers repo built with Hydra
+
+</details>
+
+<!-- ## :star:&nbsp; Stargazers Over Time
+[![Stargazers over time](https://starchart.cc/ashleve/lightning-hydra-template.svg)](https://starchart.cc/ashleve/lightning-hydra-template) -->
+
+<br>
+
+## License
+
+This project is licensed under the MIT License.
+
+```
+MIT License
+
+Copyright (c) 2021 ashleve
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+```
+
+<br>
+<br>
+<br>
+<br>
+
+**DELETE EVERYTHING ABOVE FOR YOUR PROJECT**
+
+---
+
+<div align="center">
+
+# Your Project Name
+
+<a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch&logoColor=white"></a>
+<a href="https://pytorchlightning.ai/"><img alt="Lightning" src="https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning&logoColor=white"></a>
+<a href="https://hydra.cc/"><img alt="Config: Hydra" src="https://img.shields.io/badge/Config-Hydra-89b8cd"></a>
+<a href="https://github.com/ashleve/lightning-hydra-template"><img alt="Template" src="https://img.shields.io/badge/-Lightning--Hydra--Template-017F2F?style=flat&logo=github&labelColor=gray"></a><br>
+[![Paper](http://img.shields.io/badge/paper-arxiv.1001.2234-B31B1B.svg)](https://www.nature.com/articles/nature14539)
+[![Conference](http://img.shields.io/badge/AnyConference-year-4b44ce.svg)](https://papers.nips.cc/paper/2020)
+
+</div>
+
+## Description
+
+What it does
+
+## How to run
+
+Install dependencies
+
+```bash
+# clone project
+git clone https://github.com/YourGithubName/your-repo-name
+cd your-repo-name
+
+# [OPTIONAL] create conda environment
+conda create -n myenv python=3.8
+conda activate myenv
+
+# install pytorch according to instructions
+# https://pytorch.org/get-started/
+
+# install requirements
+pip install -r requirements.txt
+```
+
+Train model with default configuration
+
+```bash
+# train on CPU
+python train.py trainer.gpus=0
+
+# train on GPU
+python train.py trainer.gpus=1
+```
+
+Train model with chosen experiment configuration from [configs/experiment/](configs/experiment/)
+
+```bash
+python train.py experiment=experiment_name.yaml
+```
+
+You can override any parameter from command line like this
+
+```bash
+python train.py trainer.max_epochs=20 datamodule.batch_size=64
+```

models/configs/callbacks/default.yaml

@@ -0,0 +1,24 @@
+model_checkpoint:
+ _target_: pytorch_lightning.callbacks.ModelCheckpoint
+ monitor: "val/acc" # name of the logged metric which determines when model is improving
+ mode: "max" # "max" means higher metric value is better, can be also "min"
+ save_top_k: 1 # save k best models (determined by above metric)
+ save_last: True # additionaly always save model from last epoch
+ verbose: False
+ dirpath: "checkpoints/"
+ filename: "epoch_{epoch:03d}"
+ auto_insert_metric_name: False
+
+early_stopping:
+ _target_: pytorch_lightning.callbacks.EarlyStopping
+ monitor: "val/acc" # name of the logged metric which determines when model is improving
+ mode: "max" # "max" means higher metric value is better, can be also "min"
+ patience: 100 # how many validation epochs of not improving until training stops
+ min_delta: 0 # minimum change in the monitored metric needed to qualify as an improvement
+
+model_summary:
+ _target_: pytorch_lightning.callbacks.RichModelSummary
+ max_depth: -1
+
+rich_progress_bar:
+ _target_: pytorch_lightning.callbacks.RichProgressBar

models/configs/datamodule/mnist.yaml

@@ -0,0 +1,7 @@
+_target_: src.datamodules.mnist_datamodule.MNISTDataModule
+
+data_dir: ${data_dir} # data_dir is specified in config.yaml
+batch_size: 64
+train_val_test_split: [55_000, 5_000, 10_000]
+num_workers: 0
+pin_memory: False

models/configs/debug/default.yaml

@@ -0,0 +1,28 @@
+# @package _global_
+
+# default debugging setup, runs 1 full epoch
+# other debugging configs can inherit from this one
+
+defaults:
+ - override /log_dir: debug.yaml
+
+trainer:
+ max_epochs: 1
+ gpus: 0 # debuggers don't like gpus
+ detect_anomaly: true # raise exception if NaN or +/-inf is detected in any tensor
+ track_grad_norm: 2 # track gradient norm with loggers
+
+datamodule:
+ num_workers: 0 # debuggers don't like multiprocessing
+ pin_memory: False # disable gpu memory pin
+
+# sets level of all command line loggers to 'DEBUG'
+# https://hydra.cc/docs/tutorials/basic/running_your_app/logging/
+hydra:
+ verbose: True
+
+ # use this to set level of only chosen command line loggers to 'DEBUG':
+ # verbose: [src.train, src.utils]
+
+# config is already printed by hydra when `hydra/verbose: True`
+print_config: False

models/configs/debug/limit_batches.yaml

@@ -0,0 +1,12 @@
+# @package _global_
+
+# uses only 1% of the training data and 5% of validation/test data
+
+defaults:
+ - default.yaml
+
+trainer:
+ max_epochs: 3
+ limit_train_batches: 0.01
+ limit_val_batches: 0.05
+ limit_test_batches: 0.05

models/configs/debug/overfit.yaml

@@ -0,0 +1,10 @@
+# @package _global_
+
+# overfits to 3 batches
+
+defaults:
+ - default.yaml
+
+trainer:
+ max_epochs: 20
+ overfit_batches: 3

models/configs/debug/profiler.yaml

@@ -0,0 +1,12 @@
+# @package _global_
+
+# runs with execution time profiling
+
+defaults:
+ - default.yaml
+
+trainer:
+ max_epochs: 1
+ profiler: "simple"
+ # profiler: "advanced"
+ # profiler: "pytorch"

models/configs/debug/step.yaml

@@ -0,0 +1,9 @@
+# @package _global_
+
+# runs 1 train, 1 validation and 1 test step
+
+defaults:
+ - default.yaml
+
+trainer:
+ fast_dev_run: true

models/configs/debug/test_only.yaml

@@ -0,0 +1,9 @@
+# @package _global_
+
+# runs only test epoch
+
+defaults:
+ - default.yaml
+
+train: False
+test: True

models/configs/experiment/example.yaml

@@ -0,0 +1,37 @@
+# @package _global_
+
+# to execute this experiment run:
+# python train.py experiment=example
+
+defaults:
+ - override /datamodule: mnist.yaml
+ - override /model: mnist.yaml
+ - override /callbacks: default.yaml
+ - override /logger: null
+ - override /trainer: default.yaml
+
+# all parameters below will be merged with parameters from default configurations set above
+# this allows you to overwrite only specified parameters
+
+# name of the run determines folder name in logs
+name: "simple_dense_net"
+
+seed: 12345
+
+trainer:
+ min_epochs: 10
+ max_epochs: 10
+ gradient_clip_val: 0.5
+
+model:
+ lin1_size: 128
+ lin2_size: 256
+ lin3_size: 64
+ lr: 0.002
+
+datamodule:
+ batch_size: 64
+
+logger:
+ wandb:
+ tags: ["mnist", "${name}"]

models/configs/hparams_search/mnist_optuna.yaml

@@ -0,0 +1,60 @@
+# @package _global_
+
+# example hyperparameter optimization of some experiment with Optuna:
+# python train.py -m hparams_search=mnist_optuna experiment=example
+
+defaults:
+ - override /hydra/sweeper: optuna
+
+# choose metric which will be optimized by Optuna
+# make sure this is the correct name of some metric logged in lightning module!
+optimized_metric: "val/acc_best"
+
+# here we define Optuna hyperparameter search
+# it optimizes for value returned from function with @hydra.main decorator
+# docs: https://hydra.cc/docs/next/plugins/optuna_sweeper
+hydra:
+ sweeper:
+ _target_: hydra_plugins.hydra_optuna_sweeper.optuna_sweeper.OptunaSweeper
+
+ # storage URL to persist optimization results
+ # for example, you can use SQLite if you set 'sqlite:///example.db'
+ storage: null
+
+ # name of the study to persist optimization results
+ study_name: null
+
+ # number of parallel workers
+ n_jobs: 1
+
+ # 'minimize' or 'maximize' the objective
+ direction: maximize
+
+ # total number of runs that will be executed
+ n_trials: 25
+
+ # choose Optuna hyperparameter sampler
+ # docs: https://optuna.readthedocs.io/en/stable/reference/samplers.html
+ sampler:
+ _target_: optuna.samplers.TPESampler
+ seed: 12345
+ n_startup_trials: 10 # number of random sampling runs before optimization starts
+
+ # define range of hyperparameters
+ search_space:
+ datamodule.batch_size:
+ type: categorical
+ choices: [32, 64, 128]
+ model.lr:
+ type: float
+ low: 0.0001
+ high: 0.2
+ model.lin1_size:
+ type: categorical
+ choices: [32, 64, 128, 256, 512]
+ model.lin2_size:
+ type: categorical
+ choices: [32, 64, 128, 256, 512]
+ model.lin3_size:
+ type: categorical
+ choices: [32, 64, 128, 256, 512]

models/configs/log_dir/debug.yaml

@@ -0,0 +1,8 @@
+# @package _global_
+
+hydra:
+ run:
+ dir: logs/debugs/runs/${name}/${now:%Y-%m-%d}_${now:%H-%M-%S}
+ sweep:
+ dir: logs/debugs/multiruns/${name}/${now:%Y-%m-%d}_${now:%H-%M-%S}
+ subdir: ${hydra.job.num}

models/configs/log_dir/default.yaml

@@ -0,0 +1,8 @@
+# @package _global_
+
+hydra:
+ run:
+ dir: logs/experiments/runs/${name}/${now:%Y-%m-%d}_${now:%H-%M-%S}
+ sweep:
+ dir: logs/experiments/multiruns/${name}/${now:%Y-%m-%d}_${now:%H-%M-%S}
+ subdir: ${hydra.job.num}

models/configs/log_dir/evaluation.yaml

@@ -0,0 +1,8 @@
+# @package _global_
+
+hydra:
+ run:
+ dir: logs/evaluations/runs/${name}/${now:%Y-%m-%d}_${now:%H-%M-%S}
+ sweep:
+ dir: logs/evaluations/multiruns/${name}/${now:%Y-%m-%d}_${now:%H-%M-%S}
+ subdir: ${hydra.job.num}

models/configs/logger/comet.yaml

@@ -0,0 +1,7 @@
+# https://www.comet.ml
+
+comet:
+ _target_: pytorch_lightning.loggers.comet.CometLogger
+ api_key: ${oc.env:COMET_API_TOKEN} # api key is loaded from environment variable
+ project_name: "template-tests"
+ experiment_name: ${name}

models/configs/logger/csv.yaml

@@ -0,0 +1,7 @@
+# csv logger built in lightning
+
+csv:
+ _target_: pytorch_lightning.loggers.csv_logs.CSVLogger
+ save_dir: "."
+ name: "csv/"
+ prefix: ""

models/configs/logger/many_loggers.yaml

@@ -0,0 +1,9 @@
+# train with many loggers at once
+
+defaults:
+ # - comet.yaml
+ - csv.yaml
+ # - mlflow.yaml
+ # - neptune.yaml
+ - tensorboard.yaml
+ - wandb.yaml

models/configs/logger/mlflow.yaml

@@ -0,0 +1,10 @@
+# https://mlflow.org
+
+mlflow:
+ _target_: pytorch_lightning.loggers.mlflow.MLFlowLogger
+ experiment_name: ${name}
+ tracking_uri: null
+ tags: null
+ save_dir: ./mlruns
+ prefix: ""
+ artifact_location: null