Notebooks/SavtaDepth_Colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SavtaDepth Colab.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKUz-9qCbHRH"
      },
      "source": [
        "<center> <img alt=\\\"DAGsHub\\\" width=500px src=https://raw.githubusercontent.com/DAGsHub/client/master/dagshub_github.png> </center>\n",
        "\n",
        "# SavtaDepth Colab Environment\n",
        "\n",
        "### ***This notebook is meant to be run on Google Colab***\n",
        "\n",
        "This notebook is a tool to setup and orchestrate (read: Run) experiments for [SavtaDepth](https://dagshub.com/OperationSavta/SavtaDepth) on Google Colab, while maintaining a relatively clean environment and using version control to promote reproducibility. It is a WIP, but following (and modifying) the cells below should let you git clone a project into colab, download the data, run it, and push it to https://DAGsHub.com which is a free platform for open source data science. \n",
        "\n",
        "SavtaDepth is an Open Source Data Science project. We'd love to get help from the community, so if you'd like to contribute head over to the [project page](https://dagshub.com/OperationSavta/SavtaDepth) to get started."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJNCkwSIa7To"
      },
      "source": [
        "# General Setup\n",
        "Run this before anything else"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Tq2Dl33XDqw"
      },
      "source": [
        "%reload_ext autoreload\n",
        "%autoreload 2"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UiFeCt5xXEK1"
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xn8URgDmWBKm"
      },
      "source": [
        "Clone Git Repo from DAGsHub"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jxOO9c2hU_zM"
      },
      "source": [
        "!git clone https://dagshub.com/OperationSavta/SavtaDepth.git\n",
        "%cd SavtaDepth/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4bndN7kVagXj"
      },
      "source": [
        "*Note: Currently you can't see the intermediate output so it's completely opaque and long process + you can't set up credentials for google within the shell instance*\n",
        "\n",
        "This installs conda, creates a virtual environment and installs all relevant requirements."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAsG5scFYjqd"
      },
      "source": [
        "%%bash\n",
        "MINICONDA_INSTALLER_SCRIPT=Miniconda3-py37_4.8.3-Linux-x86_64.sh\n",
        "MINICONDA_PREFIX=/usr/local\n",
        "wget https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT\n",
        "chmod +x $MINICONDA_INSTALLER_SCRIPT\n",
        "./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX\n",
        "rm $MINICONDA_INSTALLER_SCRIPT\n",
        "make env"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TR0asdFLaeAV"
      },
      "source": [
        "Since locally we might not be working with a GPU, here we install the pytorch version that should use the GPU provided by Google Colab. After that we install the rest of the requirements from the requirements.txt file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n3U6qqVFWYRm"
      },
      "source": [
        "!bash -c \"source activate savta_depth && conda install -y pytorch torchvision cudatoolkit=10.1 -c pytorch && make load_requirements\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WOzZu9GQaaeI"
      },
      "source": [
        "### Pull DVC files from our remote"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sNmiqub0WbqT"
      },
      "source": [
        "!bash -c \"source activate savta_depth && dvc pull -r dvc-remote\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qNNIfodJc_0g"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "# Setup is done!\n",
        "If you've made it here, everything is set up. You have the code and data in the file viewer to the left. You can edit the files located in SavtaDepth/src/code/ as you like. You can see the YAML defining the project pipeline in dvc.yaml. If you change dependencies or outputs (for example, add an addition code file for the training stage), make sure you edit the pipeline to reflect this.\n",
        "\n",
        "### Once you are done with you changes run the cell below to run the pipeline end-to-end\n",
        "* You can run this multiple times if you've made a change and want to test it\n",
        "* If you want to run only a specific stage you can change the `dvc repro` command to any other command you like.\n",
        "\n",
        "▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fwOfOwnrZWMe"
      },
      "source": [
        "!bash -c \"source activate savta_depth && dvc repro\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PAxz-29WhN12"
      },
      "source": [
        "---\n",
        "# Commiting Your Work and Pushing Back to DAGsHub\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kyse-hAuZY9X"
      },
      "source": [
        "!git status"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ib12i6aOhbgI"
      },
      "source": [
        "# Add the files you want to commit\n",
        "!git add {your files here}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "duM9An0Khr_F"
      },
      "source": [
        "Run the following 2 cells without modifications. They will prompt you for a commit message, and for credentials to push back to DAGsHub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V9SYYA9Zh3f9"
      },
      "source": [
        "**Commiting**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_fjym_38hhgl"
      },
      "source": [
        "os.environ['COMMIT_MESSAGE'] = input('Enter the commit message for you commit: ')\n",
        "!git commit -m \"${COMMIT_MESSAGE}\"\n",
        "os.environ['COMMIT_MESSAGE'] = \"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mWtgXU0ph6Dv"
      },
      "source": [
        "**Pushing to DAGsHub**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z7LoVk9Zh9GC"
      },
      "source": [
        "# If this stage fails, make sure to remove outputs as it will show your password to whoever it is shared with.\n",
        "from getpass import getpass\n",
        "import os\n",
        "\n",
        "os.environ['USER'] = input('Enter the username of your DAGsHub account: ')\n",
        "os.environ['PASSWORD'] = getpass('Enter the password of your DAGsHub account: ')\n",
        "os.environ['REPO_URL'] = input('Enter the url of your DAGsHub project: ').split('https://')[-1]\n",
        "os.environ['DAGSHUB_AUTH'] = os.environ['USER'] + ':' + os.environ['PASSWORD']\n",
        "\n",
        "!git push https://$DAGSHUB_AUTH@$REPO_URL.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MXlzJbuLiFWb"
      },
      "source": [
        "***NOTE: ALWAYS RUN THIS CELL AFTER THE PREVIOUS***, it will delete your DAGsHub password in case you share this notebook with someone"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6-yZzyeAiGPI"
      },
      "source": [
        "os.environ['PASSWORD'] = os.environ['DAGSHUB_AUTH'] = \"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FImsqcvYie77"
      },
      "source": [
        "## Push data back to your DVC Remote\n",
        "For this step you must first create a DVC remote on some cloud provider. We recommend Google Cloud Storage. If you're not sure how to set up a DVC remote, [follow these instructions](https://dagshub.com/docs/getting-started/set-up-remote-storage-for-data-and-models/#create-a-storage-bucket) (you only need to go through creating a storage bucket and adding permissions)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4XWSvPQjfLb"
      },
      "source": [
        "Add your storage bucket to DVC (replace {bucket-name} with the bucket name you chose)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kGx03PpijiAW"
      },
      "source": [
        "!bash -c \"source activate savta_depth && dvc remote add my-dvc-remote gs://{bucket-name}\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i8uYEa64imDl"
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZsM5epDbiotM"
      },
      "source": [
        "!bash -c \"source activate savta_depth && dvc push -r my-dvc-remote\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ne7pU7bMj18m"
      },
      "source": [
        "# That's it, you can now create a PR on DAGsHub."
      ]
    }
  ]
}