How to install DIGITS

DIGITS is only officially supported on Ubuntu 14.04. However, DIGITS has been successfully used on other Linux variants as well as on OSX.

While the official documentation states CUDA and cuDNN as prerequisites, one can easily get things done without the same. Make sure you have Caffe installed from the Nvidia fork of Caffe. For installation instructions, follow this.

Grab the source

$> cd $HOME
$> git clone https://github.com/NVIDIA/DIGITS.git digits

Throughout the docs, we’ll refer to your install location as DIGITS_HOME ($HOME/digits in this case), though you don’t need to actually set that environment variable.

You may be tempted to install the requirements with pip install -r requirements.txt. Use the loop statement below to install the packages in order and avoid errors.

$> for req in $(cat requirements.txt); do pip install $req; done

If you want to install these packages without using a virtual environment, replace “pip install” with “sudo pip install”.

DIGITS uses graphviz to visualize network architectures. You can safely skip this step if you don’t want the feature.

$> sudo apt-get install graphviz

Set Environment Variables

Make sure you set the CAFFE_HOME variable before deploying the DIGITS server.

$> export CAFFE_HOME=<caffe-home>

Where <caffe-home> is the absolute path to the directory where caffe is installed (one which you cloned from GitHub).

Starting the server

You can run DIGITS in two ways:

Development mode

% ./digits-devserver

Starts a development server that listens on port 5000 (but you can change the port if you like – try running it with the –help flag).

Then, you can view your server at http://localhost:5000/.

Production mode

% ./digits-server

Starts a production server (gunicorn) that listens on port 8080 (http://localhost:8080). If you get any errors about an invalid configuration, use the development server first to set your configuration.

If you have installed the nginx.site to your nginx sites-enabled/ directory, then you can view your app at http://localhost/.

Usage

Check out the Getting Started page.

Caffe Installation for Mac OSX 10.10

Install Dependencies

[TIP : Though the official documentation suggests installing Anaconda, it would be better to avoid using the same. The following installation procedure assumes the absence of Anaconda]

Install Homebrew Package Manager

  • Paste the following in a terminal prompt. The script explains what it will do and then pauses before it does it. This package manager would be of great use throughout the installation tasks.
$> ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Get CUDA 7.0

  • Install CUDA 7.0 (for OSX) from here.
  • Install latest standalone CUDA driver from here (apparently, one included in CUDA Toolkit is outdated)

Install dependencies via homebrew

  • We will need to edit the OpenCV installation file a bit.
$> brew edit opencv
  • Then replace the following lines
$> args « "-DPYTHON#{py_ver}_LIBRARY=#{py_lib}/libpython2.7.#{dylib}"
args « "-DPYTHON#{py_ver}_INCLUDE_DIR=#{py_prefix}/include/python2.7"
  • With
$> args « "-DPYTHON_LIBRARY=#{py_prefix}/lib/libpython2.7.dylib"
$> args « "-DPYTHON_INCLUDE_DIR=#{py_prefix}/include/python2.7"
  • Install snappy, leveldb, gflags, glog, szip, lmdb, hdf5 and opencv.
$> brew install —fresh -vd snappy leveldb gflags glog szip lmdb homebrew/science/opencv homebrew/science/hdf5
  • Install protobuf.
$> brew install —build-from-source —with-python —fresh -vd protobuf
  • Install boost libraries for python.
$> brew install —build-from-source —fresh -vd boost boost-python

Note : Go through the logs of the installation of the above dependencies (displayed on your stdout) to make sure everything installed correctly. Some of the dependencies might be skipped in the installation and would lead to problems at a later stage.

Download Caffe

Create a directory where you would like to install caffe. For all future reference, this will be called the . From inside the directory, execute the following commands :

[Normal Caffe users]
$> git clone https://github.com/BVLC/caffe.git
[Caffe users who will use DIGITS in future]
$> git clone https://github.com/NVIDIA/caffe

$> cd caffe
$> cp Makefile.config.example Makefile.config

Edit the Makefile.config :

Keep track of all the path variables set in the the Makefile.config (the smallest of mistakes can lead you off track for days together). Here’s my complete Makefile.config for the hasty ones here… We’ll tackle each step in the upcoming section.

###---- Makefile.config ----###
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
# CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
        -gencode arch=compute_20,code=sm_21 \
        -gencode arch=compute_30,code=sm_30 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_50,code=sm_50 \
        -gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
        /usr/local/lib/python2.7/site-packages/numpy/core/include/
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
        # $(ANACONDA_HOME)/include/python2.7 \
        # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
USE_PKG_CONFIG := 1

BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?=

Now lets look at the file step-by-step.

We leave this part commented out as we wont be using cuDNN in our installation. It speeds up the Caffe processes but for a general (simplistic and working) installation, it can be left out.

# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1

We build caffe without GPU support (following the idea of having a simplistic,working installation).

# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1

Pretty self-explanatory with the comments. This is the path to your local installation of CUDA.

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

You can chose from ATLAS, MKL, or OpenBLAS for your BLAS choices.
OSX has in-built BLAS libs so leave the BLAS:=atlas (default)

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

Use this only if you need MatCaffe interface for Caffe. Following our idea of simplistic installation, we will skip this part.

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

Now this is a tricky part. To install Caffe with the python interface, PyCaffe (Recommended) you need to give the paths to your python include libs and the path where you have numpy stored. Path to numpy include folder must be given with great caution. You might land into unnecessary trouble by specifying this path incorrectly. You might use the ‘locate’ command to find the paths to the respective libraries.

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
        /usr/local/lib/python2.7/site-packages/numpy/core/include/

This was the most important tweak required to get PyCaffe up and running. You might have different versions of python installed on your local machine and in your homebrew instance. This would lead to problems when importing caffe from your python interpreter. To overcome this problem, make sure that you provide the correct path to your brewed python.

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/

The rest of the parts can be left out as they are not used anyways (commented out) and hold not much importance. You are all set to compile Caffe! Almost there!

Compile :

Now hoping that all the above steps went as planned, we can finally compile Caffe. So keeping your fingers crossed, execute the following from your       :

$> make clean
$> make all
$> make py [if you are running nvidia/caffe for digits]
$> make test
$> make runtest

These commands will take a few minutes to execute. I’d suggest you skim through the output of these commands being printed on the stdout and make sure you see no alarming warnings/errors. You might see quite a few warnings of unused variables and parallel threads (-pthread). These are not things you should worry about. 

The output of ‘make runtest’ would be something like this.

runtest-output

PyCaffe

Now to install and configure the python interface to Caffe, first lets make sure we have all the python dependencies installed. The requirements.txt file handles this list.

for req in $(cat python/requirements.txt); do pip install $req; done

Now, to compile the PyCaffe interface, from the do

make pycaffe

IMP : Don’t forget to set the PYTHONPATH variable in your ~/.bash_profile (OSX) or ~/.bashrc (Linux) to the caffe python.

export PYTHONPATH=/python:$PYTHONPATH

Some versions of the official documentation show that the PYTHONPATH variable needs to be set to /python/caffe, however this is an error in the documentation and the above path should be set.

You might also run the following command to create a distribute directory with all the Caffe headers, compiled libraries, binaries, etc. needed for distribution to other machines.

make distribute

To finally import caffe in the python interpreter, go the python folder in and execute

$> cd /python
$> python
Python 2.7.9 (default, Jan 29 2015, 06:27:40)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe

Done!

Now you are all set to rock! Play around with a few examples like Training LeNet on MNIST or see how to use the pre-trained models like ImageNet.

End-Note :

  • Hope this page helps reduce some effort required to install Caffe on your machines. To view our bigger attempts at reducing this overload, checkout the CloudCV organisation page.
  • I have taken references from a lot of sources I found online and one can easily find them through the sources cited in above bullet. This wiki is for educational purposes only.

DIGITS Tutorial : Example run of LeNet Model on MNIST

We would be using the DIGITS framework to train a LeNet Model on the famous Handwritten Numerical-digits Dataset MNIST. We assume that you have installed Caffe and DIGITS already, please follow the respective links to install them if you haven’t done that already. So let’s get started.

Start the Server

Start the digits server by executing the following. $DIGITS_HOME is the path where you installed Digits.

$> cd $DIGITS_HOME
$> ./digits-devserver

The first time DIGITS is run, you may be asked to provide some configuration options.

$> ./digits-devserver
  ___ ___ ___ ___ _____ ___
 |   \_ _/ __|_ _|_   _/ __|
 | |) | | (_ || |  | | \__ \
 |___/___\___|___| |_| |___/

DIGITS requires at least one DL backend to run.
==================================== Caffe =====================================
Where is caffe installed?

        Suggested values:
        (P*) [PATH/PYTHONPATH] <PATHS>
>> ~/caffe
Using "/home/username/caffe"
Saved config to /home/username/digits/digits/digits.cfg
 * Running on http://0.0.0.0:5000/

Most values are set silently by default. If you need more control over your configuration, try one of these commands:

# Set more options before starting the server
$> ./digits-devserver --config
# Advanced usage
$> python -m digits.config.edit --

Using the Webapp

When you fire up the digits server for the first time, you must see something like the below image on the url : 0.0.0.0:5000/ or http://localhost:5000

home-page-1

Create Database

To extract the MNIST dataset, DIGITS provides automated python scripts. Run the following from your <digits-home> folder.

$> python tools/download_data/main.py mnist ~/data/mnist

Now, in the browser application, in the Datasets section on the left side of the page, click on the blue Images button and select Classification which will take you to the “New Image Classification Dataset” page.

  • Change the image type to Grayscale
  • Change the image size to 28 x 28
  • Type in the path to the MNIST training images
    • /home/username/digits-1.0/mnist_10k if you are using the web installer
  • Give the dataset a name
  • Click on the Create button

While the model creation job is running, you should see the expected completion time on the right side of the new page that pops up. When the data set has completed training, go back to the home page, by clicking DIGITS in the top left hand part of the page. You should now see that there is a trained data set.

home-page-2

Training a Model

In the Models section on the left side of the page, click on the blue Images button and select Classification which will take you to the “New Image Classification Model” page. For this example, do the following:

  • Choose the MNIST dataset in the “Select Dataset” field
  • Choose the LeNet network in the “Standard Networks” tab
  • Give the model a name
  • Click on the Create button

While training the model, you should see the expected completion time on the right side. The page will also show you a near real-time ROC plot of the loss versus epoch (time-frame) and another graph will show the state of the learning rate of training.

training-model

Testing the Model

To test the model, scroll to the bottom of the page. On the left side are tools for testing the model.

  • Click on the Upload Image field which will bring up a local file browser and choose a file
    • If you’ve used the web installer, choose one in /home/username/digits-1.0/test_digits
  • Or, find an image on the web and paste the URL into the Image URL field
  • Click on Classify One Image

DIGITS will display the top five classifications and corresponding confidence values.

classified-one-image

Voilà!

We are done training and testing out Model! Stick around for more updates and feel free to comment in case of queries. I would highly recommend the digits-users and caffe-users google groups for general discussion and doubts solutions.