Context

Deep Draw is a project from Le wagon data science school in Paris, batch #1002 (Sept.-Dec. 2022). The objective is to develop, train and apply neural networks models on the QuickDraw dataset published by Google Creative Lab. 100 categories of sketches have been selected and were used to train a CNN-based model and a RNN-based model in order to categorize drawings.

Acknowledgments

👉 Thanks to our supervizor Laure de Grave and our Lead Teacher Vincent Moreau for their help and investment on this project.

👉 Thanks to Google Creative Lab for the quickdraw-dataset from googlecreativelab repository

Summary

1. Initialize our Repository Github for deepdraw
2. Downloading, loading and prepare the Quick Draw dataset for CNN Model
3. Initialize and Run the CNN model
4. Create an API and fast API with streamlit 👉 it Will be our user interface
5. Store the work with Mlflow
6. Create a Docker container and push it in production with GCP
7. Going further 👉 do the same with a sequential data and an RNN model

1️⃣ Project Setup 🛠

deepdraw directory

We create our working environment diagrammed by this tree directory

.
├── Dockerfile                      # Contain our docker
├── Makefile                        # Task manager
├── README.md
├── accueil_deep_draw.png
├── build
│   └── lib
│       └── deep_draw
│           └── fast_api.py
├── deep_draw                       # Main project directory
│   ├── __init__.py
│   ├── dl_logic                    # Deep-Learning classification directory
│   │   ├── __init__.py
│   │   ├── categories.yaml         # Listing of our choosen categories
│   │   ├── cnn.py                  # CNN model
│   │   ├── data.py                 # Loading , cleaning, encoding data
│   │   ├── params.py               # Manage main variables
│   │   ├── preprocessor.py         # Preprocessing data
│   │   ├── registry.py             # Manage model
│   │   ├── rnn.py                  # RNN model
│   │   ├── test_categories.yaml
│   │   ├── tfrecords.py            # Encoding data bitmap --> tfrecords obj
│   │   └── utils.py
│   ├── fast_api.py                 # Initialize API
│   └── interface
│       ├── Deep_Draw.py
│       ├── __init__.py
│       ├── accueil_deep_draw.png
│       ├── app.py
│       ├── main.py
│       ├── pages
│       │   ├── Probabilities_📊.py
│       │   └── Submit_🎉.py
│       └── utils.py
├── deep_draw.egg-info
├── notebooks                       # Stockage notebooks
├── packages.txt
├── raw_data                        # Stockage data
│   ├── dataset.py
│   ├── ndjson_simplified
│   └── npy
├── requirements.txt                # all the dependencies we need to run the package
├── requirements_prod.txt
└── setup.py                        # package installer

2️⃣ Preprocess the data 📡

Convolutional Neural Network model

💻 Encoding from bitmap format to tfrecords

For our CNN model, we use the data in .npy type from QuickDraw dataset. This allow us to use bitmap format for our images. One categorie (cats for exemple) contain 100 000 differents draws .

The real challenge is to load and run the model for at least 100 categories, corresponding to 10 000 000 draws !!! 🙊

Thats' why we need to convert the data in an object tensorflow. With it, we can split the data into many packs of 32 draws and make the model easily and faster. Then, we can avoid the expected problemes from RAM memory.

💻 Decoding from tfrecords to bitmap format

Recurrent Neural Network model

💻 Encoding from ndjson format to tfrecords

💻 Decoding from tfrecords to ndjson format

3️⃣ Make and run the models

CNN Model - initialize, compile and train

A conventionnal CNN model is initialized using the initialize_cnn method. Three Conv2D layers followed by three MaxPooling2D layers are used before the Flatten and Dense layers. The output layers uses the softmax activation function to predict 100 probabilities.

The model is compiled using compile_cnn. An Adam optimizer, a sparse categorical crossentropy loss function and the accuracy metrics his monitored.

#Initialize a CNN Model

model = Sequential()

    model.add(Conv2D(16, (3,3), activation='relu', input_shape=(28,28,1)))
    model.add(MaxPooling2D((2,2)))

    model.add(Conv2D(32, (3,3), activation='relu', padding='same'))
    model.add(MaxPooling2D((2,2)))

    model.add(Conv2D(64, (3,3), activation='relu', padding='same'))
    model.add(MaxPooling2D((2,2)))

    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    #model.add(Dropout(0.4))
    model.add(Dense(num_classes, activation = 'softmax'))

#Compile

model.compile(
        optimizer='adam',
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'])

The final accuracy is around 80% which is sufficient for categorizing sketches.

Here is a 3D visualization of the CNN model

CNN Modelisation results

Here is the final confusion matrix and the final classification report.

Activation map

the activation map shows how neurones specialize whithin the first Conv2D layer. 3 examples from 3 categories 🐱 🐷 🐸 are represented bellow.

RNN Model - initialize, compile and train

The RNN model is initialized using the initialize_rnn_tfrecords method.

One Masking layer followed by two LSTM layers are used before the Dense layer. The output layers uses the softmax activation function to predict 100 probabilities.

The RNN model is compiled as the same way than Like the CNN model.

#Initialize a RNN Model

model = Sequential()

    model.add(layers.Masking(mask_value=1000, input_shape=(1920,3)))
    model.add(layers.LSTM(units = 20, activation= 'tanh', return_sequences= True))
    model.add(layers.LSTM(units = 20, activation= 'tanh', return_sequences= False))

    model.add(Dense(50, activation='relu'))
    model.add(Dense(num_classes, activation = 'softmax'))

The final accuracy for the RNN model is around 75% which is sufficient for categorizing sketches.

RNN Modelisation results

Here is the final confusion matrix and the final classification report.

3️⃣ The streamlite interface

4️⃣ Build an API using Dockers and Fast API