7. The ML Experiment Workflow

Downloading Data

In this version of the project, we are using the titanic dataset from the Kaggle competition. After getting and setting up the auth key on your system (see how to configure kaggle keys), you can use the command:

make data

This will call the src/data/make_dataset.py the script, that downloads the data inside the folders data/raw on the root of the project.

The process diagram:

├── src                                   ├── data
|   ├── data                              |   ├── raw  
|   |   └── make_dataset.py       ->      |   |   ├── train.csv
|   |                                     |   |   └── test.csv 
                                         

Now we have data in the project, but you may think "Ok, what's the next step?"

Analyze the Data!

It's time for some Data Analysis, and the notebooks folder is there for that purpose! Create a new notebook, import the raw data, then explore, understand, plot beautiful charts and get some insights!

    ├── notebooks
    |   └── 0-eda.ipynb 
    | 

-Ok! I can also create a notebook to clean and transform data right?

NO, you don't, you are an MLOps DS now!

All the code that you need for cleaning, process, and feature creation will be inside the src directory. Use that knowledge you get from DA to start building the data transforming pipeline!

Prepare data: cleaning and data imputation!

Put the code to clean, validate, and fill missing in the src/data/prepare_data.py. When you call the

 make prepare

the data from the data/raw will be used as input, and the transformations inside the script will be applied. The output data will the stored on data/prepared.

The process diagram:

    ├── src                                   ├── data
    |   ├── prepare                   <-      |   ├── raw 
    |   |   └── prepare_data.py               |   |   └── (...)
    |   |                                     |   ├── prepared
    |   |                             ->      |   |   ├── train.csv
    |   |                                     |   |   └── test.csv 

Features: it's time to normalize, standardize and build new features!

During your analysis you'd found that normalizing data will be beneficial to your model, and created some new features that bring a lot of improvements for the model accuracy, then, it's time to code.

The src/data/build_features.py is called when you use the make featurize, it's will get the prepared data as input and store the result inside the data/processed

    ├── src                                   ├── data
    |   ├── features                  <-      |   ├── prepared
    |   |   └── build_features.py             |   |   └── (...)
    |   |                                     |   ├── processed
    |   |                             ->      |   |   ├── train.csv
    |   |                                     |   |   └── test.csv 

You can use the make targets (data, prepare, featurize) to perform the data transformation, and also use the DVC repro command to execute all these steps that are already defined inside the dvc.yml.