CLI-driven Workflow

In this document, we will cover the common tasks involved in building a machine learning model and guide how you can use our CLI commands to accomplish these tasks:
Build a baseline machine learning model with Experiments.
Optimize hyperparameters using Sweep.
Update and store models on Model registry.
Here, we will use the MNIST database to create an image classification model. All our CLI commands are one-liners but you can also select from command option prompts.
Requirements
To follow this guide, you should first have the following setup.
​Organization — a dedicated organization for you or your team
​Project — a space for your machine learning model and mounted datasets
​VESSL Client — Python SDK and CLI to manage ML workflows and resources on VESSL
If you have not created an Organization or a Project, first follow the instructions on the end-to-end guides.
1. Experiment — Build a baseline model
1-1. Configure your default organization and project
Let's start by configuring the client with the default organization and project we have created earlier. This is done by executing vessl configure.
one-liner
prompter
1
vessl configure  \
2
    --organization "YOUR_ORGANIZATION_NAME"  \
3
    --project "YOUR_PROJECT_NAME"
Copied!
1
vessl configure
2
​
3
Please grant CLI access from the URL below.
4
https://vessl.ai/cli/grant-access?token=o0dyb21eu8fw
5
​
6
Waiting...
7
​
8
[?] Default organization: YOUR_ORGANIZATION_NAME
9
 > YOUR_ORGANIZATION_NAME
10
​
11
[?] Default project: YOUR_PROJECT_NAME
12
 > YOUR_PROJECT_NAME
Copied!
You also can check and re-configure your default organization and project by specifying options after vessl configure.
1
vessl configure list # check your current default organization and project
2
vessl configure --renew-token # renew access token and default account
3
vessl configure organization # change your default organization
4
vessl configure project # change your default project
Copied!
1-2. Create and mount a dataset
To create a dataset on VESSL, run vessl dataset create. Let's create a dataset from the public AWS S3 dataset we have prepared: s3://savvihub-public-apne2/mnist. You can check that your dataset was created successfully by clicking the output link.
one-liner
prompter
1
vessl dataset create "vessl-mnist" \
2
  --is-public --external-path "s3://savvihub-public-apne2/mnist"
Copied!
1
vessl dataset create --is-public -e "s3://savvihub-public-apne2/mnist"
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Dataset name: YOUR_DATASET_NAME
Copied!
1-3. Create a machine learning experiment
To create an experiment, use vessl experiment create. Let's run an experiment using VESSL's managed clusters. First, specify the cluster and resources options. Then, specify the image URL — in this case, we are pulling a Docker image from VESSL's Amazon ECR Public Gallery. Next, we are going to mount the dataset we have created previously. Finally, let's specify the start command that will be executed in the experiment container. Here, we will use the MNIST Keras example from our GitHub repository.
one-liner
prompter
1
vessl experiment create \
2
  --cluster "aws-apne2-prod1" \
3
  --resource "v1.cpu-4.mem-13" \
4
  --image-url "public.ecr.aws/vessl/kernels:py36.full-cpu" \
5
  --dataset "/input:vessl-mnist"\
6
  --command "git clone https://github.com/vessl-ai/examples.git && pip install -r examples/mnist/keras/requirements.txt && python examples/mnist/keras/main.py --save-model --save-image"
Copied!
1
vessl experiment create --dataset /input:YOUR_DATASET_NAME
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Project: YOUR_PROJECT_NAME
5
[?] Cluster: aws-apne2-prod1
6
 > aws-apne2-prod1
7
​
8
[?] Resource: v1.cpu-4.mem-13
9
   v1.cpu-0.mem-1
10
   v1.cpu-2.mem-6
11
   v1.cpu-2.mem-6.spot
12
 > v1.cpu-4.mem-13
13
   v1.cpu-4.mem-13.spot
14
   v1.t4-1.mem-13
15
   v1.t4-1.mem-13.spot
16
   v1.t4-1.mem-54
17
   v1.t4-1.mem-54.spot
18
   v1.t4-4.mem-163
19
   v1.t4-4.mem-163.spot
20
   v1.k80-1.mem-52
21
   v1.k80-1.mem-52.spot
22
​
23
[?] Image URL: public.ecr.aws/vessl/kernels:py36.full-cpu
24
 > public.ecr.aws/vessl/kernels:py36.full-cpu
25
   public.ecr.aws/vessl/kernels:py37.full-cpu
26
   public.ecr.aws/vessl/kernels:py36.full-cpu.jupyter
27
   public.ecr.aws/vessl/kernels:py37.full-cpu.jupyter
28
   tensorflow/tensorflow:1.14.0-py3
29
   tensorflow/tensorflow:1.15.5-py3
30
   tensorflow/tensorflow:2.0.4-py3
31
   tensorflow/tensorflow:2.2.1-py3
32
   tensorflow/tensorflow:2.3.2
33
   tensorflow/tensorflow:2.4.1
34
   tensorflow/tensorflow:2.3.0
35
​
36
Start command: git clone https://github.com/vessl-ai/examples.git && pip install -r examples/mnist/keras/requirements.txt && python examples/mnist/keras/main.py --save-model --save-image
Copied!
Note that you can also integrate a GitHub repository with your project so you don't have to git clone every time you create an experiment. For more information about these features, please refer to our doc's project repository & project dataset page.
1-4. View experiment results
The experiment may take a few minutes to complete. You can get the details of the experiment, including its status, using vessl experiment read or by clicking the output link.
one-liner
prompter
1
vessl experiment read "YOUR_EXPERIMENT_NAME"
Copied!
1
vessl experiment read 
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Project: YOUR_PROJECT_NAME
5
[?] Experiment: 1-quasar-bat #1
6
 > 1-quasar-bat #1
Copied!
1-5. Create a model
In VESSL, you can create a model from a completed experiment. First, let's start by creating a model repository using vessl model-repository create and specifying the repository name.
one-liner
prompter
1
vessl model-repository create "tutorial-mnist"
Copied!
1
vessl model-repository create
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Model repository name: YOUR_MODEL_REPOSITORY_NAME
Copied!
Then, let's get a list of experiments in the project and their experiment ID.
1
vessl experiment list
Copied!
Finally, run vessl model create with options including the destination repository and experiment ID. Make sure that the option value for --experiment-id is an integer, not a string.
one-liner
prompter
1
vessl model create "tutorial-mnist" \
2
  --model-name "v0.0.1" \
3
  --source "experiment" \
4
  --experiment-id YOUR_EXPERIMENT_ID
Copied!
1
vessl model create --model-name "v0.0.1"
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Project: YOUR_PROJECT_NAME
5
[?] Model repository: YOUR_MODEL_REPOSITORY_NAME
6
 > YOUR_MODEL_REPOSITORY_NAME
7
​
8
[?] Source: From an experiment
9
 > From an experiment
10
   From local files
11
​
12
[?] Experiment: 1-quasar-bat #1
13
 > 1-quasar-bat #1
14
​
15
[?] Paths (Press -> to select and <- to unselect):
16
   X my_model 0 B
17
   X my_model/keras_metadata.pb 7.7 KB
18
   X my_model/saved_model.pb 88.8 KB
19
   X my_model/variables 0 B
20
   X my_model/variables/variables.data-00000-of-00001 1.2 MB
21
 > X my_model/variables/variables.index 1.4 KB
Copied!
You can see that the model has been created successfully by specifying the repository name and selecting the model number.
1
vessl model read "tutorial-mnist"
Copied!
You can get a list of model repositories and models you have created inside the project by using the following commands.
1
vessl model-repository list # get a list of model repositories
2
vessl model list # get a list of models 
Copied!
2. Sweep — Optimize hyperparameters
So far, we ran a single machine learning experiment and saved it as a model inside a model repository. In this section, we will use a sweep to find the optimal hyperparameter value. First copy and paste the following command and while the sweep is running we will explain each options.
one-liner
prompter
1
vessl sweep create \
2
  --objective-type "maximize" \
3
  --objective-goal "0.99" \
4
  --objective-metric "val_accuracy" \
5
  --num-experiments 4 --num-parallel 2 --num-failed 2 \
6
  --parameter "optimizer categorical list adam sgd adadelta" \
7
  --parameter "batch_size int space 64 256 8" \
8
  --algorithm random \
9
  --cluster "aws-apne2-prod1" \
10
  --resource "v1.cpu-4.mem-13" \
11
  --image-url "public.ecr.aws/vessl/kernels:py36.full-cpu" \
12
  --dataset "/input:vessl-mnist" \
13
  --command "git clone https://github.com/vessl-ai/examples.git && pip install -r examples/mnist/keras/requirements.txt && python examples/mnist/keras/main.py --save-model --save-image"
Copied!
1
vessl sweep create --dataset "/input:YOUR_DATASET_NAME"
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Project: YOUR_PROJECT_NAME
5
[?] Objective type: maximize
6
 > maximize
7
   minimize
8
​
9
Objective metric: val_accuracy
10
Objective goal: 0.99
11
Maximum number of experiments: 4
12
Number of experiments to be run in parallel: 2
13
Maximum number of experiments to allow to fail: 2
14
[?] Sweep algorithm: random
15
   grid
16
 > random
17
   bayesian
18
​
19
Parameter #1 name: optimizer
20
[?] Parameter #1 type: categorical
21
 > categorical
22
   int
23
   double
24
​
25
[?] Parameter #1 range type: list
26
   space
27
 > list
28
​
29
Parameter #1 values (space separated): adam sgd adadelta
30
Add another parameter (y/n): y
31
Parameter #2 name: batch_size
32
[?] Parameter #2 type: int
33
   categorical
34
 > int
35
   double
36
​
37
[?] Parameter #2 range type: space
38
 > space
39
   list
40
​
41
Parameter #2 values ([min] [max] [step]): 64 256 8
42
Add another parameter (y/n): n
43
[?] Cluster: aws-apne2-prod1
44
 > aws-apne2-prod1
45
​
46
[?] Resource: v1.cpu-4.mem-13
47
   v1.cpu-0.mem-1
48
   v1.cpu-2.mem-6
49
   v1.cpu-2.mem-6.spot
50
 > v1.cpu-4.mem-13
51
   v1.cpu-4.mem-13.spot
52
   v1.t4-1.mem-13
53
   v1.t4-1.mem-13.spot
54
   v1.t4-1.mem-54
55
   v1.t4-1.mem-54.spot
56
   v1.t4-4.mem-163
57
   v1.t4-4.mem-163.spot
58
   v1.k80-1.mem-52
59
   v1.k80-1.mem-52.spot
60
​
61
[?] Image URL: public.ecr.aws/vessl/kernels:py36.full-cpu
62
 > public.ecr.aws/vessl/kernels:py36.full-cpu
63
   public.ecr.aws/vessl/kernels:py37.full-cpu
64
   public.ecr.aws/vessl/kernels:py36.full-cpu.jupyter
65
   public.ecr.aws/vessl/kernels:py37.full-cpu.jupyter
66
   tensorflow/tensorflow:1.14.0-py3
67
   tensorflow/tensorflow:1.15.5-py3
68
   tensorflow/tensorflow:2.0.4-py3
69
   tensorflow/tensorflow:2.2.1-py3
70
   tensorflow/tensorflow:2.3.2
71
   tensorflow/tensorflow:2.4.1
72
   tensorflow/tensorflow:2.3.0
73
​
74
Start command: git clone https://github.com/vessl-ai/examples.git && pip install -r examples/mnist/keras/requirements.txt && python examples/mnist/keras/main.py --save-model --save-image
Copied!
The first part of the command defines the key objective and number of experiments.
--objective-type — target object (either to minimize or maximize the metric)
--objective-goal — target metric name as defined and logged using vessl.log()
--objective-metric — target metric value
--num-experiments — total number of experiments
--num-parallel — the number of experiments to run in parallel
--num-failed — the number of failed experiments before the sweep terminates
Next, we specified the details of the parameters and which algorithm to use. In this example, the optimizer is a categorical type and the option values are listed as an array. The batch_size is an int value and the search space is set using max, min, and step.
The command is then followed by cluster, resource, image, dataset, and command options, similar to the vessl experiment create explained above.
You may find it easier to run vessl sweep create and specify the options through command prompts. For more information on sweep, refer to our sweep documentation.​
3. Model Registry — Update and store the best model
Now that we ran multiple experiments using a sweep, let's find the optimal experiment. vessl sweep best-experiment returns the experiment information with the best specified metric value. In this example, the command will return the details of the experiment with the maximum val_accuracy.
one-liner
prompter
1
vessl sweep best-experiment "grove-scowl"
Copied!
1
vessl sweep best-experiment
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Project: YOUR_PROJECT_NAME
5
[?] Sweep: YOUR_SWEEP_NAME
6
 > YOUR_SWEEP_NAME
Copied!
Let's create a v0.0.2 model with vessl model create from the output of the best sweep experiment. You can get the experiment ID using the vessl experiment read command.
one-liner
prompter
1
vessl model create "tutorial-mnist" \
2
  --model-name "v0.0.2" \
3
  --source "experiment" \
4
  --experiment-id 8589948240
Copied!
1
vessl model create --model-name "v0.0.2"
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
Project: YOUR_PROJECT_NAME
5
[?] Model repository: YOUR_MODEL_REPOSITORY_NAME
6
 > YOUR_MODEL_REPOSITORY_NAME
7
​
8
[?] Source: From an experiment
9
 > From an experiment
10
   From local files
11
​
12
[?] Experiment: 1-quasar-bat #1
13
 > 1-quasar-bat #1
14
​
15
[?] Paths (Press -> to select and <- to unselect):
16
   X my_model 0 B
17
   X my_model/keras_metadata.pb 7.7 KB
18
   X my_model/saved_model.pb 88.8 KB
19
   X my_model/variables 0 B
20
   X my_model/variables/variables.data-00000-of-00001 1.2 MB
21
 > X my_model/variables/variables.index 1.4 KB
Copied!
Finally, you can view the performance of your model by using vessl model read and specifying the model repository, followed by the model number.
one-liner
prompter
1
vessl model read "tutorial-mnist" "2"
Copied!
1
vessl model read
2
​
3
Organization: YOUR_ORGANIZATION_NAME
4
[?] Model repository: YOUR_MODEL_REPOSITORY_NAME
5
 > YOUR_MODEL_REPOSITORY_NAME
6
​
7
[?] Model: 1
8
 > 1
9
   2
Copied!
We covered the overall workflow of VESSL using the client CLI. We can also repeat the same process using the client SDK or through Web UI. Now, try this guide with your own code and dataset.