# Introduction

## Optimizer¶

The Comet `Optimizer`

is used to dynamically find the best set of
hyperparameter values that will minimize or maximize a particular
metric. It can make suggestions for what hyperparameter values to try
next, either in serial or in parallel (or a combination).

Comet's Optimizer has many benefits over traditional hyperparameter optimizer search services because of its integration with Comet's Experiments. In addition, Comet's hyperparameter search has a powerful architecture for customizing your search or sweep. You can easily switch search algorithms, or perform phased searches.

In its simplest form, you can use the hyperparameter search this way:

# file: example-1.py from comet_ml import Optimizer # We only need to specify the algorithm and hyperparameters to use: config = { # We pick the Bayes algorithm: "algorithm": "bayes", # Declare your hyperparameters in the Vizier-inspired format: "parameters": { "x": {"type": "integer", "min": 1, "max": 5}, }, # Declare what we will be optimizing, and how: "spec": { "metric": "loss", "objective": "minimize", }, } # Next, create an optimizer, passing in the config: # (You can leave out API_KEY if you already set it) opt = Optimizer(config) # define fit function here! # Finally, get experiments, and train your models: for experiment in opt.get_experiments( api_key="API_KEY", project_name="optimizer-search-01"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss)

That's it! Comet will provide you with an Experiment object already set up with the suggested parameters to try. You merely need to train the model and log the metric to optimize ("loss" in this case).

You can also use the same optimizer in parallel. To use in parallel, you need to take the optimizer config options and save them in a file:

# file: example-2.config { # We pick the Bayes algorithm: "algorithm": "bayes", # Declare your hyperparameters in the Vizier-inspired format: "parameters": { "x": {"type": "integer", "min": 1, "max": 5}, }, # Declare what we will be optimizing, and how: "spec": { "metric": "loss", "objective": "minimize", }, }

Now, we use a slight variation of the code above. We have only moved
the optimizer config options out of the script into their own file,
and pass in the filename via the `sys.argv`

arguments:

# file: example-2.py from comet_ml import Optimizer import sys # Next, create an optimizer, passing in the config: # (You can leave out API_KEY if you already set it) opt = Optimizer(sys.argv[1]) # define fit function here! # Finally, get experiments, and train your models: for experiment in opt.get_experiments( api_key="API_KEY", project_name="optimizer-search-02"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss)

You can call this file directly, passing in the name of the optimizer config file:

$ python example-2.py example-2.config

and it will run exactly as it did above with `example-1.py`

. However,
you can also use `comet optimize`

to run the code in parallel:

```
$ comet optimize -j 2 example-2.py example-2.config
```

where `-j`

indicates the number of processes to run in parallel. This
will run as it did before, but running processes in parallel. For more
details on the `comet optimize`

options, see below.

We have designed the Optimizer API to be intuitive, but powerful. The above information is perhaps most of what you need. However, the rest of this page documents all of the Optimizer features, options, and settings.

See the Optimizer class for more details on creating an optimizer.

## Optimizer Configuration¶

The Optimizer configuration dictionary (either specified in code, or in a config file) has the following five sections:

- "algorithm" - string, which search algorithm to use
- "spec" - dictionary, the algorithm-specific specifications
- "parameters" - dictionary, the parameter distribution space descriptions
- "name" - string, a personalizable name to associate with this search instance (optional)
- "trials" - integer, the number of trials per experiment to run (optional, defaults to 1)

The algorithm must be one of the three possible algorithms: "random", "grid", or "bayes".

Each of these entries are described in the next section.

A complete example of an optimizer config:

{"algorithm": "bayes", "spec": { "maxCombo": 0, "objective": "minimize", "metric": "loss", "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "parameters": { "hidden-layer-size": {"type": "integer", "min": 5, "max": 100}, "hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]}, }, "name": "My Bayesian Search", "trials": 1, }

### Optimizer Algorithms¶

There are three different search/sweep algorithms that you can use with the optimizer:

- "grid" - Sweep algorithm based on picking parameter values from discrete, possibly sampled, regions
- "random" - Random sampling algorithm
- "bayes" - Bayesian algorithm based on distributions, balancing exploitation and exploration

Example:

{"algorithm": "bayes"}

For most searches, we recommend using "bayes". However, for some cases, one of the algorithms may be more appropriate. See below for more details.

#### Bayes Algorithm¶

As mentioned, the Bayes algorithm may be the best choice for most of your Optimizer uses. It provides a well-tested algorithm that balances exploring unknown space, with exploiting the best known so far. The Comet Bayes algorithm implements the adaptive Parzen-Rosenblatt estimator.

The "bayes" search algorithm uses the following options in the "spec":

- "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning no limit)
- "objective" - string "minimize" or "maximize", for the objective metric (default "minimize")
- "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
- "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
- "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default 20)
- "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)

The Comet optimizer will never assign the same set of parameters
twice, unless running multiple trials of an experiment, or reassigning
a non-completed experiment (see below). Depending on the algorithm you
have chosen, the number of possible experiments to run may be finite,
or infinite. For example, the "bayes" algorithm always samples from
continuous parameter distributions, but the "grid" algorithm always
breaks distributions into grids. However, some parameter
types (including "categorical" and "discrete") there are only a finite
number of options. To see the computed value of maximum number of
possible experiments for an algorithm, see `Optimizer.status()`

.

In either the finite or infinite case, to limit the number of
experiment combinations to run set `maxCombo`

to a non-zero
value. Notice that this is the "maximum combinations" not the total
maximum number of experiments to assign. For example, the total
number of experiments assigned is `maxCombo`

* `trials`

. But this can
also be effected by the SPEC setting "retryAssignLimit" which will
re-assign experiments until they are "completed" or
the "retryAssignLimit" value is met. Therefore, if each experiment never
completes, you would assign `maxCombo`

* `trials`

* (`retryAssignLimit`

+ 1)
number of experiments.

Example:

{"algorithm": "bayes", "spec": { "maxCombo": 0, "objective": "minimize", "metric": "loss", "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

#### Grid Algorithm¶

The "grid" algorithm is useful for performing a wide, initial search of a set of parameter values. Comet's grid algorithm is slightly more flexible than many, as each time you run it, you will sample from the set of possible grids defined by the parameter space distribution. The "grid" algorithm does not use past experiments to inform future experiments: it merely collects the objective metric for you to explore.

The "grid" search algorithm uses the following options:

- "randomize" - boolean, if True, then the grid is traversed randomly; otherwise traversed in order (default False)
- "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning no limit)
- "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
- "gridSize" - integer, when creating a grid, the number of bins per parameter (default 10)
- "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
- "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default" 20)
- "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)

Example:

{"algorithm": "grid", "spec": { "randomize": True, "maxCombo": 0, "metric": "loss", "gridSize": 10, "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

#### Random Algorithm¶

The "random" algorithm is slightly more flexible than the "grid" algorithm, in that it will continue to sample from the set of possible parameter values, until you stop the search, or have the "max combinations" value set. The "random" algorithm, like the "grid" algorithm, does not use past experiment metrics to inform future experiments.

The "random" search algorithm uses the following options:

- "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning no limit)
- "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
- "gridSize" - integer, when creating a grid, the number of bins per parameter (default 10)
- "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
- "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default" 20)
- "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)

Example:

{"algorithm": "random", "spec": { "maxCombo": 100, "metric": "loss", "gridSize": 10, "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

### Specifying Optimizer Parameters¶

There are four kinds of parameters: "integer", "double" or "float", "discrete" (for a list of numbers), and "categorical" (for a list of strings).

The format of each parameter was inspired by Google's Vizier, and exemplified by the open source version called Advisor.

#### Integers¶

Integers can be distributed in one of five ways: "linear", "uniform", "normal", "loguniform", or "lognormal".

{"PARAMETER-NAME": {"type": "integer", "scalingType": "linear" | "uniform" | "normal" | "loguniform" | "lognormal", "min": INTEGER, "max": INTEGER, }, ... }

See below for more details on "scalingType" for each algorithm.

NOTE: "integer" `type`

with "linear" `scalingType`

when using the
"bayes" algorithm indicates an independent distribution. This is useful
for using integer values that have no relationship with one another,
such as seed values. If your distribution is meaningful (e.g., 2 is
closer to 1 than it is to 6) then you should use the "uniform"
`scalingType`

.

You can also provide a list of integers (or any numbers) using the "discrete" type:

{"PARAMETER-NAME": {"type": "discrete", "values": [NUMBER, ...], }, ... }

Example:

{"algorithm": "bayes", "spec": {...}, "parameters": { "hidden-layer-size": {"type": "integer", "min": 5, "max": 100}, "hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

#### Double/Float¶

Doubles (also called floats) can be distributed in one of five ways: "linear", "uniform", "normal", "loguniform", or "lognormal".

{"PARAMETER-NAME": {"type": "double" |"float", "scalingType": "linear" | "uniform" | "normal" | "loguniform" | "lognormal", "min": FLOAT, "max": FLOAT, }, ... }

See below for more details on "scalingType" for each algorithm.

You can also provide a list of doubles/floats (or any numbers) using the "discrete" type:

{"PARAMETER-NAME": {"type": "discrete", "values": [NUMBER, ...], }, ... }

Example:

{"algorithm": "bayes", "spec": {...}, "parameters": { "momentum": {"type": "double", "min": 0.0, "max": 0.9}, "learning-rate": {"type": "discrete", "values": [0.01, 0.1, 0.8]}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

#### Scaling Types¶

Both "integer" and "double"/"float" allow the following scaling types (indicated with "scalingType"):

- "linear" - for integers, means an independent distribution (used for things like seed values); for double, the same as uniform
- "uniform" - a uniform distribution between "min" and "max"
- "normal" - a normal distribution centered around "mu" with standard deviation of "sigma"
- "lognormal" - a log-normal distribution centered around "mu" with standard deviation of "sigma"
- "loguniform" - a log-uniform distribution between "min" and "max". Computes
`exp(uniform(log(min), log(max)))`

Examples:

{"algorithm": "bayes", "spec": {...}, "parameters": { "momentum": {"type": "float", "min": 0.0, "max": 0.9, "scalingType": "uniform"}, "learning-rate": {"type": "float", "min": 0.001, "max": 1.0, "scalingType": "loguniform"}, "dropout": {"type": "float", "mu": 0.5, "sigma": 0.1, "scalingType": "normal"}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

#### Categorical¶

Categorical, like "discrete", is a type for a list of values. In this case, the values must be strings.

{"PARAMETER-NAME": {"type": "categorical", "values": ["LIST", "OF", "STRINGS"], }, ... }

Example:

{"algorithm": "bayes", "spec": {...}, "parameters": { "activation": {"type": "categorical", "values": ["sigmoid", "relu", "leaky-relu"]}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

#### Additional Optimizer Parameter Settings¶

For each parameter, you may also specify "gridSize".

Note: `gridSize`

is only used in the `grid`

and `random`

algorithms.

##### Grid Size¶

Each parameter is considered a distribution for those algorithms that sample randomly. Those algorithms include "bayes" and "random". However, other algorithms need to know a resolution size for how to divide up the parameter space into discrete bins. Those algorithms include "grid". For those, an additional entry named "gridSize" can be set for each parameter. For example, the following defines a parameter "x" ranging between 0.0 and 100.0, and broken up into 25 bins for the grid search.

Example:

{"x": {"type": "double", "scalingType": "uniform", "min": 0.0, "max": 100.0, "gridSize": 25, }, }

NOTE: the bins won't be exactly 0-25, 25-50, 50-75, and 75-100. Rather, the divisions are created based on sampled values.

## Comet Optimize¶

`comet`

is a command-line utility that is installed with `comet_ml`

.
`optimize`

is one of the commands that `comet`

can use. The format is:

$ comet optimize [options] [PYTHON_SCRIPT] OPTIMIZER

For more information on `comet optimize`

please see Command-Line Utilities

## End-to-end Example¶

Now, let's see a complete end-to-end program using Keras with the Comet Optimizer.

import comet_ml import logging logging.basicConfig(level=logging.INFO) LOGGER = logging.getLogger("comet_ml") from tensorflow.keras.datasets import mnist from tensorflow.keras.layers import Dense from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import RMSprop from tensorflow.keras.utils import to_categorical def build_model_graph(experiment): model = Sequential() model.add( Dense( experiment.get_parameter("first_layer_units"), activation="sigmoid", input_shape=(784,), ) ) model.add(Dense(128, activation="sigmoid")) model.add(Dense(128, activation="sigmoid")) model.add(Dense(10, activation="softmax")) model.compile( loss="categorical_crossentropy", optimizer=RMSprop(), metrics=["accuracy"], ) return model def train(experiment, model, x_train, y_train, x_test, y_test): model.fit( x_train, y_train, batch_size=experiment.get_parameter("batch_size"), epochs=experiment.get_parameter("epochs"), validation_data=(x_test, y_test), ) def evaluate(experiment, model, x_test, y_test): score = model.evaluate(x_test, y_test, verbose=0) LOGGER.info("Score %s", score) def get_dataset(): num_classes = 10 # the data, shuffled and split between train and test sets (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train = x_train.reshape(60000, 784) x_test = x_test.reshape(10000, 784) x_train = x_train.astype("float32") x_test = x_test.astype("float32") x_train /= 255 x_test /= 255 print(x_train.shape[0], "train samples") print(x_test.shape[0], "test samples") # convert class vectors to binary class matrices y_train = to_categorical(y_train, num_classes) y_test = to_categorical(y_test, num_classes) return x_train, y_train, x_test, y_test # Get the dataset: x_train, y_train, x_test, y_test = get_dataset() # The optimization config: config = { "algorithm": "bayes", "name": "Optimize MNIST Network", "spec": {"maxCombo": 10, "objective": "minimize", "metric": "loss"}, "parameters": { "first_layer_units": { "type": "integer", "mu": 500, "sigma": 50, "scalingType": "normal", }, "batch_size": {"type": "discrete", "values": [64, 128, 256]}, }, "trials": 1, } opt = comet_ml.Optimizer(config) for experiment in opt.get_experiments(project_name="my_project"): # Log parameters, or others: experiment.log_parameter("epochs", 10) # Build the model: model = build_model_graph(experiment) # Train it: train(experiment, model, x_train, y_train, x_test, y_test) # How well did it do? evaluate(experiment, model, x_test, y_test) # Optionally, end the experiment: experiment.end()

## Troubleshooting¶

### Continue from Crashed/Paused Optimizer¶

If you pause your search, or if your optimizer script would ever crash
you can recover your search and pick up immediately from where you
left off. You need only define the `COMET_OPTIMIZER_ID`

in the
environment, and run your script again. The `COMET_OPTIMIZER_ID`

is
printed in the terminal at the start of each sweep. It also is logged
with each experiment in the Other tab. Here is an example or a script
crashing, and continuing with the search:

$ python script.py COMET INFO: COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60 ... it crashes for some reason $ edit script.py $ export COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60 $ python script.py COMET INFO: COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60

You can also supply the optimizer id to the Optimizer class rather
than the name of the filename containing the optimizer config. For
example, consider again `example-2.py`

from above:

# file: example-2.py from comet_ml import Optimizer import sys # Next, create an optimizer, passing in the config: # (You can leave out API_KEY if you already set it) opt = Optimizer(sys.argv[1]) # define fit function here! # Finally, get experiments, and train your models: for experiment in opt.get_experiments( api_key="API_KEY", project_name="optimizer-search-03"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss)

Recall that you can start that program up, like:

$ python example-2.py example-2.config

or using `comet optimize`

:

```
$ comet optimize -j 2 example-2.py example-2.config
```

To use your same script and start up where you left off, you only need the Comet Optimizer id. When you start up a new optimizer, you will see a line displayed similar to:

COMET INFO: COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d

You can set this Comet environment variable in the terminal, and your search will use the existing Optimizer rather than creating a new one.

$ export COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d $ python example-2.py example-2.config

or

$ export COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d $ comet optimize -j 2 example-2.py example-2.config

You can also just pass the Optimizer id on the command line instead of
the filename if you have written your script in the style of
`example-2.py`

:

$ python example-2.py 303faefd8194400694ec9588bda8338d

or

```
$ comet optimize -j 2 example-2.py 303faefd8194400694ec9588bda8338d
```

You can also have `comet optimize`

pass along arguments to your
script. Simply add those after the config following two dashes, like
this:

$ comet optimize -j 4 script.py opt.config -- --project-name "test-007"

Then you can use the argparse module, like so:

# example-3.py from comet_ml import Optimizer, Experiment import argparse parser = argparse.ArgumentParser() ## Add your own args here: parser.add_argument("--project-name", default=None) ## These passed on from "comet optimize": parser.add_argument("optimizer", default="test1_optimizer.json") parser.add_argument("--trials", "-t", type=int, default=None) parsed = parser.parse_args() count = 0 for experiment in opt.get_experiments(): loss = train(experiment.params["x"]) msg = experiment.log_metric("loss", loss) count += 1 print("Optimizer job done! Completed %s experiments." % count)

The above program can then be used alone, or with the `comet optimize`

to run scripts in parallel with custom command-line arguments. Called
normally:

```
$ python example-3.py opt.config --project-name "my-project-01"
```

or in parallel:

```
$ comet optimize example-3.py opt.config -- --project-name "my-project-01"
```

### What if an experiment doesn't finish?¶

By default, all of the algorithms will not release duplicate sets of parameters (except when trials is greater than 1). But what should you do if an experiment crashes and never notifies the Optimizer?

You have two choices:

- You can run the Optimizer search with the "retryAssignLimit" spec settings:

{"algorithm": "bayes", "spec": { "retryAssignLimit": 1, ... }, "parameters": {...}, "name": "My Bayesian Search", "trials": 1, }

Using a `retryAssignLimit`

value greater than zero will continue to
assign the parameter set until an experiment marks it as "completed"
or the number of retries is equal to the `retryAssignLimit`

.

- You can run the Optimizer search/sweep again. You can either run all of the parameter value combinations again, or a subset thereof.