Advanced

Installation

If you have a complex Python environment, you can also use this format to install comet_ml:

/path/to/python -m pip install comet_ml

Other useful pip variations:

  1. python3 -m pip install comet_ml --user : using Python 3, installs comet_ml to ~/.local
  2. pip install comet_ml --upgrade : using the default Python, upgrades comet_ml to latest version
  3. pip3 install comet_ml --upgrade --upgrade-strategy eager : using Python 3, upgrades comet_ml and all of its dependencies

The flags --user, --upgrade, and --upgrade-strategy can be used in many combinations.

Info

If you're installing the SDK on shared servers or as part of a CI/CD system it's encouraged to use the --upgrade flag for pip as our SDK is constantly evolving.

Python Configuration

In order to interact with Comet.ml you will need to let it know what your Comet API key is. Recall that you can get (and reset) your Comet API key at the top right of the screen of any project view.

There are a number of methods for you to set your API key. They are in order of highest precedence:

  1. An argument passed to Experiment()
  2. An environment variable named COMET_API_KEY
  3. A setting in the .comet.config file in your current folder
  4. A setting in the .comet.config file in your HOME directory

Info

Python 2.7 reverses the precedence of 3 and 4.

That is, an explicit API key passed to Experiment() has top priority, and would override any other method. To get started immediately, you can use the following:

import comet_ml
experiment = comet_ml.Experiment("YOUR-API-KEY")
Alternatively, an environment variable named COMET_API_KEY has the second-highest priority, and would override any configuration file setting. In a Unix-like environment, you could set it like this:

export COMET_API_KEY=”YOUR-API-KEY”

However, in general, we recommend using a file named .comet.config (note the initial dot in the filename) in your home directory. That way, it will always be found, and you'll never have to set it again. The file format of the .comet.config file is as follows:

[comet]
api_key=YOUR-API-KEY

After you create the file, you can then create an Experiment() without having to put your private Comet API key in your source code.

If you have a Comet REST API key, you can also add it to your .comet.config file, like so:

[comet]
api_key=YOUR-API-KEY
rest_api_key=YOUR-REST-API-KEY

You can get (or reset) your REST API key under your personal settings at https://www.comet.ml/YOUR-ID/settings/account.

For more information, please see REST API, Getting Started.

Comet Configuration Variables

Here is a list of all of the available configuration variables:

Configuration item Comet Config Section/Name Environment variable Description
API key [comet]
api_key
COMET_API_KEY The API key used for creating an Experiment
REST API key [comet]
rest_api_key
COMET_REST_API_KEY The REST API key
Current Workspace [comet]
workspace
COMET_WORKSPACE Use this workspace when creating new experiments
Current Project [comet]
project_name
COMET_PROJECT_NAME Use this project when creating new experiments
Logging file [comet_logging]
file
COMET_LOGGING_FILE Use the given file for storing log messages. This can also contain patterns such as "comet-{project}.log" (See below)
Logging file level [comet_logging]
file_level
COMET_LOGGING_FILE_LEVEL By default, the log outputs will contains INFO and higher level (WARNING and ERROR) log messages. This configuration item can be used to change the level of the logged messages.
Logging file overwrite [comet_logging]
file_overwrite
COMET_LOGGING_FILE_OVERWRITE Overwrites the log file on each run, if True
Console logging [comet_logging]
console
COMET_CONSOLE Set the logging level for console messages (e.g., INFO, DEBUG)
Logging auto values [comet_logging]
metrics_ignore
COMET_LOGGING_METRICS_IGNORE List of framework:metric names to not auto-log
[comet_logging]
parameters_ignore
COMET_LOGGING_PARAMETERS_IGNORE List of framework:parameter names to not auto-log
[comet_logging]
others_ignore
COMET_LOGGING_OTHERS_IGNORE List of framework:other names to not auto-log
Offline Directory [comet]
offline_directory
COMET_OFFLINE_DIRECTORY Set the offline directory for OfflineExperiment()
Display Summary [comet]
display_summary
COMET_DISPLAY_SUMMARY If False, do not display upload summary for experiments

Remember: any configuration setting defined as an environment variable will override the value set in the .comet.config file.

Comet File Logging

As mentioned, you can use a number of patterns in the COMET_LOGGING_FILE name. These are keywords surrounded by curly braces, as follows:

COMET_LOGGING_FILE Pattern Example Expanded Form Meaning
{user} comet-{user}.log comet-jones.log Uses the OS username
{project} comet-{project}.log comet-general.log Uses the COMET_PROJECT_NAME if set, else "general"
{pid} {project}-{pid}.log general-2635.log Uses the OS Process ID
{datetime} comet-{datetime}.log comet-20190814-151135.log Uses a Year, Month, Day, Hour, Minute, Seconds time format

There is an additional flag that can only be set in the environment: COMET_DISABLE_AUTO_LOGGING. Use this to disable Comet’s automatic tracking functionality. For a complete list of items logged per Python framework, see Automatic Logging. Comet.ml will not interfere with your run, unless you attempt to use one of the auto logging Python libraries, but you have imported comet_ml after the library. For example:

import tensorflow
import comet_ml

experiment = comet_ml.Experiment()
When you create an Experiment then you will get an ImportError. To fix, simply move the import comet_ml above the other import statement.

In some rare cases, something will fail (such as the connection between your computer and comet.ml). In that case, your code will keep on running even though the run will not be logged with Comet.ml. If you wish for your code to stop if it is not being logged, we suggest the following pattern:

from comet_ml import Experiment

exp = Experiment()
if exp.alive is False:
    raise Exception("Something went wrong")

That is, after an Experiment has been created, you may check the alive property. If alive is False then you know something has failed, and you could stop your run.

Comet Auto Logging

Comet can automatically log certain values when using one of the following machine learning frameworks:

Framework What is automatically logged
tensorflow steps and metrics
sklearn hyperparameters
keras graph description, steps, metrics, hyperparameters, optimizer config, and number of trainable parameters
mlflow hyperparameters, assets, models, plus lower-level framework items (e.g., tensorflow's metrics)
pytorch graph description, steps, and loss
fastai all pytorch items, plus epochs, and metrics

However, there may be items that are logged that you wish to ignore, or items ignored that you wish to log. You can control exactly what is auto-logged using the appropriate logging config variables. Each of the ignore variables should be set to a list of framework:name items, separated by commas. Here is an example showing the default values:

[comet_logging]
metrics_ignore = keras:batch_size,keras:batch_batch
others_ignore = 
parameters_ignore = keras:verbose,keras:do_validation,keras:validation_steps

The metrics_ignore line indicates that the metric keras:batch_size will not be auto-logged to Comet. The reason that it is not logged as a metric is because it does not typically change over time, and is usually regarded as a parameter. However, if it did change over time, you can remove it from this list and it then would be logged to Comet.

When you run a script using Comet, it will print out any auto-logged framework:name that is ignored. You can then adjust these configuration variables so that you have complete control over what is logged.

HTTP Proxy

You can instruct comet_ml to use an HTTP proxy by setting one of the common HTTPS variables in an appropriate manner for your operating system. Comet will look for environment variables named HTTPS_PROXY or https_proxy, for all operating systems first, and if it cannot find them, it looks for proxy information from Mac OSX System Configuration for Mac OS X and Windows Systems Registry for Windows. If both lowercase and uppercase environment variables exist (and disagree), the lowercase is preferred.

You can use any standard format for setting the proxy:

  • "hostname.com:9999"
  • "http://hostname.com:9999"
  • "http://username:password@hostname.com:1080"
  • "socks4://hostname.com:1080"
  • "socks5://hostname.com:1080"

For example, in bash you can:

export HTTPS_PROXY="http://username:password@hostname.com:22"
python train.py

Info

Note that you must use the HTTPS version (not HTTP). Also, to use the socks4 or socks5 protocol, you must have the "PySocks" Python library installed.

Context Managers

In addition to the default method of logging shown in Getting Started, you can also use the Python contextual managers with experiment.train(), with experiment.validate(), or with experiment.test() which will allow metrics and hyperparameters will be prefixed accordingly. For example, "accuracy" will be reported as "train_accuracy" in the with experiment.train() context manager. Here is an example using all three context managers:

import comet_ml
experiment = comet_ml.Experiment()

model = build_model()

with experiment.train():
    train_model(model)

with experiment.test():
    test_model(model)

with experiment.validate():
    validate_model(model)

The Experiment object logs various parameters of your experiment to Comet.ml. For example:

from comet_ml import Experiment
experiment = Experiment()
batch_size = 4 # A hyperparameter used somewhere in the code.

experiment.log_parameter("batch_size", batch_size)

By default your experiment will be added to the project for Uncategorized Experiments named general. You can also log your experiment to a specific project using the project_name argument of the Experiment class:

from comet_ml import Experiment

#if "my project name" does not already exist, it will be created.
experiment = Experiment(project_name="my project name")
batch_size = 4

experiment.log_parameter("batch_size", batch_size)
You can also log a custom list of hyperparameters to your experiment via a dictionary.

from comet_ml import Experiment
experiment = Experiment(project_name="my project name",
                        auto_param_logging=False)
batch_size = 128
num_classes = 10
epochs = 20

params = {
    "batch_size":batch_size,
    "epochs":epochs,
    "num_classes":num_classes
    }

experiment.log_parameters(params)

Reporting from SDK to a workspace project

When initializing a team experiment provide the workspace argument. This will assign the experiment to the correct workspace.

from comet_ml import Experiment
experiment = Experiment(workspace="my-workspace-name",
    project_name="a project")

# Your code.

Info

Every workspace member can use their own API key. If workspace is set it will be reported to that workspace. Remember, you can set your API Key in your config file. See Quick Start for more details.

Troubleshooting

Trouble shooting

The first step in trying to track down a problem should be to generate a full log of the experiment. To do that, you can set two variables: COMET_LOGGING_FILE and COMET_LOGGING_FILE_LEVEL as described above. For example, you can define the variables in the environment, or in your .comet.config file.

Here is how you can set them in the bash environment:

$ export COMET_LOGGING_FILE=/tmp/comet.log
$ export COMET_LOGGING_FILE_LEVEL=debug

Here is the contents of a sample .comet.config file:

[comet_logging]
file = /tmp/comet.log
file_level = debug

Or, you can also define them at the same time as you run your script:

$ COMET_LOGGING_FILE_LEVEL=debug \
    COMET_LOGGING_FILE=/tmp/comet.log \
    COMET_API_KEY=MY_API_KEY \
    python script.py

In these examples, the debugging logs will be in /tmp/comet.log, but you can put it where you like, and name it whatever you prefer. This log will show details on all of the steps of your experiment, and any details about failures. If you still have problems, please share this file with us via the Comet.ml Slack channel.

Another item to check is to make sure that your comet_ml version is up to date. You can find the latest version number on the Python Packaging comet_ml page. To upgrade, you can use the command:

$ pip install comet_ml --upgrade

In some cases, you may want to also update all of the packages that comet_ml depends on. You can do that with:

$ pip install comet_ml --upgrade --upgrade-strategy eager