Warnings and Errors

For further assistance on any of these Python warnings or errors, or if you see an error message that is not noted here, please ping us on our slack channel

Error

ImportError: Please import comet before importing these modules: ...

This error occurs when you try to create an Experiment (or another kind of experiment, such as OfflineExperiment) but have imported comet_ml after one of the supported auto-logging machine learning libraries (such as torch, fastai, keras, or tensorflow). You have two choices to resolve this error: you can either move comet_ml to be imported first (recommended), or you can completely disable Comet's auto logging facility by setting COMET_DISABLE_AUTO_LOGGING=1 in the environment, or in your Comet config file. See Quick Start for more details.

Error

COMET ERROR: Run will not be logged

This error is shown with a Python stack trace and indicates that the initial handshake between Comet and the server failed. This is usually a local networking issue or production downtime. Please reach out on our slack channel if you encounter this error.

Error

COMET ERROR: Failed to set run source code

Comet failed to read the source code file for this experiments. This could be due to using a dispatcher.

Rate limits

The Comet API may rate limit submission of requests for your experiments. Such limits are managed as an allowed number of operations per time window, where an operation might be read or an update. In that case, a warning symbol will appear on your experiment list.

Online Experiments

For online experiments, the following rate limits are in effect:

  • logging metrics: 10,000 per minute
  • logging parameters: 8,000 per minute
  • logging output: 10,000 per minute
  • logging everything else: 8,000 per minute

Offline Experiments

For offline experiments, the following rate limits are in effect:

  • logging metrics: 80,000 per minute
  • logging parameters: 80,000 per minite
  • logging output: 80,000 per minute
  • logging everything else: 80,000 per minute

Info

Offline experiments have a rate limit because you may be uploading multiple experiments in parallel, or attempting to upload too many too quickly.

Size and Count limits

For all experiments, you can log 15,000 total values for each metric, per experiment. If your metric count goes beyond this limit, then the values are downsampled.

REST API

When using the REST API (from the Python SDK, or using the URL endpoints directly), the following limits are in effect per experiment.

Info

Each of the following REST API items typically only need to be logged once per experiment.

  • 10 environment detail updates
  • 10 git metadata updates
  • 10 graph (model) updates
  • 10 OS packages updates
  • 10 code updates

In addition when using the REST API, the following limits are in effect:

  • 1,000 HTML updates per experiment
  • each REST API key is allowed to make 15,000 submissions per hour

Solutions to rate limits

If you notice you are hitting rate limits based on normal experiments, please try reporting on each epoch, rather than each step.

If you still encounter rate limits, consider using the OfflineExperiment interface. This only requires that you change:

experiment = Experiment(...)

to

experiment = OfflineExperiment(..., offline_directory="/path/to/save/experiments")

After the experiment is completed, you then run:

comet upload /path/to/save/experiments/*.zip

in order to send your experiment to comet.ml.

For more information on how to best handle rate limits, please reach out to support@comet.ml or via our Slack channel.