Warnings and Errors
For further assistance on any of these Python warnings or errors, or if you see an error message that is not noted here, please ping us on our slack channel
ImportError: Please import comet before importing these modules: ...
This error occurs when you try to create an
Experiment (or another kind of experiment, such as
OfflineExperiment) but have imported
comet_ml after one of the supported auto-logging machine learning libraries (such as torch, fastai, keras, or tensorflow). You have two choices to resolve this error: you can either move comet_ml to be imported first (recommended), or you can completely disable Comet's auto logging facility by setting
COMET_DISABLE_AUTO_LOGGING=1 in the environment, or in your Comet config file. See Quick Start for more details.
COMET ERROR: Run will not be logged
This error is shown with a Python stack trace and indicates that the initial handshake between Comet and the server failed. This is usually a local networking issue or production downtime. Please reach out on our slack channel if you encounter this error.
COMET ERROR: Failed to set run source code
Comet failed to read the source code file for this experiments. This could be due to using a dispatcher.
The Comet API may rate limit submission of requests for your experiments. Such limits are managed as an allowed number of operations per time window, where an operation might be read or an update. In that case, a warning symbol will appear on your experiment list.
For online experiments, the following rate limits are in effect:
- logging metrics: 10,000 per minute
- logging parameters: 8,000 per minute
- logging output: 10,000 per minute
- logging everything else: 8,000 per minute
For offline experiments, the following rate limits are in effect:
- logging metrics: 80,000 per minute
- logging parameters: 80,000 per minite
- logging output: 80,000 per minute
- logging everything else: 80,000 per minute
Offline experiments have a rate limit because you may be uploading multiple experiments in parallel, or attempting to upload too many too quickly.
Size and Count limits¶
For all experiments, you can log 15,000 total values for each metric, per experiment. If your metric count goes beyond this limit, then the values are downsampled.
When using the REST API (from the Python SDK, or using the URL endpoints directly), the following limits are in effect per experiment.
Each of the following REST API items typically only need to be logged once per experiment.
- 10 environment detail updates
- 10 git metadata updates
- 10 graph (model) updates
- 10 OS packages updates
- 10 code updates
In addition when using the REST API, the following limits are in effect:
- 1,000 HTML updates per experiment
- each REST API key is allowed to make 15,000 submissions per hour
Solutions to rate limits¶
If you notice you are hitting rate limits based on normal experiments, please try reporting on each epoch, rather than each step.
If you still encounter rate limits, consider using the
OfflineExperiment interface. This only requires that you change:
experiment = Experiment(...)
experiment = OfflineExperiment(..., offline_directory="/path/to/save/experiments")
After the experiment is completed, you then run:
comet upload /path/to/save/experiments/*.zip
in order to send your experiment to comet.ml.