Python API

API

The API class is used as a Python interface to the Comet.ml Python API.

You can use an instance of the API() class to quickly and easily access all of your logged information at comet.ml, including metrics, parameters, tags, and assets.

Example calls to get workspace, project, and experiment data:

  • API.get(): gets all of your personal workspaces
  • API.get(WORKSPACE): gets all of your projects from WORKSPACE
  • API.get(WORKSPACE, PROJECT_NAME): get all APIExperiments in WORKSPACE/PROJECT
  • API.get_experiment(WORKSPACE, PROJECT_NAME, EXPERIMENT_ID): get an APIExperiment
  • API.get_experiment("WORKSPACE/PROJECT_NAME/EXPERIMENT_ID"): get an APIExperiment
  • API.get_experiments(WORKSPACE): get all APIExperiments in WORKSPACE
  • API.get_experiments(WORKSPACE, PROJECT_NAME): get all APIExperiments in WORKSPACE/PROJECT
  • API.get_experiments(WORKSPACE, PROJECT_NAME, PATTERN): get all APIExperiments in WORKSPACE/PROJECT/PATTERN

Examples:

>>> import comet_ml
>>> comet_api = comet_ml.api.API()

## Return all of my workspace names in a list:
>>> comet_api.get()

## Get an APIExperiment:
>>> experiment = comet_api.get("cometpublic/comet-notebooks/example 001")

## Get metrics:
>>> experiment.get_metrics("train_accuracy")

The API instance also gives you access to the low-level Python API function calls:

>>> comet_api.delete_experiment(experiment_key)

For more usage examples, see Comet Python API examples.

API.server_url


API.__init__

__init__(self, api_key=None, cache=True, version="v2", rest_api_key=None)

Application Programming Interface to the Comet Python interface.

Args:

  • api_key: Optional. Your private COMET_API_KEY.
  • cache: Bool, whether to cache on values or not.
  • version: Optional. The version of the REST API to use.

Note: api_key may be defined in environment (COMET_API_KEY)

or in a .comet.config file.

Example:

>>> from comet_ml.api import API
>>> comet_api = API(api_key="08ac6a75a2be4d7c9aac2c39e0004f6e")
>>> comet_api.get("my-workspace")
['project1', 'project2', ...]

API.add_registry_model_version_stage

add_registry_model_version_stage(self, workspace, registry_name, version, stage)

Adds a stage to a registered model version.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model
  • version: str, version of model to update
  • stage: str, "production", or "staging", etc.

API.archive_experiment

archive_experiment(self, experiment_key)

Archive one experiment.

Args:

  • experiment_key: the experiment ID to archive

API.archive_experiments

archive_experiments(self, experiment_keys)

Archive list of experiments.

Args:

  • experiment_keys: the experiment IDs to archive

API.clear_cache

clear_cache(self)

Used when cache is on, but you have added/changed data outside of this API instance.

Note: you could also just start with no cache.

>>> api = API(cache=False)

Or, if you had started with cache, turn it off:

>>> api = API(cache=True)
>>> api.do_cache(False)

API.create_project

create_project(self, workspace, project_name, project_description=None, public=False)

Create a project.


API.create_project_share_key

create_project_share_key(self, project_id)

Get the share keys for a private project ID.

Args:

  • project_id: String, the ID of the project

Example:

>>> api = API()
>>> SHARE_KEY = api.create_project_share_key(PROJECT_ID)

See also: API.get_project_share_keys(), and API.delete_project_share_key().


API.delete_experiment

delete_experiment(self, experiment_key)

Delete one experiment.


API.delete_experiments

delete_experiments(self, experiment_keys)

Delete list of experiments.

Args:

  • experiment_keys: a list of experiment keys to delete.

API.delete_project

delete_project(self, workspace=None, project_name=None, project_id=None, delete_experiments=False)

Delete a project.

Args:

  • workspace: the name of the workspace (required if project_id not given)
  • project_name: the name of the project (required if project_id not given)
  • project_id: the project id (required, if workspace and project name not given)
  • delete_experiments: if True, delete all of the experiments, too

API.delete_project_share_key

delete_project_share_key(self, project_id, share_key)

Delete a share key for a private project ID.

Args:

  • project_id: String, the ID of the project
  • share_key: String, the share key to delete

Example:

>>> api = API()
>>> SHARE_KEYS = api.get_project_share_keys(PROJECT_ID)
>>> api.delete_project_share_key(PROJECT_ID, SHARE_KEYS[0])

See also: API.get_project_share_keys(), and API.create_project_share_key().


API.delete_registry_model

delete_registry_model(self, workspace, registry_name)

Deletes a registered model.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model

API.delete_registry_model_version

delete_registry_model_version(self, workspace, registry_name, version)

Deletes a registered model version.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model
  • version: str, version of model to update

API.delete_registry_model_version_stage

delete_registry_model_version_stage(self, workspace, registry_name, version, stage)

Removes a stage from a registered model version.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model
  • version: str, version of model to update
  • stage: str, "production", or "staging", etc.

API.do_cache

do_cache(self, *endpoints)

Cache the given endpoints.

Example:

>>> from comet_ml.api import API
>>> comet_api = API()
>>> comet_api.do_cache("experiments", "projects")

API.do_not_cache

do_not_cache(self, *endpoints)

Do not cache the given endpoints.

Example:

>>> from comet_ml.api import API
>>> comet_api = API()
>>> comet_api.do_not_cache("experiments", "projects")

API.download_registry_model

download_registry_model(self, workspace, registry_name, version=None, output_path="./", expand=True, \
    stage=None)

Download and save all files from the registered model.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model
  • version: str, version string of the model
  • output_path: optional, str, the output directory; defaults to current directory
  • expand: if True, the downloaded zipfile is unzipped; if False, then the zipfile is copied to the output_path
  • stage: optional, a textual tag such as "production" or "staging"

API.gen_experiments

gen_experiments(self, workspace, project_name=None, pattern=None)

Get APIExperiments by workspace, workspace + project, or workspace + project + regular expression pattern.


API.get

get(self, workspace=None, project_name=None, experiment=None)

Get the following items:

  • list of workspace names, given no arguments
  • list of project names, given a workspace name
  • list of experiment names/keys, given workspace and project names
  • an experiment, given workspace, project, and experiment name/key

workspace, project_name, and experiment can also be given as a single string, delimited with a slash.


API.get_account_details

get_account_details(self)

Return the username and the default workspace name for the authorized user.

Returns:

{
'userName': 'USERNAME',
'defaultWorkspaceName': 'WORKSPACE',
}

API.get_archived_experiment

get_archived_experiment(self, workspace, project_name, experiment)

Get a single archived APIExperiment by workspace, project, experiment.


API.get_archived_experiments

get_archived_experiments(self, workspace, project_name=None, pattern=None)

Get archived APIExperiments by workspace, workspace + project, or workspace + project + regular expression pattern.


API.get_experiment

get_experiment(self, workspace, project_name, experiment)

Get a single APIExperiment by workspace, project, experiment.


API.get_experiment_by_id

get_experiment_by_id(self, experiment)

Get an APIExperiment by experiment id.


API.get_experiments

get_experiments(self, workspace, project_name=None, pattern=None)

Get APIExperiments by workspace, workspace + project, or workspace + project + regular expression pattern.


API.get_metrics_for_chart

get_metrics_for_chart(self, experiment_keys, metrics=None, parameters=None, independent=True, \
    full=False)

Get multiple metrics and parameters from a set of experiments. This method is designed to make custom charting easier.

Args:

  • experiment_keys: a list of experiment keys
  • metrics: an optional list of metric names (e.g., "loss")
  • parameters: an optional list of parameter names (e.g., "learning-rate")
  • independent: Bool, get independent results?
  • full: Bool, fetch the full result?

Note: you should pass in a list of metric names, or a list of

parameter names, or both.

Returns: a dictionary of experiment keys with the following

structure. {EXPERIMENT_KEY: {'params'} will be None if there are no parameters passed in.

>>> from comet_ml.api import API
>>> comet_api = API()
>>> comet_api.get_metrics_for_chart([experiment_key1, experiment_key2, ...],
  ["loss"], ["hidden_layer_size"])
{EXPERIMENT_KEY: {
'experiment_key': EXPERIMENT_KEY,
'steps': STEPS,
'epochs': None,
'metrics': [
  {'metricName': 'loss',
  'values': [VALUE, ...],
  'steps': [STEP, ...],
  'epochs': [EPOCH, ...],
  'timestamps': [TIMESTAMP, ...],
  'durations': [DURATION, ...],
  }],
'params': {'hidden_layer_size': VALUE, ...},
}, ...}

API.get_project

get_project(self, workspace, project_name)

Return the details of a project in a workspace.

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project

Returns a dict of project details if the workspace/project exists, otherwise None.

Examples:

>>> api.get_project("workspace", "project-name")
{'projectId': 'project-id',
'projectName': 'project-name',
'ownerUserName': 'user name',
'projectDescription': 'my description',
'workspaceName': 'workspace',
'numberOfExperiments': 14,
'lastUpdated': 1571747775420,
'public': False}

>>> api.get_project("workspace", "non-existent-project-name")
None

API.get_project_by_id

get_project_by_id(self, project_id)

Return the details of a project given its project id.

Args:

  • project_id: String, the ID of the project

Returns a dict of project details if the project_id exists, otherwise None.

Examples:

>>> api.get_project_by_id("2727432637263")
{'projectId': '2727432637263',
'projectName': 'project name',
'ownerUserName': 'user name',
'projectDescription': 'my description',
'workspaceName': 'workspace',
'numberOfExperiments': 14,
'lastUpdated': 1571747775420,
'public': False}

>>> api.get_project_by_id("non-existent-project-id")
None

API.get_project_notes

get_project_notes(self, workspace, project_name)

Get the notes of a project.

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project

Returns: a string

Example:

>>> api.get_project_notes("my-workspace", "my-project")
"These are my project-level notes"

API.get_project_share_keys

get_project_share_keys(self, project_id)

Get the share keys for a private project ID.

Args:

  • project_id: String, the ID of the project

Example:

>>> api = API()
>>> SHARE_KEYS = api.get_project_share_keys(PROJECT_ID)

See also: API.create_project_share_key(), and API.delete_project_share_key().


API.get_projects

get_projects(self, workspace)

Return the details of the projects in a workspace.

Args:

  • workspace: String, the name of the workspace

Returns a list of project details in workspace.


API.get_query_variables

get_query_variables(self, workspace, project_name)

Return the query variables of a project in a workspace. Used with API.query().

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project

Returns a list of QueryVariables, like:

[Metadata('user_name'),
Metadata('start_server_timestamp'),
...]

API.get_registry_model_count

get_registry_model_count(self, workspace)

Get a count of the number of registered models in this workspace.

Args:

  • workspace: the name of workspace

API.get_registry_model_details

get_registry_model_details(self, workspace, registry_name, version=None)

Get the details of a registered model in a workspace. If version is given then it will return the details of the workspace/registry-name/version. Otherwise, it will return the details of the workspace/registry-name.

Args:

  • workspace: the name of workspace
  • registry_name: the name of the model
  • version: optional, the version str of the model

Example:

>>> from comet_ml import API
>>> api = API()
>>> api_exp = api.get("workspace/project/765643463546345364536453436")
>>> api_exp.get_registry_model_details("myworkspace", "model-name")
{
"registryModelId": "someRegistryModelId",
"modelName": "someModelName",
"description": "someDescription",
"isPublic": "[Boolean]",
"createdAt": "[long, when this model was created in the DB]",
"lastUpdated": "[long, last time this model was updated in the DB]",
"userName": "someUserName",
"versions": [
  {
  "registryModelItemId": "someRegistryModelItemId",
  "experimentModel": {
  "experimentModelId": "someExperimentModelId",
  "experimentModelName": "someExperimentModelName",
  "experimentKey": "someExperimentKey"
  },
  "version": "someVersion",
  "comment": "someComment",
  "stages": ["production", "staging"],
  "userName": "someUserName",
  "createdAt": "[long, when this model item was created in the DB]",
  "lastUpdated": "[long, last time this model item was updated in the DB]",
  "assets": [
  {
  "fileName": "someFileName",
  "fileSize": "[Long, file size]",
  "runContext": "someRunContext",
  "step": "[Integer, step asset was logged during]",
  "link": "link to download asset file",
  "createdAt": "[Long, timestamp asset was created in DB]",
  "dir": "someDirectory",
  "canView": "[Boolean, whether the asset is viewable as an image]",
  "audio": "[Boolean, whether the asset is an audio file]",
  "histogram": "[Boolean, whether the asset is a histogram file]",
  "image": "[Boolean, whether the asset was stored as an image]",
  "type": "the type of asset",
  "metadata": "Metadata associated with the asset",
  "assetId": "someAssetId",
  }
  ],
  }
]
}

API.get_registry_model_names

get_registry_model_names(self, workspace)

Get a list of model names associated with this workspace.

Args:

  • workspace: the name of workspace

Returns: list of model names


API.get_registry_model_notes

get_registry_model_notes(self, workspace, registry_name)

Get the notes of a registered model in a workspace.

Args:

  • workspace: the name of workspace
  • registry_name: the name of the model

API.get_registry_model_versions

get_registry_model_versions(self, workspace, registry_name)

Get a list of the version strings of a registered model in a workspace.

Args:

  • workspace: the name of workspace
  • registry_name: the name of the model

API.get_workspaces

get_workspaces(self)

Return a list of names of the workspaces for this user.


API.move_experiments

move_experiments(self, experiment_keys, target_workspace, target_project_name, symlink=False)

Move/symlink list of experiments to another workspace/project_name

Args:

  • experiment_keys: (list) - list of experiment keys
  • target_workspace: (str) - workspace name to move experiments to
  • target_project_name: (str) - project name to move experiments to
  • symlink: (bool, optional) - if True, then create a symlink in target_workspace/target_project_name.

Example:

# Move all experiments with a particular tag:
>>> from comet_ml.query import Tag
>>> from comet_ml import API
>>> api = API() # assumes configured with api_key
>>> experiments = api.query("workspace", "project", Tag("My tag"))
>>> api.move_experiments([e.id for e in experiments],
  "other-workpsace",
  "other-project")

API.query

query(self, workspace, project_name, query, archived=False)

Perform a query on a workspace/project to find matching APIExperiment. Queries are composed of

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project
  • query: a query expression (see below)
  • archived: (optional boolean), query the archived experiments if True
((QUERY-VARIABLE OPERATOR VALUE) & ...)

# or:

(QUERY-VARIABLE.METHOD(VALUE) & ...)

where:

QUERY-VARIABLE is Metric(NAME), Parameter(NAME), Other(NAME), Metadata(NAME), or Tag(VALUE).

OPERATOR is any of the standard mathematical operators ==, <=, >=, !=, <, >.

METHOD is between(), contains(), startswith(), or endswith().

You may also place the bitwise ~ not operator in front of an expression which means to invert the expression. Use & to combine additional criteria. Currently, | (bitwise or) is not supported.

VALUE can be any query type, includeing string, boolean, double, datetime, or timenumber (number of seconds). None and "" are special values that mean NULL and EMPTY, respectively. Use API.get_query_variables(WORKSPACE, PROJECT_NAME) to see query variables and types for a project.

When using datetime, be aware that the backend is using UTC datetimes. If you do not receive the correct experiments via a datetime query, please check with the web UI query builder to verify timezone of the server.

query() returns a list of matching APIExperiments().

Examples:

# Find all experiments that have an acc metric value > .98:
>>> api.query("workspace", "project", Metric("acc") > .98)
[APIExperiment(), ...]

# Find all experiments that have a loss metric < .1 and
# a learning_rate parameter value >= 0.3:
>>> loss = Metric("loss")
>>> lr = Parameter("learning_rate")
>>> query = ((loss < .1) & (lr >= 0.3))
>>> api.query("workspace", "project", query)
[APIExperiment(), ...]

# Find all of the experiments tagged "My simple tag":
>>> tagged = Tag("My simple tag")
>>> api.query("workspace", "project", tagged)
[APIExperiment(), ...]

# Find all experiments started before Sept 24, 2019 at 5:00am:
>>> q = Metadata("start_server_timestamp") < datetime(2019, 9, 24, 5)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]

# Find all experiments lasting more that 2 minutes (in seconds):
>>> q = Metadata("duration") > (2 * 60)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]

Notes:

  • Use ~ for not on any expression
  • Use ~QUERY-VARIABLE.between(2,3) for values not between 2 and 3
  • Use (QUERY-VARIABLE == True) for truth
  • Use (QUERY-VARIABLE == False) for not true
  • Use (QUERY-VARIABLE == None) for testing null
  • Use (QUERY-VARIABLE != None) or ~(QUERY-VARIABLE == None) for testing not null
  • Use (QUERY-VARIABLE == "") for testing empty
  • Use (QUERY-VARIABLE != "") or ~(QUERY-VARIABLE == "") for testing not empty
  • Use Python's datetime(YEAR, MONTH, DAY, HOUR, MINUTE, SECONDS) for comparing datetimes, like Metadata("start_server_timestamp") or Metadata("end_server_timestamp")
  • Use seconds for comparing timenumbers, like Metadata("duration")
  • Use API.get_query_variables(WORKSPACE, PROJECT_NAME) to see query variables and types.

Do not use 'and', 'or', 'not', 'is', or 'in'. These are logical operators and you must use mathematical operators for queries. For example, always use '==' where you might usually use 'is'.


API.restore_experiment

restore_experiment(self, experiment_key)

Restore one experiment.

Args:

  • experiment_key: the experiment ID to restore

API.set_project_notes

set_project_notes(self, workspace, project_name, notes)

Set the notes of a project. Overwrites any previous notes.

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project notes: String, the full notes

Returns: a JSON message

Example:

>>> api.set_project_notes("my-workspace", "my-project",
...                       "These are my project-level notes")
{'msg': 'saved', 'code': 200, 'data': None, 'sdk_error_code': 0}

API.stop_experiment

stop_experiment(self, experiment_key)

Stop a running experiment.

Args:

  • experiment_key: the experiment ID

Example:

import comet_ml

# Start an online experiment:
experiment = comet_ml.Experiment()

# Perhaps somewhere else, while experiment
# is running:
api = comet_ml.API()
api.stop_experiment(experiment.id)

API.update_cache

update_cache(self)

Deprecated: Use API.clear_cache()


API.update_project

update_project(self, workspace, project_name, new_project_name=None, description=None, public=None)

Update the metadata of a project by project_name and workspace.

Args:

  • workspace: name of workspace
  • project_name: name of project
  • new_project_name: new name of project (optional)
  • description: new description of project (optional)
  • public: new setting of visibility (optional)

Example:

>>> api_experiment.update_project("mywork", "oldproj",
...     new_project_name="newproj", description="desc",
...     public=True)

API.update_project_by_id

update_project_by_id(self, project_id, new_project_name=None, description=None, public=None)

Update the metadata of a project by project_id.

Args:

  • project_id: project id
  • new_project_name: new name of project (optional)
  • description: new description of project (optional)
  • public: new setting of visibility (optional)

Example:

>>> api_experiment.update_project_by_id("2627523253623",
...     new_project_name="newproj", description="desc",
...     public=True)

API.update_registry_model

update_registry_model(self, workspace, registry_name, new_name=None, description=None, public=None)

Updates a registered model's name, description, and/or visibility.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model
  • new_name: optional str, new name of model
  • description: optional str, new description of model
  • public: optional bool, new visibility of model

API.update_registry_model_notes

update_registry_model_notes(self, workspace, registry_name, notes)

Updates a registered model's notes.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model notes: str, notes of model

API.update_registry_model_version

update_registry_model_version(self, workspace, registry_name, version, comment=None, stages=None)

Update a registered model version's comments and stages.

Args:

  • workspace: str, the name of the workspace
  • registry_name: str, the name of the model
  • version: str, version of model to update
  • comments: optional, str, comments of model version
  • stages: optional, list, new list of stages, e.g. ["production", "staging"]

API.use_cache

use_cache(self, cache=None)

Turn cache on/off or return cache.

Examples:

>>> from comet_ml.api import API
>>> comet_api = API()
>>> comet_api.use_cache(False)
>>> comet_api.use_cache()
False

>>> comet_api.use_cache(True)
>>> comet_api.use_cache()
True