Python Panels SDK API

This page documents the comet_ml.API for Comet Python Panels.

Other Python Panel Resources:


API

The API class is used as a Python interface to the Comet.ml Python Panels API.

You can use an instance of the API() class to quickly and easily access all of your logged information at comet.ml, including metrics, parameters, tags, and assets.

Example calls to get workspace, project, and experiment data:

  • API.get(): gets all of your personal workspaces
  • API.get(WORKSPACE): gets all of your projects from WORKSPACE
  • API.get(WORKSPACE, PROJECT_NAME): get all APIExperiments in WORKSPACE/PROJECT
  • API.get_experiment(WORKSPACE, PROJECT_NAME, EXPERIMENT_KEY): get an APIExperiment
  • API.get_experiment("WORKSPACE/PROJECT_NAME/EXPERIMENT_KEY"): get an APIExperiment
  • API.get_experiments(WORKSPACE): get all APIExperiments in WORKSPACE
  • API.get_experiments(WORKSPACE, PROJECT_NAME): get all APIExperiments in WORKSPACE/PROJECT
  • API.get_experiments(WORKSPACE, PROJECT_NAME, PATTERN): get all APIExperiments in WORKSPACE/PROJECT/PATTERN

Examples:

>>> import comet_ml
>>> api = comet_ml.api.API()

## Return all of my workspace names in a list:
>>> api.get()

## Get an APIExperiment:
>>> experiment = api.get("cometpublic/comet-notebooks/example 001")

## Get metrics:
>>> experiment.get_metrics("train_accuracy")

Within Python Panels, the API() class is limited only to read methods. To log new data to Comet, see the Python SDK.


API.get

get(self, workspace=None, project_name=None, experiment=None)

Get the following items:

  • list of workspace names, given no arguments
  • list of project names, given a workspace name
  • list of experiment names/keys, given workspace and project names
  • an experiment, given workspace, project, and experiment name/key

workspace, project_name, and experiment can also be given as a single string, delimited with a slash.


API.get_experiment

get_experiment(self, experiment_key, metadata=None)

Get a single APIExperiment by workspace, project, experiment.


API.get_experiment_asset

get_experiment_asset(self, experiment_key, asset_id, return_type="raw")

Get an asset, given the asset_id.

Args:

  • asset_id: the asset ID
  • return_type: the type of object returned. Default is "binary". Options: "binary" or "text"

Example:

>>> api_experiment.get_asset("298378237283728", return_type="json")
{...}

API.get_experiment_asset_by_name

get_experiment_asset_by_name(self, experiment_key, file_name,
    return_type=None)

Get an asset by name.


API.get_experiment_asset_list

get_experiment_asset_list(self, experiment_key, asset_type="all")

Get a list of assets associated with the experiment.

Args:

  • asset_type: Optional String, type of asset to return. Can be "all", "image", "histogram_combined_3d", "video", or "audio".

Returns a list of dictionaries of asset properties, like:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_asset_list(key)
[{'fileName': 'My Filename.png',
'fileSize': 21113,
'runContext': None,
'step': None,
'link': 'https://www.comet.ml/api/asset/download?experimentKey=KEY&assetId=ASSET_ID',
'createdAt': 1565898755830,
'dir': 'assets',
'canView': False,
'audio': False,
'video': False,
'histogram': False,
'image': True,
'type': 'image',
'metadata': None,
'assetId': ASSET_ID}, ...]

>>> api.get_experiment_asset_list(key, "image")
[{'fileName': 'My Filename.png',
'fileSize': 21113,
'runContext': None,
'step': None,
'link': 'https://www.comet.ml/api/asset/download?experimentKey=KEY&assetId=ASSET_ID',
'createdAt': 1565898755830,
'dir': 'assets',
'canView': False,
'audio': False,
'video': False,
'histogram': False,
'image': True,
'type': 'image',
'metadata': None,
'assetId': ASSET_ID}, ...]

API.get_experiment_by_id

get_experiment_by_id(self, experiment)

API.get_experiment_by_key

get_experiment_by_key(self, experiment_key)

Get an APIExperiment by experiment key.


API.get_experiment_code

get_experiment_code(self, experiment_key)

Get the associated source code for this experiment.

Example:

>>> api.get_experiment_code(key)
'import comet_ml
experiment = comet_ml.Experiment()
experiment.end()'

API.get_experiment_curves

get_experiment_curves(self, experiment_key)

Get all curve assets of an experiment as bytes.

Example:

>>> api.get_experiment_curves(key)
[...]

API.get_experiment_graph

get_experiment_graph(self, experiment_key)

Alias of APIExperiment.get_model_graph()

Get the associated graph/model description for this experiment.

Example:

>>> api_experiment.get_model_graph()
{"class_name": "Sequential", ...}

API.get_experiment_html

get_experiment_html(self, experiment_key)

Get the HTML associated with the experiment.


API.get_experiment_images

get_experiment_images(self, experiment_key)

Get all image assets of an experiment as bytes.

Example:

>>> api.get_experiment_images(key)
[...]

API.get_experiment_installed_packages

get_experiment_installed_packages(self, experiment_key)

Get the associated installed packages for this experiment.

Example:

>>> api_experiment.get_installed_packages()
['absl-py==0.8.1', 'adal==1.2.2', 'alabaster==0.7.12', ...]

API.get_experiment_metadata

get_experiment_metadata(self, experiment_key)

Returns the JSON metadata for an experiment

Returns:

{
  'archived': False,
  'durationMillis': 7,
  'endTimeMillis': 1586174765277,
  'experimentKey': 'EXPERIMENT-KEY',
  'experimentName': None,
  'fileName': None,
  'filePath': None,
  'optimizationId': None,
  'projectId': 'PROJECT-ID',
  'projectName': 'PROJECT-NAME',
  'running': False,
  'startTimeMillis': 1586174757596,
  'throttle': False,
  'workspaceName': 'WORKSPACE-NAME',
}

API.get_experiment_metric

get_experiment_metric(self, experiment_key, metric)

Return an experiment metric values.

Args:

  • experiment_key: (string), key of experiment
  • metric: (string), name of a metric

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_metric(key, 'val_loss')

API.get_experiment_metrics_summaries

get_experiment_metrics_summaries(self, experiment_key)

Return the experiment metrics summary.

Args:

  • experiment_key: (string), key of experiment

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_metrics_summaries(key)
[{'name': 'val_loss',
'valueMax': '0.24951280827820302',
'valueMin': '0.13101346811652184',
'valueCurrent': '0.13101346811652184',
'timestampMax': 1558962367938,
'timestampMin': 1558962367938,
'timestampCurrent': 1558962376383,
'stepMax': 500,
'stepMin': 1500,
'stepCurrent': 1500},
...]

API.get_experiment_model_asset_list

get_experiment_model_asset_list(self, experiment_key, model_name)

Get an experiment model's asset list by model name.

Args:

  • model_name: str, the name of the model

Returns: a list of asset dictionaries with these fields:

  • fileName
  • fileSize
  • runContext
  • step
  • link
  • createdAt
  • dir
  • canView
  • audio
  • histogram
  • image
  • type
  • metadata
  • assetId

Example:

>>> from comet_ml import API
>>> api = API()
>>> api.get_experiment_model_asset_list(key, "Model Name")
[
{
  "assetId": 74374637463476,
  "audio": False,
  "canView": False,
  "createdAt": 7337347634,
  "dir": "trained-models",
  "fileName": "model.h5",
  "fileSize": 254654,
  "histogram": False,
  "image": False,
  "link": "https://link-to-download-asset-file",
  "metadata": None,
  "remote": False,
  "runContext": "train",
  "step": 54,
  "type": "asset",
}
]

API.get_experiment_model_graph

get_experiment_model_graph(self, experiment_key)

Get the associated graph/model description for this experiment.

Example:

>>> api.get_experiment_model_graph(key)
{"class_name": "Sequential", ...}

API.get_experiment_models

get_experiment_models(self, experiment_key)

Given an experiment id, return a list of model data associated with an experiment.

Args:

  • experiment_id: the experiment's key

Returns [{'experimentModelId': 'MODEL-ID' 'experimentKey': 'EXPERIMENT-KEY', 'modelName': 'MODEL-NAME'}, ...]


API.get_experiment_name

get_experiment_name(self, experiment_key, max_length=None)

Get the experiment name for this experiment key.


API.get_experiment_others_summaries

get_experiment_others_summaries(self, experiment_key)

Get the other items logged in summary form.

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_others_summary(key)
[{'name': 'trainable_params',
  'valueMax': '712723',
  'valueMin': '712723',
  'valueCurrent': '712723',
  'timestampMax': 1558962363411,
  'timestampMin': 1558962363411,
  'timestampCurrent': 1558962363411},
...]

API.get_experiment_output

get_experiment_output(self, experiment_key)

Get the associated standard output for this experiment.

Example:

>>> api.get_experiment_output(key)
"Experiment is live"

API.get_experiment_parameters_summaries

get_experiment_parameters_summaries(self, experiment_key)

Return the experiment parameters summary.

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_parameters_summaries()
[{'name': 'batch_size',
  'valueMax': '120',
  'valueMin': '120',
  'valueCurrent': '120',
  'timestampMax': 1558962363411,
  'timestampMin': 1558962363411,
  'timestampCurrent': 1558962363411},
...]

API.get_experiment_tags

get_experiment_tags(self, experiment_key)

Get the associated tags for this experiment.

Example:

>>> api.get_experiment_tags(key)
["best"]

API.get_experiment_url

get_experiment_url(self, experiment_key)

Get the url of the experiment.

Example:

>>> api.get_experiment_url()
"https://www.comet.ml/username/34637643746374637463476"

API.get_experiments

get_experiments(self, workspace, project_name=None, pattern=None)

Get APIExperiments by workspace, workspace + project, or workspace + project + regular expression pattern.


API.get_metrics_for_chart

get_metrics_for_chart(self, experiments, metrics=[], parameters=[],
    independent_metrics=True, fetch_full=False)

Get multiple metrics and parameters from a set of experiments. This method is designed to make custom charting easier.

Args:

  • experiments: a list of experiments or experiment keys
  • metrics: an optional list of metric names (e.g., "loss"); OR
  • parameters: an optional list of parameter names (e.g., "learning-rate")
  • independent: Bool, get independent results?
  • full: Bool, fetch the full result?

Note: you should pass in a list of metric names, or a list of

parameter names, or both.

Returns: a dictionary of experiment keys with the following

structure. {EXPERIMENT_KEY: {'params'} will be None if there are no parameters passed in.

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_metrics_for_chart([experiment_key1, experiment_key2, ...],
  ["loss"], ["hidden_layer_size"])
{EXPERIMENT_KEY: {
'experiment_key': EXPERIMENT_KEY,
'steps': STEPS,
'epochs': None,
'metrics': [
  {'metricName': 'loss',
  'values': [VALUE, ...],
  'steps': [STEP, ...],
  'epochs': [EPOCH, ...],
  'timestamps': [TIMESTAMP, ...],
  'durations': [DURATION, ...],
  }],
'params': {'hidden_layer_size': VALUE, ...},
}, ...}

API.get_model_graph

get_model_graph(self, experiment_key)

Get the associated graph/model description for this experiment.

Example:

>>> api_experiment.get_model_graph()
{"class_name": "Sequential", ...}

API.get_panel_experiment_keys

get_panel_experiment_keys(self)

Get the current panel's experiment keys.

Example:

from comet_ml import API

api = API()
experiment_keys = api.get_panel_experiment_keys()

This will return a list containing all of the current experiment keys. Specifically:

  • If on a Project View, the experiment keys from the visible experiment table page
  • If a filter is set, all of the matching experiments
  • If a Query is set for this Panel, the matching experiments

API.get_panel_experiments

get_panel_experiments(self)

Get the current panel's experiments.

Example:

from comet_ml import API

api = API()
experiment = api.get_panel_experiments()

Like the API.get_experiment_keys() method, but returns a list of APIExperiments rather than just their keys.


API.get_panel_metrics_names

get_panel_metrics_names(self, include_sys=False)

Get the current panel's experiments' metrics' names.

Args:

  • include_sys (bool): if True, include the system metrics

Example:

from comet_ml import API, ui

api = API()
metrics_names = api.get_panel_metrics_names()
metric_name = ui.dropdown("Select a metric name:", metrics_names)

As shown, the metrics_names could be useful in creating a dropdown selection for plotting.

Note that, by default, this list does not includes all metrics. Use include_sys=True to include system metrics as well.


API.get_panel_options

get_panel_options(self)

Get the current panel's options as a Python dictionary from the Option tab for this Panel instance.

Example:

from comet_ml import API

api = API()
options = api.get_panel_options()

The Panel Options can be edited when in the Options tab of the Panel Code editor, and set when you are adding a Panel instance to a View.


API.get_panel_project_id

get_panel_project_id(self)

Get the current panel's ID.

Example:

from comet_ml import API

api = API()
project_id = api.get_panel_project_id()

The project_id can be useful to retrieve other Project-level information.


API.get_panel_project_name

get_panel_project_name(self)

Get the current panel's name.

Example:

from comet_ml import API

api = API()
project_name = api.get_panel_project_name()

The Project name could be useful in creating report-like Panels.


API.get_panel_report

get_panel_report(self)

Get a string detailing the number of hits the panel has made to the cloud.


API.get_panel_request_count

get_panel_request_count(self)

Get the number of requests made to the Comet cloud server.


API.get_panel_workspace

get_panel_workspace(self)

Get a list of workspace names.

Example:

from comet_ml import API

api = API()
workspace = api.get_panel_workspace()

The Workspace name could be useful in creating report-like Panels.


API.get_project

get_project(self, workspace=None, project_name=None, project_id=None)

Return the details of a project in a workspace.

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project

Returns a dict of project details if the workspace/project exists, otherwise None.

Examples:

>>> api.get_project("workspace", "project-name")
{'projectId': 'project-id',
'projectName': 'project-name',
'ownerUserName': 'user name',
'projectDescription': 'my description',
'workspaceName': 'workspace',
'numberOfExperiments': 14,
'lastUpdated': 1571747775420,
'public': False}

>>> api.get_project("workspace", "non-existent-project-name")
None

API.get_project_columns

get_project_columns(self, workspace=None, project_name=None,
    project_id=None)

Given a workspace and project_name return the column names, types, etc.


API.get_project_experiments

get_project_experiments(self, workspace, project_name)

Return the metadata JSONs of the experiments in a project.


API.get_project_notes

get_project_notes(self, project_id)

Get the notes of a project.

Args:

  • project_id: String, the ID of the project

Returns: a string

Example:

>>> api.get_experiment_project_notes("my-workspace", "my-project")
"These are my project-level notes"

API.get_projects

get_projects(self, workspace)

Return the details of the projects in a workspace.

Args:

  • workspace: String, the name of the workspace

Returns a list of project details in workspace.


API.get_query_variables

get_query_variables(self, workspace=None, project_name=None,
    project_id=None)

Return the query variables of a project in a workspace. Used with API.query().

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project

Returns objects used in forming queries, like:

[Metadata("user_name"),
Metadata("start_server_timestamp"),
Tag("my_tag"),
...]

API.get_workspaces

get_workspaces(self)

Return a list of names of the workspaces for this user.


API.query

query(self, workspace, project_name, query, archived=False)

Perform a query on a workspace/project to find matching APIExperiment. Queries are composed of

Args:

  • workspace: String, the name of the workspace
  • project_name: String, the name of the project
  • query: a query expression (see below)
  • archived: (optional boolean), query the archived experiments if True
((QUERY-VARIABLE OPERATOR VALUE) & ...)

# or:

(QUERY-VARIABLE.METHOD(VALUE) & ...)

where:

QUERY-VARIABLE is Environment(NAME), Metric(NAME), Parameter(NAME), Other(NAME), Metadata(NAME), or Tag(VALUE).

OPERATOR is any of the standard mathematical operators ==, <=, >=, !=, <, >.

METHOD is between(), contains(), startswith(), or endswith().

You may also place the bitwise ~ not operator in front of an expression which means to invert the expression. Use & to combine additional criteria. Currently, | (bitwise or) is not supported.

VALUE can be any query type, includeing string, boolean, double, datetime, or timenumber (number of seconds). None and "" are special values that mean NULL and EMPTY, respectively. Use API.get_query_variables(WORKSPACE, PROJECT_NAME) to see query variables and types for a project.

When using datetime, be aware that the backend is using UTC datetimes. If you do not receive the correct experiments via a datetime query, please check with the web UI query builder to verify timezone of the server.

query() returns a list of matching APIExperiments().

Examples:

# Find all experiments that have an acc metric value > .98:
>>> api.query("workspace", "project", Metric("acc") > .98)
[APIExperiment(), ...]

# Find all experiments that have a loss metric < .1 and
# a learning_rate parameter value >= 0.3:
>>> loss = Metric("loss")
>>> lr = Parameter("learning_rate")
>>> query = ((loss < .1) & (lr >= 0.3))
>>> api.query("workspace", "project", query)
[APIExperiment(), ...]

# Find all of the experiments tagged "My simple tag":
>>> tagged = Tag("My simple tag")
>>> api.query("workspace", "project", tagged)
[APIExperiment(), ...]

# Find all experiments started before Sept 24, 2019 at 5:00am:
>>> q = Metadata("start_server_timestamp") < datetime(2019, 9, 24, 5)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]

# Find all experiments lasting more that 2 minutes (in seconds):
>>> q = Metadata("duration") > (2 * 60)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]

Notes:

  • Use ~ for not on any expression
  • Use ~QUERY-VARIABLE.between(2,3) for values not between 2 and 3
  • Use (QUERY-VARIABLE == True) for truth
  • Use (QUERY-VARIABLE == False) for not true
  • Use (QUERY-VARIABLE == None) for testing null
  • Use (QUERY-VARIABLE != None) or ~(QUERY-VARIABLE == None) for testing not null
  • Use (QUERY-VARIABLE == "") for testing empty
  • Use (QUERY-VARIABLE != "") or ~(QUERY-VARIABLE == "") for testing not empty
  • Use Pythons datetime(YEAR, MONTH, DAY, HOUR, MINUTE, SECONDS) for comparing datetimes, likeMetadata("start_server_timestamp")orMetadata("end_server_timestamp")`
  • Use seconds for comparing timenumbers, like Metadata("duration")
  • Use API.get_query_variables(WORKSPACE, PROJECT_NAME) to see query variables and types.

Do not use and, or, not, is, or in. These are logical operators and you must use mathematical operators for queries. For example, always use == where you might usually use is.


API.query_project

query_project(self, workspace, project_name, predicates, archived=False)

Given a workspace, project_name, and predicates, return matching experiments.