keras

Automatically logging keras experiments

Comet.ml will monitor training, logging metrics, parameters, and histograms to your keras code without requiring you to do anything other than adding these lines of code to your Keras script, and setting the experiment parameters to the appropriate values for what you would like to log:

from comet_ml import Experiment
experiment = Experiment(
    auto_metric_logging=True,
    auto_param_logging=True,
    auto_histogram_weight_logging=True,
    auto_histogram_gradient_logging=True,
    auto_histogram_activation_logging=True,
)

# Your code here...

For more information on getting started, see details on the Comet config file.

Keras Auto-logging Controls

The Comet Keras auto-logger can automatically log:

  • model/graph description
  • steps and epochs
  • metrics (such as loss and accuracy)
  • hyperparameters
  • optimizer config
  • number of trainable parameters
  • histograms for weights and biases
  • histograms for activations
  • histograms for gradients

Each of these can be controlled through an experiment parameter, environment variable, or a configuration setting:

Item Experiment Parameter Environment Setting Configuration Setting
model/graph description log_graph COMET_AUTO_LOG_GRAPH comet.auto_log.graph
metrics auto_metric_logging COMET_AUTO_LOG_METRICS comet.auto_log.metrics
metric logging rate auto_metric_step_rate COMET_AUTO_LOG_METRIC_STEP_RATE comet.auto_log.metric_step_rate
hyperparameters auto_param_logging COMET_AUTO_LOG_PARAMETERS comet.auto_log.parameters
command-line arguments parse_args COMET_AUTO_LOG_CLI_ARGUMENTS comet.auto_log.cli_arguments
weights/biases auto_histogram_weight_logging COMET_AUTO_LOG_HISTOGRAM_WEIGHTS comet.auto_log.histogram_weights
gradients auto_histogram_gradient_logging COMET_AUTO_LOG_HISTOGRAM_GRADIENTS comet.auto_log.histogram_gradients
activations auto_histogram_activation_logging COMET_AUTO_LOG_HISTOGRAM_ACTIVATIONS comet.auto_log.histogram_activations
histogram logging rate auto_histogram_epoch_rate COMET_AUTO_LOG_HISTOGRAM_EPOCH_RATE comet.auto_log.histogram_epoch_rate

In addition, you can control exactly which inputs at which layers are logged for activations and gradient histogram logging.

Histograms

With the keras auto-logger, you can log histograms for:

  • weights and biases
  • gradients
  • activations

These histograms appear on the Histogram Tab in the user interface:

The histograms for weights and biases are logged for all layers before training, and for every auto_histogram_epoch_rate epochs (default 1).

The histogram activations are logged for the selected inputs/targets (comet.keras.histogram_activation_index_list with a default of the first pattern), at the selected layers (comet.keras.histogram_activation_layer_list with a default of the last layer).

For activation histograms:

Item Configuration Setting
which inputs comet.keras.histogram_activation_index_list
which layers comet.keras.histogram_activation_layer_list

Info

To auto-log activations, you must be using TensorFlow, version 1.13 or greater. Inputs must be in a tensor or numpy format (e.g., DataSets are not yet supported). Note that logging activations from an input layer does not work with TensorFlow version 1.

The histogram gradients of the weights/biases with respect to the loss are logged for the selected inputs/targets (comet.keras.histogram_gradient_index_list with a default of the first pattern), at the selected layers (comet.keras.histogram_gradient_layer_list with a default of the last layer).

For gradient histograms:

Item Configuration Setting
which inputs comet.keras.histogram_gradient_index_list
which layers comet.keras.histogram_gradient_layer_list

Info

To auto-log gradients, you must be using TensorFlow, version 2.2.1 or greater, or Keras 2.4.3 and greater. Inputs and targets must be in a tensor or numpy format (e.g., DataSets are not yet supported).

Each of input of the index lists is composed of an empty string, or a list of index positions. If more than one index is given, separate the values with a comma.

For example, to indicate that you would like to log the activations for the first three input patterns, use:

export COMET_KERAS_HISTOGRAM_ACTIVATION_INDEX_LIST="0,1,2"

Each of layer list is composed of a list of index positions, negative index positions, or layer names. If more than one value is given, separate them with a comma.

For example, to indicate that you would like to log the gradients at the first layer, last layer, and a layer named "hidden", use:

export COMET_KERAS_HISTOGRAM_GRADIENT_LAYER_LIST="0,-1,hidden"

Each of the histograms logged can have a "prefix" attached to the histogram name in the UI. The prefix is set via the configuration setting comet.keras.histogram_name_prefix and can be composed of any of the following components:

Histogram name prefix component Meaning
{model_name} Name of the keras model
{layer_name} Name of the keras layer
{layer_num} Number of the keras layer
{max_digits} Number of digits in last keras layer

By default, the prefix is {layer_num:0{max_digits}d} (using Python's formatting syntax) which would give prefixes like:

01/input/kernel:0
05/hidden/bias:0
10/output/activation:all

Finally, you can control the following items as well:

Item Meaning Default value
comet.keras.histogram_batch_size number of patterns to process at once when logging all 1000
comet.logging.metrics_ignore names of metrics to ignore keras:batch_size,keras:batch_batch
comet.logging.parameters_ignore names of parameters to ignore keras:verbose,keras:do_validation,keras:validation_steps

For more details on passing in parameters to the experiment, please see Experiment(). For more details on setting up configuration variables, see Python Configuration.

The Comet.ml Keras callback

Comet.ml logs your experiment through a callback executed when you run model.fit() in Keras. You do not need to add this callback yourself, we do it for you automatically. However, if you ever need to access the callback manually, simply call Experiment.get_keras_callback().

How to report manually

You can log additional parameters beyond what Comet.ml automatically collects using Experiment.log_parameter().

from comet_ml import Experiment

#create an experiment
experiment = Experiment(
    project_name='mnist',
)
import keras
batch_size = 128

experiment.log_parameter("batch_size", 128)

You can log an entirely customized list of parameters to your experiment by using Experiment.log_parameters().

from comet_ml import Experiment
from keras.models import Sequential
from keras.layers import Dense, Dropout

experiment = Experiment(
    project_name="my project name",
    auto_param_logging=False,
)
batch_size = 128
num_classes = 10
epochs = 20

params={
    "batch_size":batch_size,
    "epochs":epochs,
    "num_classes":num_classes}

experiment.log_parameters(params)

Context Manager (Train/Test/Validate)

You can also log specific metrics to training and test contexts with our context managers Experiment.train(), Experiment.validate() and Experiment.test()

from comet_ml import Experiment
from keras.models import Sequential
from keras.layers import Dense, Dropout

experiment = Experiment(
    project_name="my project name",
    auto_param_logging=True,
    auto_histogram_weight_logging=True,
    auto_histogram_gradient_logging=True,
    auto_histogram_activation_logging=True,
)
batch_size = 128
num_classes = 10
epochs = 20

params={
    "batch_size":batch_size,
    "epochs":epochs,
    "num_classes":num_classes}

experiment.log_parameters(params)

# define model here

with experiment.train():
    history = model.fit(x_train, y_train,
                        batch_size=batch_size,
                        epochs=epochs,
                        verbose=1,
                        callbacks=[EarlyStopping(monitor='loss', min_delta=1e-4, patience=3, verbose=1, mode='auto')])

with experiment.test():
    loss, accuracy = model.evaluate(x_test, y_test)
    print(loss, accuracy)
    metrics = {
      'loss':loss,
      'accuracy':accuracy
    }
    experiment.log_metrics(metrics)

End-to-end example

Here is a simple end-to-end Keras example which uses a Dense NN on the MNIST dataset.

For more examples using Keras, see our Comet Examples Github repository.

from comet_ml import Experiment

#create an experiment with your api key
experiment = Experiment(
    project_name='mnist',
    auto_param_logging=False,
    auto_histogram_weight_logging=True,
    auto_histogram_gradient_logging=True,
    auto_histogram_activation_logging=True,
)

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.callbacks import EarlyStopping

batch_size = 128
num_classes = 10
epochs = 20
num_nodes = 64
optimizer = 'adam'
activation = 'relu'

# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

#these will all get logged
params={'batch_size':batch_size,
        'epochs':epochs,
        'layer1_type':'Dense',
        'layer1_num_nodes':num_nodes,
        'layer1_activation':activation,
        'optimizer':optimizer
}
model = Sequential()
model.add(Dense(num_nodes, activation='relu', input_shape=(784,)))
#model.add(Dense(256, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

#print model.summary() to preserve automatically in `Output` tab
print(model.summary())

model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

#will log metrics with the prefix 'train_'
with experiment.train():
    history = model.fit(x_train, y_train,
                        batch_size=batch_size,
                        epochs=epochs,
                        verbose=1,
                        validation_data=(x_test, y_test),
                        callbacks=[EarlyStopping(monitor='val_loss', min_delta=1e-4,patience=3, verbose=1, mode='auto')])

#will log metrics with the prefix 'test_'
with experiment.test():
    loss, accuracy = model.evaluate(x_test, y_test)
    metrics = {
        'loss':loss,
        'accuracy':accuracy
    }
    experiment.log_metrics(metrics)

experiment.log_parameters(params)
experiment.log_dataset_hash(x_train) #creates and logs a hash of your data

For additional examples, please see: