9

I'm trying to train a model in keras and I'm using ModelCheckpoint to save the best model according to a monitored validation metric (in my case the Jaccard index).

While I can see the model improving in tensorboard, when I try to load the weights and evaluate the model it isn't working at all. Furthermore, by the timestamp on the file where the weights are supposed to be stored, I can tell that they are not being saved at all. The timestamp corresponds roughly to the time I started training.

Has anyone encountered such a problem before?

Ethan
  • 1,657
  • 9
  • 25
  • 39
ILM91
  • 338
  • 1
  • 7

1 Answers1

8

Do you run ModelCheckpoint on its default parameters (besides monitor)?

ModelCheckpoint has a parameter called mode which specifies the type of metric to be used. mode can take 3 values 'min' 'max' and 'auto' (which is the default):

  • min: means that you want to minimize the metric (e.g. the loss function).
  • max: means you want to maximize the metric (e.g. accuracy).
  • auto: attempts to figure what to do on its own. If you look at the code, it checks if the metric's name contains 'acc' or if it starts with 'fmeasure'. If yes it sets the mode to max, if not it sets it to min.

In your case, you monitor the jaccard index, which is a metric you want maximized, so you want the mode set to max. Normally because "jaccard" contains the string "acc", even if the mode is set to auto it should work fine.

If however you named your metric something arbitrary (e.g. my_metric), the default mode will be set to min, which means that it will store the weights that achieve the least performance on your metric, which should be the weights of the first epoch.

Suggestion: next time try with mode='max' to be sure.

MzdR
  • 280
  • 1
  • 4