5

I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights :

weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

then, in my network:

model.add(LSTM(..., class_weight=weights,...,callbacks=callbacks_list))

However, I also use a callbacks list to prevent overfitting, by imposing EarlyStopping based on validation_accuracy ...:

earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5, verbose=1, mode='auto')
callbacks_list = [earlystop]

However, my weights are only defined based on my training set... But my validation set also contains imbalanced data, in different proportions than the training set, and I would also like to give appropriate weights to have a fair evaluation of val_loss parameter...

So my question is

  1. With which weights is the val_loss computed ? With the weights given in class_weightparameter ? Or is this parameter only used for training loss ?

  2. How could I define weights that could be used for the validation set, to have a more accurate value of val_loss ?

Thanks

Valentin Calomme
  • 6,256
  • 3
  • 23
  • 54
MysteryGuy
  • 190
  • 7

2 Answers2

1

You are not supposed to apply weights to your validation set since it is supposed to measure your model's performance. If you'll do that you'll probably get better results for validation but once your model is deployed it will perform worse on new data. Weighting, resampling techniques etc. - they all should be done on the training set only!

Corel
  • 159
  • 4
0

As explained here you can define your custom metric. You only need to use validation set weight for calculating the score.

def my_metric(y_true, y_pred):
     sw=compute_sample_weight('balanced',y_true)
     return accuracy_score(y_true, y_pred, sample_weight=sw)
Mehdi
  • 344
  • 1
  • 6