How to define custom cost function that depends on input when using ImageDataGenerator in Keras?

Question

I would like to define a custom cost function

def custom_objective(y_true, y_pred):
    ....
    return L

that will depend not only on y_true and y_pred, but on some feature of the corresponding x that produced y_pred. The only way I can think of doing this is to "hide" the relevant features in y_true, so that y_true = [usual_y_true, relevant_x_features], or something like that.

There are two main problems I am having with implementing this:

1) Changing the shape of y_true means I need to pad y_pred with some garbage so that their shapes are the same. I can do this by modyfing the last layer of my model

2) I used data augmentation like so:

datagen = ImageDataGenerator(preprocessing_function=my_augmenter)

where my_augmenter() is the function that should also give me the relevant x features to use in custom_objective() above. However, training with

model.fit_generator(datagen.flow(x_train, y_train, batch_size=1), ...)

doesn't seem to give me access to the features calculated with my_augmenter.

I suppose I could hide the features in the augmented x_train, copy them right away in my model setup, and then feed them directly into y_true or something like that, but surely there must be a better way to do this?

Answer 1

Maybe you could create a two part model with:

• Inner model: original model that predicts desired outputs
• Outer model:
  • Takes y_true data as inputs
  • Takes features as inputs
  • Outputs the loss itself (instead of predicted data)

So, suppose you already have the originalModel defined. Let's define the outer model.

#this model has three inputs:
originalInputs = originalModel.input  
yTrueInputs = Input(shape_of_y_train)
featureInputs = Input(shape_of_features)

#the original outputs will become an input for a custom loss layer
originalOutputs = originalModel.output

#this layer contains our custom loss
loss = Lambda(innerLoss)([originalOutputs, yTrueInputs, featureInputs])

#outer model
outerModel = Model([originalInputs, yTrueInputs, featureInputs], loss)

Now, our custom inner loss:

def innerLoss(x):
    y_pred = x[0] 
    y_true = x[1]
    features = x[2] 

    .... calculate and return loss here .... 

Now, for this model that already contains a custom loss "inside" it, we don't actually want a final loss function, but since keras demands it, we will use the final loss as just return y_pred:

def finalLoss(true,pred):
    return pred

This will allow us to train passing just a dummy y_true.

---

But of course, we also need a custom generator, otherwise we can't get the features.

Consider you already have originalGenerator =datagen.flow(x_train, y_train, batch_size=1) defined:

def customGenerator(originalGenerator):

    while True: #keras needs infinite generators
        x, y = next(originalGenerator)

        features = ____extract features here____(x)

        yield (x,y,features), y 
            #the last y will be a dummy output, necessary but not used

You could also, if you want the extra functionality of randomizing batch order and use multiprocessing, implement a class CustomGenerator(keras.utils.Sequence) following the same logic. The help page shows how.

So, let's compile and train the outer model (this also trains the inner model so you can use it later for predicting):

outerModel.compile(optimizer=..., loss=finalLoss)
outerModel.fit_generator(customGenerator(originalGenerator), batchesInOriginalGenerator, 
                         epochs=...)