9

I am training a CNN on some text data. The sentences are padded and embedded and fed to a CNN. The model architecture is:

model = Sequential()
model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
model.add(Conv1D(128, 5, activation='relu'))
model.add(GlobalMaxPooling1D())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())


model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())


model.add(Dense(25, activation = 'relu'))
#model.add(Dropout(0.2))
model.add(BatchNormalization())


model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

enter image description here

Any help would be appreciated.

Community
  • 1
NoLand'sMan
  • 215
  • 1
  • 3
  • 6

4 Answers4

5

Your model is completely overfitting. The training loss is constantly decreasing but the validation loss isn't. This means that the your current model is complex enough to 'memorize' the patterns in the training data. In such situations, you need to regularize your model.

To regularize your data you may try any or all combination of the following:

1) Reduce the layers of the neural network.

2) Reduce the number of neurons in each layer of the network to reduce the number of parameters.

3) Add dropout and tune its rate.

4) Use L2 normalisation on the parameter weights and tune the lambda value.

5) If possible add more data for training.

6) Adding more data may not always be economically viable option so 'Data Augmentation' can play a big role here. Data Augmentation depends on the type of input to the neural nets. For images they comprise of rotation, translation, affine transformation like shear etc.

user1825567
  • 1,416
  • 1
  • 14
  • 24
2

You are over-fitting to the train dataset and failing to generalize to the validation dataset.

One way to reduce overfitting is regularization. The most common methods for regularizing a neural network are:

  1. Add more data.
  2. Add dropout.
  3. Reduce the number of layers.
Brian Spiering
  • 23,131
  • 2
  • 29
  • 113
0

Another possible issue could stem from imputing missing values. Say you have cases where there are observations with completely missing values and different labels. If all of the data with missing values in train have label 0, and all of the data with missing values in val have label 1, the val loss for those observations will be astronomical.

I had this exact scenario happen to me.

Chris Coffee
  • 21
  • 2
0

Also a couple of things that proved valuable besides mentioned are batch normalisation and feewer deep layers. The less complexity the more generalisation power you can have.