What if the validation step does not fit into numbers of samples?

Question

It's a bit annoying that tf.keras generator still faces this issue, unlike pytorch. There are many discussions regarding this, however, still stuck with it. Already visit:

Problem

I have a data set consist of around 21397. I wrote a custom data loader which returns the total number of samples as follows:

class NGenerator(Sequence):
  ...

    def __len__(self):
        return int(np.ceil(float(len(self.data)) / float(self.batch_size)))
  ...

From the data, I've made 5 fold subset of it. Each fold contains as follows:

Fold   Amount
1.0    4280
0.0    4280
2.0    4279
3.0    4279
4.0    4279

For each fold, I've set step_per_epoch and validation_per_epoch as follows:

# here, train_labels is the df of the subset based on fold
steps_per_epoch  = np.ceil(float(len(train_labels)) / float(batch_size)) 
validation_steps = np.ceil(float(len(val_labels)) / float(batch_size))

Now, to make an OOF score, we predict on the validation set and wanted to store results as follows:

batch_size = 64
oof = np.zeros(len(df))
...
for each_fold, (trn_idx, val_idx) in enumerate(skf...): 
   train_labels = df.iloc[self.trn_idx].reset_index(drop=True) 
   val_labels = df.iloc[self.val_idx].reset_index(drop=True) 
   ....
   train_gen, val_gen = ..., model.fit()

   pred = model.predict(val_gen, steps=validation_steps)
   oof[self.val_idx] = np.argmax(pred, axis=1)  < --------- HERE

After training, at indexing time (oof), it throws a size mismatch of shape between 4280 and 4288. So, it looks like, with this step size and batch size, the model is predicting 8 samples of the next batch. Next, we set batch_size equal to 40 which dividable by the total number of the subset (4280). Good enough but (of course) faced again size mismatch in Fold 2 of shape between 4279 and 4280. One of the simple workarounds is to add 3 samples in fold 2,3,4 -_-

Any general tips to get rid of it? Thanks.

score 1 · Accepted Answer · edited Nov 28 '20 at 04:02

Did not have time to go through all your code however I thought the code below might be useful to you. The variable-length should be set to the number of samples. Then the code determines a batch size and steps per epoch such that length = batch_size*steps per epoch. Variable b_max should be set to the maximum batch size you will allow based on memory capacity. Note if the length is a prime number batch size will end up as 1 and steps will end up as length.

def get_bs(length, b_max):
   batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and 
   length/n<=b_max],reverse=True)[0]  
   steps=(length/batch_size)
   return batch_size, steps

I use this to set validation steps so during validation the samples in the validation set are processed exactly once. An example is shown below.

batch_size, steps = get_bs(2048, 90)
print ('batch_size = ', batch_size, '   steps = ', steps)
# result is batch_size =  64    steps =  32.0

What if the validation step does not fit into numbers of samples?

Problem

1 Answers1