While using Neural Networks (TensorFlow: Deep Neural Regressor), when increasing your training data size from a sample to the whole data (say a 10x larger dataset), what changes should you make to the model architecture (deeper/wider), learning rate and hyper parameters in general?
How much of trial and error how much of heuristic logic is involved in making these changes?
I don't think you ought to change much in the model definition.
You should, however, consider the amount of time it takes to train on the complete dataset. If it takes too long and you are still in the testing phase, you want to reduce the number of epochs to obtain results faster and make changes in the model accordingly.
I suggest plotting all the metrics and try to understand if the trend is positive or negative. If it's positive the changes you are doing are correct of course!
Then, once you are happy with the hyperparameters, put like epochs=100 and leave the model to train for a whole night, then plot again the learning curves and decide when to stop earlier, or use early_stopping.
The depth and width of your DNN are used to model the complexity and not the size of your data. So, if you are already in a situation where you have enough data to sufficiently train your model, increasing the size of the training data does not require you to change anything, except maybe reducing the number of epochs. For example, to model the data complexity of the MNIST dataset you will not need hundreds of layers, even if you would have billions of images to train on.
However, there is a situation in which increasing the depth and width can make sense: If you first did not have a lot of data, and therefore you created a small DNN to prevent overfitting which does not sufficiently model the complexity of your data, and then you get a huge amount of additional data, it makes sense to increase the depth and/or width of your DNN.
A good point is following rule:
Your network should be capable of overfitting on your training data. When you can not not overfit on your training-data you should increase your depth/width. But it is hard to say by how much, it is sometimes more an art than a science.
Of course it does not mean that you should overfit on your data.
