Neural Networks - Loss and Accuracy correlation

Question

I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the application of the two redundant in the training epochs? Moreover, why aren't they correlate?

Answer 1

Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due to the backpropagation algorithm that requires the loss function to be differentiable. When your preferred loss is not directly optimizable (like the accuracy) you use a loss function that behaves similarly to proxy the true metric. In case of binary classification you would use a sigmoid at the end and a log loss to approximate accuracy. They are highly correlated.

Answer 2

Yes, they both measure the exactness of y and y_hat and yes they're usually correlated. Sometimes the loss function might not be accuracy but you're still interested in measuring the accuracy even though you're not optimizing it directly. Google's TensorFlow MNIST example minimizes/optimizes cross entropy loss but displays accuracy to the user when reporting results, and this is completely fine.

Sometimes you don't want to optimize accuracy directly. For example, if you have serious class imbalance, your model will maximize accuracy by simply always picking the most common class, but this would not be a useful model. In this case entropy / log-loss would be a better loss function to optimize.