If the output of an algorithm when interacting with the [encryption] protocol matches that of a simulator given some inputs, it ‘need not know’ anything more than those inputs.
There is no evidence of deep learning breaking modern cryptography. Deep learning is simply glorified gradient descent. With a reasonable cipher you get no indication of almost finding the key, so I see no hope of deep learning breaking a black box cipher.
In order to use deep learning for cryptography we would need to find a notion of gradually or partially solving the problem, not an easy task.
We have seen some work to use deep learning to build new ciphers, using adversarial training. But there is not much evidence these ciphers are secure against human cryptanalysis.
