The identification of polynomial time with "feasible" is known as Cobham's thesis. As you've observed, it might not be accurate in theory, but in practice it serves as quite a good model.
Note that in cryptography you are concerned about the gap between the effort for the user of a cryptosystem and that of an attacker - you want the system to run quickly even on low-resources settings (like embedded systems), but you want security even against an adversary which has huge resources, e.g. months of computing time and a supercomputer.
In practice, this gap seems to be best realized by requiring a superpolynomial difference between the user's runtime and the attacker's. It could, in theory, be done by using a large gap in the polynomial exponent and taking a large security parameter. However, such gaps in current systems are mostly quadratic, as far as I know (see Boaz Barak, Mohammad Mahmoody-Ghidary, Merkle Puzzles are Optimal) and even if you set the security parameter to $2^{30}$ (which is very large for practical use cases), it isn't impractical for a dedicated attacker with large resources to run in time $2^{60}$.
In case of $P = NP$, letting the cryptosystem's user run in $EXP$ doesn't help, as $P = NP$ implies $EXP = NEXP$ via a padding argument, so the user still only has a polynomial advantage over an attacker.