Is it feasible to build a whole branch of cryptography on a family of pseudo-random sources?
In theory, yes. If there was an efficient and Cryptographically Secure Pseudo Random Number Generator built from a chaotic system, then that could serve as the foundation of reasonably practical symmetric cryptography, and even signature.
Problem is we know no such thing. PRNGs built from a chaotic system, and having an even mildly convincing security argument¹, pale in efficiency compared to a modern CSPRNG² (unless we extend the definition of "chaotic system" well beyond the usual iterated continuous functions over $\mathbb R$, or discrete approximations thereof).
A (Cryptographically Secure) Pseudo Random Number Generator, in it's modern definition, is a powerful-enough tool to build all the other symmetric cryptography functionalities: CPA and CCA(2)-secure cipher, block cipher, Message Authentication Code, authenticated encryption, hash… Some examples:
- A cipher can be constructed from a (CS)PRNG by using the key and a truly random random IV to seed the PRNG, and constructing the ciphertext by XOR of the PRNG's output with the plaintext³. The security directly follows from that of the PRNG, and that's such a good and common way to construct a cipher that it has a name: stream cipher.
- A block cipher can be constructed from a PRNG as a Feistel cipher, by using the PRNG to construct the round functions. The key, round number, and right half of the block seed the PRNG, which output is the value to XOR with the left half.
These constructions are demonstrably cryptographically secure if the PRNG is. But with the exception of stream ciphers, they are not used in practice, primarily for efficiency reasons. Common CSPRNG are constructed from block ciphers or/and hashes, rather than the other way around.
Is it feasible to build public key crypto primitives from PRNG?
Yes for signature. We can build secure hash from secure PRNG, then secure signature from hash, by various approaches, including SPHINCS. By this route, any efficient PRNG leads to a plausible signature scheme.
For encryption and key exchange, I doubt that a method with a security proof or even a convincing argument is known. I stand unconvinced by attempts to build asymmetric encryption from continuous chaotics systems by more direct routes⁴.
¹ We can't ask for a proof in the mathematical sense, since we have no such proof of security for any CPRNG. But we do not want to accept as security argument the fact of passing a predefined randomness test, such as NIST SP800-22rev1a or dieharder. An experimental test should be at least: impossibility for skilled human cryptographers knowing the design of the PRNG, assisted by classical computers, to distinguish from true randomness the output of the PRNG seeded with true randomness. And we'd want to extend that to such impossibility starting with a minimum value of some parameter(s) of the PRNG, like state size or/and number of rounds, with the actually used parameter(s) set comfortably larger.
² Such as the one derived from ChaCha by considering the key and IV to be the seed, and all-zero plaintext.
³ Decryption is similar with plaintext and ciphertext exchanged, except that encryption draws the IV and makes it a preamble of the ciphertext, while decryption extracts the IV from the preamble.
⁴ One such attempt (paywalled) uses Chebyshev polynomials $T_r$ of large degree $r$. A private key is $r$, the matching public key is $T_r(x)$ for some public fixed randomly-chosen real $x\in[-1,1]$. For any integer $s>0$ it holds $T_r(T_s(x))=T_s(T_r(x))$ and (ignoring issues of how that's computed) that allows an analog of Diffie-Hellman key exchange, and from that ElGamal encryption. When I first read it, I stood unconvinced by the argumentation-free assertion of security, as well as by some aspects of feasibility (e.g. that with 2048 bits of precision for real values, the integers $r$ and $s$ can be chosen as random 910-bit integers, rather than as product of primes at most 133 as in the article).
Update: The cryptosystem was found to be insecure, see this article (paywalled). It's still presented in this chapter on Public-Key Cryptography (paywalled) in a much later book on Chaos-Based Cryptography (paywalled), with acknowledgement of insecurity. I find it telling of the state of that whole academic field. And that's at it's best: most claims of security, made with comparably weak arguments, are never seriously investigated and proven wrong.
