OAEP security with variable length hash function

Question

I'm implementing a hobby cryptosystem for fun and to increase my knowledge on the subject, and I was wondering if the OAEP construct was still sufficient as an all-or-nothing-transform if variable length hash functions (specifically SHAKE256) are used for the $G$ and $H$ random oracles.

I already found a paper showing that OAEP was functional as an all-or-nothing-transform, but I'd like to use SHAKE256 as a hash function because it allows for arbitrary-length messages.

My current implementation is here. I pad the message to a minimum of 32 bytes, and then then my $k0$ length, or the length of the additional information added, is another 32 bytes.

I'm wondering if this use of SHAKE256 is theoretically secure, or if there is a problem with using a variable output hash function with OAEP. I'm not concerned with side channel attacks, this is a purely educational implementation.

Answer 1

Beware that if $n < m$, then $\operatorname{SHAKE256-}\!n(x)$ is a prefix of $\operatorname{SHAKE256-}\!m(x)$, so the two functions are not really independent random oracles as the usual OAEP theorems posit.

If you set $G(x) = \operatorname{SHAKE256}(0 \mathbin\| x)$ and $H(s) = \operatorname{SHAKE256}(1 \mathbin\| s)$, that should be adequate to (conjecturally) satisfy the hypotheses of the theorems without requiring additional analysis to study the possibility of collisions between the inputs to $G$ and $H$.

Alternatively, if the inputs to $G$ and $H$ are guaranteed to have distinct lengths in your application, then $G(x) = \operatorname{SHAKE256}(x)$ and $H(s) = \operatorname{SHAKE256}(s)$ should work too. But it won't hurt, and might be safer to avoid mistakes, if you always use a unique prefix, whether it be a 0 bit vs. a 1 bit, or the string G oracle vs. H oracle, etc.