The arbitrary constant of step 3 should be 1 to 119 bytes (or 1 to 959 bits). If it was empty, a trivial attack would predict 512-bit outputs following the first one after a reseeding. If it was larger, there would be extra SHA-512 rounds making the generator slower, and as an aside there would be even so slightly less entropy in each 512-bit output block.
With such parameter, and if initialized with enough entropy at step 1, and if it's internal state does not leak or become corrupted, then there is no known attack on this (hybrid) random number generator.
Even under these hypothesis, it has a number of theoretical drawbacks:
- When generating many consecutive outputs, the $S \leftarrow \text{SHA512}(S)$ steps tends to reduce the entropy in $S$, like to about $512-\ln(2+k)$ bit of entropy after $k$ previous iterated hashes (see Entropy when iterating cryptographic hash functions). While this is not a practical issue at all, this can be viewed as unfortunate. A common way to fix that non-issue, and make the generator twice faster in step 3, is to reduce step 3 to: $\mathsf{output} \leftarrow \operatorname{SHA-512}(S\mathbin\|\underline k)$ where $k$ is an incremental counter of output blocks since the last reseeding (or just since step 1), and $\underline k$ is a representation of $k$ as bytestring per some convention (like, big-endian binary over 64 bits). A similar technique is used in MGF1 (with 4 bytes, but 8 is more natural in the context of SHA-512).
- We are using SHA-512 outside of it's initial stated design goals of resistance to collision and (first and second) preimage. These goals do not imply that for random input, the output is indistinguishable from random, which we assume. While there is no known attack against that property of SHA-512, we may have better assurance if we use $\operatorname{HMAC-SHA-512}(S,X)$ (where the first input is the key parameter of HMAC) instead of $\operatorname{SHA-512}(S\mathbin\|X)$ at steps 2 and 3. PBKDF2 essentially does that.
Importantly, this is lacking at least one most critical feature: something giving assurance that the generator is effectively seeded with actual entropy at step 1 (including but not limited to: step 1 is actually performed before any successful execution of step 3). Such failures do occur! Sometime that's because the amount of actual entropy has been overestimated (causing things ranging from output occasionally identical across reboots or different instances of the same generator on different machines, real or virtual; to keys that can be found by enumeration by an attacker aware of the generator and how it's entropy source works). That can be an accidental hardware defect. Or perhaps an adversary deliberately blows a liquefied gas to literally freeze what gathers entropy.
As a relatively minor aside, there is nothing stated to insure timely reseeding. OTOH, if a reseeding is necessary, we are next to disaster, and there is no assurance that reseeding will advert it.
Also: risk of disclosure by some side channel are not considered. Not that it's particularly likely with SHA-512.
And of course, this ignores more complex and occasionally overkill prescriptions in NISTÂ 800-90 or AIS31. Which can be a problem when it comes to get rubber-stamped.