In every stream cipher (where applicable) that I know of the counter is always separate from the nonce.
At least for counter mode many libraries actually leave that to the user; quite often counter mode (also known as CTR-, ICM- or SIC- mode) implementations require a full size block. So in that case the libraries leave it open to the user, and just perform increments for each counter, modulus two to-the-power-of the block size in bits.
What is the reason for why the counter is not just part of the nonce and addition is modulo 64?
It can be part of the nonce. It is probably better to say that the nonce part is part of the counter block though. Which is actually always the case unless it is entirely absent, i.e. there is one message, or the counter is kept as state for it to cover multiple messages.
The only reason I can think of it that it 'complicates' a proof or reduction because in such a setting a nonce will not provide the full security.
Yes, it does complicate calculations. One of the problem I see with generating an entirely random nonce of the same size as the counter block is that you don't know the distance to the next nonce, in which case chance will probably assign the largest message to the nonce with the least distance to the next nonce, which may produce overlap.
It is much easier to have one smaller random value for the nonce and a known maximum size. If the nonce is unique and the message is below the easy-to-determine message size then the scheme is secure.
But in practice it will in fact provide more security than the alternative: no one encrypts ~2^64 data and therefore nonce entropy is always left on the table.
First of all, you forgot the unit. If the message counter takes up half of the counter block then precisely $2^{68}$ bytes of data can be encrypted per message. However, most implementations simply use mod $2^{128}$ incrementation for the counter block as previously mentioned.
This means that:
- The nonce should be placed at the most significant part of the counter block, and in most implementations it's assumed to be big endian (so to the lower indices / to the left if it is a byte array).
- The size of the nonce is undetermined, so the nonce size can be determined to be the bits / bytes left after assigning enough space for the per-message counter in the least significant bits.
- The user of the library is required to keep track that the counter does not overflow.
- The maximum counter value two to-the-number-of bits reserved for it, and the data size in bytes is the maximum counter value times 16 (for a 128 bit block cipher obviously).
- The counter should of course start at zero because otherwise some counter values cannot be reached.
- The nonce could also be a unique message number or other non-overlapping number such as a generic sequence number.
Note that, regardless of this, there are users on this site that do prefer a fully random initial counter block even when used for different messages. Personally I prefer clearly defined boundaries between nonce and counter within the counter block.
In the end each counter block is required to be unique and any scheme that gets you there with reasonable certainty can be considered secure. Or, more formally, the chances of a counter block colliding should be negligible, often indicated as having an $\epsilon$ so small that it is irrelevant.
