Tamper-proofing log files

Question

Problem Overview

I want to securely store log files so the contents are secret, and they can't be modified without detection.

The files will be encrypted using authenticated encryption (AES in GCM mode), with a random IV and symmetric key for each file. The symmetric key will be encrypted using the public part of an RSA key pair. Both the IV and encrypred symmetric key will be included in the additional authenticated data.

This gives me confidentiality, integrity and authenticity - but only for each individual log file.

For example, let's say I have log files 2013-01-01.log, 2013-01-05.log and 2013-02-09.log - an attacker could delete 2013-01-05.log without detection.

I've come up with 2 possible solutions.

Possible Solution 1

The program could maintain an encrypted (and possible RSA-signed) 'counter file', which would contain a sequence number that would be incremented every time we write a new log file. The sequence number would become part of the log filename, and would also be included in the additional authenticated data. We could therefore detect any 'gaps' from missing files.

Possible Solution 2

The program could maintain an encrypted (and possible RSA-signed) 'database file', which would contain the filenames of all previously written log files. We could therefore detect any 'gaps' from missing files.

The Question

I'd like feedback on my 2 possible solutions - do they work, do they need some changed, have I missed anything?

Or are there better solutions to my problem?

Answer 1

"Efficient, Compromise Resilient and Append-only
Cryptographic Schemes for Secure Audit Logging" ​ (PDF)
gives a publicly verifiable approach that allows fine-grained verification,
but it is in the Random Oracle Model.

The Simple Method:
The verifier and logger start with a seed for a
forward-secure pseudo-random number generator.
To denote a valid ending of a log, put the string of
the next $b$ bits of the PRNG's output into the log.
To add a log entry, get the next $\:b+k\:$ bits of the PRNG's output,
put into the log the encryption of the log entry and the mac of that
ciphertext using the last $k$ of the $\:b+k\:$ bits of PRNG output as the
mac key, then erase those $\:b+k\:$ bits and the previous PRNG state.

Answer 2

Take a printer, and have the log file come out of the machine on paper. Ensure fire doesn't exist near the paper. Anything else will not work: if an attacker can wind back time log files can die and you cannot tell. All techniques for assuring time cannot be run backwards amount to doing this in some form, perhaps by sending data to another computer. But if you can do that, just send the log files and the attacker cannot even delete them!

Answer 3

Edit: You've clarified in the comments that confidentiality of the logfile's contents is important.

Given an AEAD function $C = E_k(iv, plaintext, aad)$, a safe construct is

$$ C = E_k(iv, contents, filename). $$

There is no need to include either the key or the IV in the additional authenticated data, and I would recommend against doing so. It is plausible that a particular AEAD scheme could leak the contents of the authenticated data, since this is not generally a design requirement.

This scheme will allow you to detect manipulation of the contents of a file, file renaming, and file deletion. You can detect logfile deletion simply by the lack of presence of a file for a particular date. If you have no log data for a date, simply generate an empty file and encrypt it with this scheme; the filename's inclusion in the authenticated data will prevent an attacker from being able to "replay" an encrypted empty file for other dates.

You may additionally want to use something like chattr to set the system-wide immutable flag for your logfiles. This will prevent any user from being able to modify or delete files, without changing the flag as root and rebooting the system first. Similarly, you should consider setting the append-only flag for live logfiles that are still being written to. This will help prevent tampering with them before they can be permanently archived.

Finally, permanent storage media are great solutions to this problem as well. If your need is great enough, burn the signed logfiles to a DVD-R periodically.

Answer 4

I'm thinking there's a third potential solution. Each time you close a log file, you could append the name of the next new log file, timestamp it, then sign the log file. When it is time to create a new log file, you would read the previous log file, validate the signature, validate the time stamp, read the new log file name, and create it. You'd kickstart the whole thing by self-signing the first empty log file.

This permits you to decrypt each file on its own, which you probably do frequently for ordinary troubleshooting and maintenance activities. When you need to audit the log files, which is probably a less common annual activity, you would walk the chain of all files ensuring that all signatures are valid and that none are missing.

This isn't a perfect or complete solution, of course. Off the cuff, I think a bad guy could tamper with the system clock, setting it back to the end of two log files ago and delete the most recent log file. But clock tampering might leave other evidence in other log files. You should still export the log files to a separate secured server as soon as you close them.

Answer 5

I've been toying around with a solution to this problem using blockchain tech.

Say for every client that wants to make a server request, they have to first record their request on a blockchain. Then that client would submit that same request to your server.

Your server would check that the received request also exists on the blockchain before processing the request.

Responses would work the same way but in reverse. The server would publish a response on the blockchain and sent a normal response to the client. The client could then check to make sure that the response from the server also exists on the blockchain.

This is probably way overkill, but it's an interesting concept.