What is the recommended format/notation for crypto design?

Question

Is there a recommended format or formal notation for documenting the combination of symmetric and asymmetric encryption, key derivation and other algorithms (and their inputs and outputs) that I think meet the needs of the software I'm contributing to the design of?

At the moment we're blithely using statements like "Encrypt with AES, reusing K₁ with a fresh IV" - but of course that statement misses out details such as mode of operation, that I would certainly realise later, but wouldn't have initially written down in the design.

My goal is to document the design in a way that can be understood by other developers (with maybe a similar or better skill level), and to show all the system states, the movements between them and the encrypted/signed/etc messages that cause that - and perhaps eventually to use in a request for outside review.

As background, I have done a single undergrad cryptography course - I imagine this is the kind of thing covered later on, or in a more focussed degree?

Answer 1

TL;DR: Find the most important specification that is of the same type as the one you want to write and use its style for yourself, chances are, other people also have read it.

There is a myriad of ways to specify a crypto protocol / design.

However, there are four things that you really should take into consideration when writing a crypto-related specification.

1. The most important one first: Fancy pictures. A good picture (as in: a colorful graphic or even ASCII art) says more than a thousand words and can greatly help people quickly understand the essentials of what your protocol is all about. Examples of ASCII art can be found in the actual TLS specification and fancy picture can be found outside of that.
2. Data Encoding: ASN.1. If you want to submit data with your protocol, you need some structure within it and for notating structured data, nothing beats ASN.1 in terms of being deployed and well-understood. You should really consider using ASN.1 for specifying data structures, to which you only need to apply primitives afterwards (like "sign it!"). Don't forget to define encoding rules (like BER and DER) though. An example for an extensive use of ASN.1 would be CMS, the basis of S/MIME.
3. Protocol summary: The Handbook of Applied Cryptography (HAC) style. While fancy pictures are valueable, a written out summary is also very valueable and I in my experience the HAC does a really good job at writing them down. It consists of four parts: a) write down the TL;DR of the protocol, e.g. what does it do? b) write down the setup phase of the protocol c) write down the protocol messages as can be seen on the wire (encoding is implicit here, possibly using $\LaTeX$) and d) write down the protocol actions happening between each step. This summary should give everybody a good oversight, of what is done when and the notation also allows you to associate keys, IVs and other things with specific function calls. As a bonus for this method: Many serious cryptographers have read the HAC or have a copy of it very close at all times. Protocol 12.20 is an example for this style (PDF).
4. The details: The TLS specification style. When you want somebody to review / implement your full specification, this person likely has some experience in doing such things and usually usually has read the TLS specification at some time before, so using its notation is a no-brainer. Additionally, TLS also deals with a bunch of keys and the specification is clear enough to distinguish what to use when, by using things like digitally-signed struct.