How were the best AES whiteboxes in the CHES 2017 CTF challenge built and broken?

Question

Along CHES 2017 was a Capture the Flag Challenge, aka The WhibOx Contest.

Participants submit C source code with interface

void AES_128_encrypt(unsigned char ciphertext[16], unsigned char plaintext[16]);

that performs AES-128 encryption with some key. Other participants attempt to find that key from the submission. A mechanical referee checks if a guess of the key is correct. A rating system awards points (strawberries and bananas) as a function of how much time one's submission remains unbroken, and for breaking a submission. Survival range has been from 1 minute to a record 28 days 13h 42'.

What where the techniques used by the leading participants, both for making whiteboxes, and attacking them? The only clear thing is that there was an inflation in size; the winner is 28MByte.

Answer 1

Junwei Wang gave a talk on how the winning entry (#777 by cryptolux) was designed and how team cryptoexperts broke it. Slides, video.

The overvew of the design (slide 9) is:

• Multi-layer protection
  • Inner: encoded Boolean circuit with error detection
  • Middle: bitslicing
  • Outer: virtualization, randomly naming, duplications, dummy operations
• Code size: ~28 MB
• Code lines: ∼2.3k
• 12 global variables including:
  • computation state (2.1 MB)
  • program bytecode (15.3 MB)

And the overview of their approach to reversing it:

1. Reverse engineering ⇒ a Boolean circuit
2. Single static assignment (SSA) transformation
3. Circuit minimization
4. Data dependency analysis
5. Key recovery with algebraic analysis

Answer 2

I am from the winning attacking team: team_cryptoexperts. Recently, we released a white paper on our white-box attacking techniques 2018/098. As @Michael suggested, you can also find some expressive detail from my talk at RWC 2018 and the slides.

By the way, some of the protection techniques can be found from 2018/049.