How Far Ahead of Academia Are Government Agencies?

Question

This is a soft question regarding comparisons between government security services (eg, NSA or GCHQ) and open-source research (e.g., academia). Hopefully it's on-topic for this site!

In essence, my question is the following.

How far ahead (if at all) of of open source (specifically academic) research are governmental agencies?

There are some various thoughts I've had on this matter.

• NSA is the largest employer of mathematicians worldwide, and their primary focus is on security. So not only do they have an incredibly large number of people, but they're also very focused. On the other hand, cryptography departments at universities (that I'm aware of) don't seem to be particularly large.

• We now get onto the actual people who do the work. For example, to start at the base level at GCHQ you need a 2:1 in a three-year maths degree. (I'm from the UK, and don't really know any comparison with American GPA.) In comparison with university faculties, these will all have doctorates as a minimum, and university researcher places (at least in UK universities I'm aware of) are very competitive.

• At a first glance, this suggests that the standard of researchers in general is much higher in academia. Of course, one has to be careful: not everyone wants to do a PhD for starters; moreover, a lot of people may not want the job-insecurity of post-docs, and prefer to move into a more secure (likely higher-paying) job outside of academia. Furthermore, continuing from the first point, these people then spend all their time working on this stuff, with large amounts of resources (including many other people around).

• As an example, when DES was replaced with AES, instead of developing AES themselves, the NIST set a challenge to other researchers. The NIST then chose the best of these (supposedly, anyway; many people have apparently legitimate claims that the most secure wasn't chosen, but instead the one the the NIST 'liked' the best).

• I have heard that RSA was discovered some 8 (or maybe 3?) years before it was public, but that this was at a time when there was comparatively little open source research into such topics.

• Finally, a lot of what secret agencies do is, well, secret!

Basically, there seem to be arguments for many points, and I was wondering if anyone has more insight than I do!

Very related (but not the same) is this question: How can we reason about the cryptographic capabilities of code-breaking agencies like the NSA or GCHQ?; that question is more about some specific things (eg symmetric ciphers).

---

To be clear, I know this question isn't definitively answerable without high-up knowledge of governmental affairs, but I feel some approximations should be possible.

Of course, none of this addresses private companies (eg, Google). These employ a lot of people, likely(?) with higher entry requirements than NSA/GCHQ (see above on why that doesn't give a clear cut implication) and also have lots of money; moreover, there's likely(?) less bureaucracy.

---

If any of my above claims are wrong, please do point them out! I'm not very knowledgeable in the world of cryptography. I'm a maths PhD (discrete probability), but have interest in this stuff.

Answer 1

Disclaimer: This post is possibly opinion based.

How far ahead (if at all) are governmental agencies of open source (specifically academic) research?

This question is impossible to answer. By definition, this require a knowledge of the state of research in such agencies which is definitively something they want to be kept as secret...

thx to @fkraiem:

However one can get a feeling of it by considering the followings:

• Academic research is usually founded by companies or government agencies.

  e.g. for the Gimli paper:

  This work was supported in part by the Commission of the European Communities through the Horizon 2020 program under project number 645622 (PQCRYPTO) and project number 645421 (ECRYPT-CSA); the Austrian Science Fund (FWF) under grant P26494-N15; the ARC project NANOSEC; the Belgian Fund for Scientific Research (FNRS-F.R.S.); the Technology Foundation STW (project 13499 TYPHOON), from the Dutch government; the Netherlands Organisation for Scientific Research (NWO) under grant 639.073.005; and the U.S. National Science Foundation under grant 1314919. “Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

• In the case of important projects, academic research is paired with companies with huge computing power (Google, Microsoft Research...) e.g. SHAttered:

  This result is the product of a long term collaboration between the Cryptology Group at Centrum Wiskunde & Informatica (CWI) - the national research institute for mathematics and computer science in the Netherlands - and the Google Research Security, Privacy and Anti-abuse Group. Two years ago Marc Stevens and Elie Bursztein, who leads the Google's anti-abuse research team, began collaborating on making Marc's cryptanalytic attacks against SHA-1 practical by leveraging Google expertise and infrastructure.

  Sure such companies may? still not have the computing power of NSA, but the minds in there are definitively on par with such agencies.

I would also advise this to have a look at this: The Moral Character of Cryptographic Work by Phillip Rogaway [paper] [Usenix Talk], specially at the founding part (p. 36). We see a increase in the number of project founded by military agencies.

WARNING: this is a personal assumption, you may chose to disagree with it.

Knowing their missions (cf quote bellow) we can somewhat deduce that their level of research is most likely on par with academic work (why would you heavily fund something that would be bellow your current level of comprehension?)

e.g. for Defense Advanced Research Projects Agency (DARPA) :

The genesis of that mission and of DARPA itself dates to the launch of Sputnik in 1957, and a commitment by the United States that, from that time forward, it would be the initiator and not the victim of strategic technological surprises. source.

Their goal are the following:

• fund academic research
• but most importantly know what you are currently researching on! $\Leftarrow$ This is information they want!

---

TL;DR: I believe they are on par with current academic research.

Answer 2

Fundamentally, I do not believe you can compare them specifically because they tend to have different behaviors. I am an academic who does the non-classified work in conjunction with government teams; however, I'm not a cryptographer, but a circuit designer. I went from industry back to academia as a postdoc just to work in this space. I also don't have a security clearance, so I can specifically publish even if most of it doesn't get out.

In academia, things tend to lean toward an idea more than an application. In the application space, we have constraints and often no pricing pressures so we aren't bound to the hardware. Academic cryptographers tend to prefer complex round functions whereas the government teams usually spent time aggressively simplifying things for the constraint sets. (you see this tradeoff when see a simpler round function with more rounds) Things that the government doesn't think is a problem but academics love: "memory hard function". I can put 8GiB of RAM with through vias on the back of a POWER core on 14nm SOI. This means my IO is 10x faster at a minimum and no cache misses (I don't even put cache on dies in this case). Academic cryptographers generally are not making custom ICs, but there are exceptions.

I'd basically sum it up as academics are ahead with their constraints, and the government is ahead with their constraints.

Answer 3

From a slightly more esoteric perspective, I'd be surprised if governments are much further ahead of the public these days. As we all pass through time, we leave our fingerprints on our surroundings. Entropy inevitably increases and I believe that eventually all truth will out. It can not be contained for ever. That's why it's very difficult to hide a crime once it's discovered. There's some appropriate saying about all the people, all of the time. This isn't an opinion, it's thermodynamics.

If you read around, there is lot of detail as to what governments get up to. And that information is then propagated, so if GCHQ knows what Pyongyang knows, the Sun newspaper will know it soon too. Secrets sell copy. I'm highly cynical, but part of that cynicism is that most people want money, air time or to do the right thing.

If governments are doing something bad, that tends to leak sooner than if they just go about their national security business in ethical ways. There will always be Assanges and Snowdons. A flash drive can hold many pages of documents and they can easily be published to the entire planet. Simply posting on USENET is virtually irrevocable. That's why I know that we did go to the moon and that the CIA did not kill Kennedy. Someone would simply admit it. I'm not sure about Elvis being dead though...