The question is subjective in nature, and this comment is also subjective. It was too long to leave as an actual comment so I'm posting it as an answer, although it isn't really an answer, it's a comment. This is for posterity, I guess -- this thread is already high in Google searches.
NaCl is probably the most widely respected library. It's authored by someone (Daniel J. Bernstein) with an established track record writing secure software and who understands a lot of practical and theoretical subtle security issues. It's also aimed at high-level use and encapsulates details when possible, as opposed to the more common library approach of just providing an open-ended generic API for a lot of cryptographic primitives. That makes NaCl harder to misuse, which is important since some vulnerabilities are simple innocent misuses of libraries. Few other libraries can make similar claims.
I think good audits are hard to come by. My general impression is that very few people are qualified to perform a good audit, and many of those who are qualified are only qualified to evaluate a few areas. What's more, the well-qualified are in high demand with other things and probably have minimal interest in taking the initiative on an auditing project. It isn't lucrative because a good audit:
- takes a long time
- is unlikely to pay much (probably nothing unless bug bounties or grants are involved)
- unlikely to bring fame to the auditor unless they find something huge
- is potentially a liability to the auditor if they miss something that is discovered later
So I don't think there's much incentive to drive high-quality audits.
Popular libraries/applications have had bugs sit unnoticed for a long time, which I think is evidence that audits are rare. History shows examples of bugs going unnoticed for a long time, sometimes even if when we know of them.
The Debian weak keys bug existed for over a year and a half before it was caught. The developer who made the mistake had even explicitly described his change on a public mailing list and asked if the change was secure (the question went unanswered). Let that sink in: A popular open-source library was used for 21 months with a somewhat obvious fatal mistake that was described on a public mailing list.
Look at the list of OpenSSL vulnerabilities (you can filter the more important ones) and use the release dates to match the earliest fixed version against the oldest known vulnerable version. The percentage that existed for 2 or more years before being discovered and fixed is impressive, several lived for over 4 years. Bugs are hard to find and often long-lived.
This isn't library-specific, but I think the lesson holds: The BEAST attack against TLS 1.0 exploited a theoretical vulnerability that had been published 9 years earlier (and fixed in the practically unused TLS 1.1). Even when we know of vulnerabilities, we sometimes don't fix them if it's inconvenient and the attack doesn't seem practical.
I picked on OpenSSL a bit there because it's very popular, open-source, and documented, which made it easy. (Worth noting that the source code is a mess, which probably doesn't encourage anyone to read it.) Other libraries have some similar patterns, and less popular projects may have misleading vulnerability lists because OpenSSL's vulnerabilities are popular when they show up and likely to get a CVE. Is a contributor for a less-popular project going to report every bug they fix, evaluate it as a potential vulnerability, and then file a CVE? I don't know.
In contrast, Truecrypt doesn't have a published list of vulnerabilities, and who knows what subtle fixes are going unreported (or reported under opaque names like "security fix #XYZ") in closed source libraries like Microsoft's. How can we get even a high-level assessment of their vulnerability histories?
My personal feel is that good audits are very hard, very rare, and the majority of auditing is done by the maintainers themselves. To me, the trustworthiness of a library is more centered around who maintains it than how popular (and supposedly more "eyeballed") it is.
Again, this is my general impression and is a subjective response that doesn't fully answer a subjective question.
