Multiple Rainbow Tables and their Success Probability

Question

I am trying to understand the calculations for the success probability of finding an unknown key using rainbow tables.

Basically, I have a target success probability, a known keyspace of size $N$ and want to derive the neccessary table building parameters: the number of chains $m$, and the length of each chain $t$, and potentially the number of tables $l$ (see below).

The original paper by Oechslin gives the success probability of a single $m \times t$ table for finding a key in a keyspace of $N$ keys as $$ P_{table} = 1 - \prod_{i=1}^{t}\left(1 - \frac{m_i}{N}\right) $$

with $m_i$ recursively defined as $m_1 = m$ and $m_{n+1} = N\left(1-e^{\frac{-m_n}{N}}\right)$.

The appendix in the paper gives an explanation of this formula, which I can mostly follow. However, with $m_i$ defined like above, I cannot see how one could derive the parameters $m, t$ except by just evaluating the function, which is quite computationally expensive. These posts suggest, that one just fixes $m$ to the available amount of memory and then scales $t$ to achieve the targeted success probability. This post links to a paper discussing this problem, but as far as I can tell perfect tables are assumed and the computations do not get much simpler.

It is argued, that the rainbow tables have a similar success probability to multiple Hellmann tables (each using a different reduction function), totaling to the same size as a single rainbow table. However, I could not find a proof for this claim anywhere.

Additionally, in Section 4.3 of Oechslin's paper, multiple rainbow tables are used to achieve a higher success probability than with a single table.

• In what way do these multiple rainbow tables differ from each other?
• How does using multiple tables differ from just constructing a single larger table?
• How does using multiple rainbow tables increase the success probability?

Answer 1

With perfect rainbow tables the math for evaluating is much simpler but the math for how long it takes to the build them becomes complex.

It all has to do with collisions and since we eliminate almost all of them the math is simple. Let's look at how collisions behave in a classic distinguishing point table. When adding a new row each value may collide with any previous $m*t$ point and the continuation to the end of the row will obviously collide as well. So any collision costs us the whole row not just the colliding point. When we split into multiple tables this changes. A collision with within the table is as before but is less likely as we separated our points between tables so each table is much smaller. A collision across table isn't that bad because because only the single point will collide the next point in each chain will be different in each table so this isn't very bad and is in many cases ignored in analysis.

Rainbow tables are similar, each column has a different hash function so a rainbow table column is similar to a separate table. Only $\frac{1}{t}$ collisions are in the same column and will continue to collide till the end of the row. Cross column collisions like cross table collisions don't propagate and aren't expensive. $t$ distinguishing points tables with $m$ rows and $\sim t$ columns each have similar coverage to a single rainbow table with $m*t$ rows and $t$ columns. The obviously both take $m*t$ space. But the lookup time in the rainbow version is half, because columns near the end are cheaper to check.

As to the math, the first formula is easy, $m_i$ are the number of distinct values in each column and the probability of appearing in each column is independent so the coverage is $1$ minus the probability of missing all columns. The next formula is the tricky one. Each column has fewer distinct values then the previous one. This comes from balls into bins theorum. Throwing $m$ balls into $n$ bins. The expected number of empty bins is $n*(1-\frac{1}{n})^m$ and subtract that from $n$ you get the number of different values. The version with $e$ is an excellent aproximation for large $m,n$.

As to multiple rainbow tables: The first column of a rainbow table is special we can easily ensure it is unique. Adding columns each column is less effective. If we break into multiple tables we need to look up in each of them independently so splitting into two tables cost us lookup time. but collisions between tables are a non issue. Try punching in some numbers to get a feel for it. If our search space n is 2^50 and we have a memory budget for 2^30 rows. Let's consider one or two rainbow tables. If we want a probability of 50% in a single table we need 869,000 columns. For two tables each with half the rows we will need each table to have a success rate of $1-sqrt(1-0.5)$ which ~0.292 This will require 790,000 columns. Which is better? The single table, because it's runtime is $869000^2/2$ which is less then $2(790000^2/2)$ But what if we want a high probability for example 95% ? for a single table we will need to go all the way up to 7,300,000 columns. But two tables with half the rows will need only 4,680,000 rows to reach 77% each. Now we punch in and see the two tables are more then twice as fast then the single table. If you fix sigma(m) for all tables and p, you can iterate over small table counts for each find the necessary t and pick the optimum.

Answer 2

• In what way do these multiple rainbow tables differ from each other?

They use different (sets of) reduction functions.

• How does using multiple tables differ from just constructing a single larger table?

With different reduction functions a collision in the $i$th column between two tables does not lead to the rest of the chains merging, while within a single table it would. The downside is that you need to look up in each table separately, so lookups are more expensive.

• How does using multiple rainbow tables increase the success probability?

Suppose you have $l$ tables of size $m \times t$.

• Compared to a single table of size $m \times t$ you clearly get much more coverage.
• Compared to a single table of size $lm \times t$ you have fewer collisions (due to the above point of avoiding chain merging).
• Compared to a single table of size $m \times lt$ you have much faster lookups: $\operatorname{O}(lt^2)$ instead of $\operatorname{O}(l^2t^2)$, plus fewer false alarms. And fewer merges.

If you compare to a single merge-free table of size $lm \times t$ you need to expend less effort to compute it, because you encounter fewer merges. And for high coverage targets it may be impossible to generate a single merge-free table, because there are not enough enough non-merging chains to reach it (see also Section 5. of Oechslin's paper).