Assume that we have a fair coin. We flip it 100 times. The outcome is all heads.
Why is it surprising?
Doesn't all outcomes have the same probability? Any particular outcome, including irregular ones, would have the same very small probability.
Why is it that observing an irregular outcome is less surprising to us than a regular one?
You are right. It is not that surprising. Each specific sequence has probability $\frac{1}{2^{100}}$. The probability to obtain $100$ heads is the same as obtaining any other particular outcome.
But usually we do not ask for a specific sequence but ask for the probability to obtain e.g. $k$ heads and this makes the difference.
There is just $\color{blue}{\binom{100}{100}=1}$ sequence to obtain $100$ heads out of $2^{100} \sim 1.3\cdot 10^{30}$.
But we have $\color{blue}{\binom{100}{50}\sim 1.0\cdot 10^{29}}$ sequences which contain $50$ heads and $50$ tails.
If the sequences that you consider "irregular" constitute the vast majority of all possible sequences, then the probability that the sequence you get is irregular is very high, and it's surprising if you get one of the relatively few "regular" sequences.
One could make the concept of "irregular" precise using something like Kolmogorov complexity. Most bit strings of a given length have no description shorter than the string itself; the ones that admit short descriptions are relatively few.
Assume a fair coin. The idea of "surprising" means it's against our "expectations". The distinction is what is our "expectation"?
If it were a specific exact sequence of heads and tails, then the all heads sequence is just as likely as any other specific sequence, $2^{-100}$. This is a very rare thing to "expect".
More likely our expectation is specified as something like "roughly an even number of heads and tails", which is not one specific sequence of coin tosses but the event of seeing one of many, many sequences. We can calculate with the binomial theorem or approximating a Normal distribution that the probability of "seeing roughly and even number of heads and tails" is very high. For example, the probability of seeing 40-60 heads in our 100 tosses is $$2^{-100} \sum_{i=40}^{60} \binom{100}{i} \approx (1.22 \times 10^{30}) / (1.26 \times 10^{30}) \approx 96.4\% $$
You're asking for a specific sequence of coin tosses of length 100. Sure, any such sequence is as rare as all others, but there are $2^{100}$ of them (all heads, all tails, HTHTHTHTHTH...HTHT, HHHTTTHHHTTT...TTH, etc). So picking one specific sequence means getting it right FOR EVERY SINGLE THROW. This is almost impossibly improbable ($P(\omega) = \frac{1}{2^{100}}$). If you were able to do that in a casino, starting from a "double-or-nothing" bet of 1$ you'd own all the money, debt contracts, resources, and companies in the world long before the end of the coin tosses.
Maybe the reason is
P(the coin is fair | a sequence with low Kolmogorov complexity) $\ll$ P(the coin is fair | a Martin-Lof random sequence)
Although
P(a sequence with low Kolmogorov complexity | the coin is fair) = P(a Martin-Lof random sequence | the coin is fair) = $\frac{1}{2^{100}}$
yet if we take the Solomonoff prior, we would use the algorithmic probability to estimate
$P(x)\approx 2^{-K(x)}$
then
P(a sequence with low Kolmogorov complexity) $\gg$ P(a Martin-Lof random sequence)
therefore
P(the coin is fair | a sequence with low Kolmogorov complexity) $\ll$ P(the coin is fair | a Martin-Lof random sequence)
Take two independent events $A$ and $B$. Then the probability of both happening, which we'll write as $P(A\cap B)$, is given by $P(A)\times P(B)$. In general, given $n$ independent events $E_n$ with probability $p_n$, the total probability of all of them happening is
$$P(\bigcap_{i=1}^nE_n) = \prod_{i=1}^np_n$$
In this case we have 100 independent events, the coin flips into a head, all happening with propability $p=\frac{1}{2}$, meaning we need to multiply $\frac{1}{2}$ by itself one hundred times, namely $(\frac{1}{2})^{100} = \frac{1}{2^{100}}$, roughly $7.9 \times 10^{-31}$.
Is such an outcome surprising? Well, it would be.
You can't flip it $100$ times and get the same outcome as head, but you can flip it $1$ to get outcome as head
You can't flip it $50$ times and get the same outcome as head, but you can flip it twice to get same outcome as head
You can't flip it $25$ times and get the same outcome as head, but it can happen as chance to get same outcome as head in $4$ flip
It's unlikely that chance can happen or you get same result for $8$ flips, because the more the flips the more, the likely the outcome changes
The possibility you get a head or tail is $\frac{1}{2}$ because there are only two choice... This means if I flip it $n$ times, I will likely have equal number of both heads and tails
If I flip the coin $n$ times, and I section the outcomes into $\frac{n}{p}$ in $p$ times, the outcome wouldn't have same possiblity, In fact every new flip behaves independent of the previous outcome
According to https://en.wikipedia.org/wiki/Mega_Millions the probability of winning is about $1$ in $300$ million. Three hundred million is approximately $2^{28}$ and $2^{100}=2^{72}*2^{28}$.
$2^{72}\approx 4.72\times 10^{21}$
Would you be surprised if you won the Megamillions that many times in a row?
All $2^{100}$ sequences are equally probable.
The sequences "all heads" and "all tails" are the most easily specified and most easily noticed sequences.
Imagine you have the coins numbered from 1 to 100 to confirm that the sequence is "all heads" all you need to do it to count zero tails.
The sequence "all coins whose number is a multiple of 5 are heads, the others tails" is equally probable but much more difficult to confirm.
To confirm, you would need to...
A Rosencrantz and Guildenstern type of game (I get the heads, you get the tails) is fairly typical of the most common coin tossing situations insofar as the thing you are most interested in is the total numbers of heads.
The specifications "all heads" and "all tails" are the only ones for which specifying the number of heads is sufficient to determine the entire sequence.
It is surprising because we can recognise a pattern, making the observed sequence remarkable.
Certainly, any other sequence has the same tiny probability, but we are not surprised by most of them, because we cannot recognise a pattern, and so do not find them remarkable. But we should also be surprised – if we noticed them – by other remarkable sequences of a hundred tosses, such as:
For a sequence of a hundred tosses we would also be surprised if any of these sequences occurred except that just three tosses did not fit the pattern.
It is rather like the fascination of science, when we suddenly think “I know what’s going on”. We do not expect to be able to predict the sequence, and when we suddenly notice that it does seem possible, we have good reason to be surprised. Even if we had a pattern that was occasionally broken, we might still be right to be surprised, though once it is broken more often we should watch out. In this case we have to ask ourselves how many similarly broken sequences there are and how many other similar corruptions of other no more complex sequences there are; if these amount to an appreciable fraction of all sequences, we would like to continue tossing and see if the pattern holds up while the fraction of comparable sequences becomes ever smaller.
