I've been reading about the No Free Lunch Theorem, but I can't quite understand what it is about. I've heard this theorem described elsewhere as the claim that "no general purpose universal optimiser exists". On the other hand, the Wikipedia article talks about 'candidate solutions" that are "evaluated one by one" - if we only consider algorithms of a particular form, then that is a much more limited claim.
Can anyone explain what this theorem actually claims?
The No Free Lunch theorem (NFL) was established to debunk claims of the form:
My optimisation strategy X is always best.
In particular, such claims arose in the area of genetic/evolutionary algorithms.
The statement is, roughly: every optimisation strategy performs badly on many problems. Therefore, there can be no always-best strategy and your claim about X is wrong.
As far as I understand it, this theory is aimed at global optimization methods like Genetic Algorithms, Neural Networks, Simulated Annealing and so on...
The main claim is that if we consider all possible problem instances of a certain problem, these methods are just as good as any other optimizer that consider the same amount of candidate solutions, even Random Walk for example.
The general purpose optimizers will probably work well on many popular/real life problem instances, but these do not represent well the entire space of possible problem instances.
