Is it really possible to prove lower bounds?

Question

Given any computational problem, is the task of finding lower bounds for such computation really possible? I suppose it boils down to how a single computational step is defined and what model we use for the proof, but given that, do we really prove a lower bound then in general? What I means is that can we prove something like "problem $X$ cannot be solved faster than $t(X)$ time" rather than "problem $X$ can be solved in $t(X)$ time or faster"?

I have tried to find information specifically on lower bounds and proofs of them, but I can't really find any of interest, any recommendations on books/papers/websites on the subject?

Answer 1

We can absolutely prove such things.

Many problems have trivial lower bounds, such as that finding the minimum of a set of $n$ numbers (that are not sorted/structured in any way) takes at least $\Omega(n)$ time. The proof for this is simple: a hypothetical algorithm that runs in $o(n)$ time can not examine all of the numbers in the input. So if we ran the algorithm on some input, we could observe that it never examined a particular element of the input. Changing that element to the minimum, we can get the algorithm to fail.

A less trivial lower bound is the $\Omega(n \log n)$ lower bound for sorting in the comparison-based model. The proof for that goes along the following lines: given an input of $n$ numbers, there are $n!$ possible outputs (the input could be any permutation of the sorted list, so the output can also be any permutation of the input). If we are limited to only doing comparisons, then our algorithm (on average) needs to perform at least $\log_2(n!)=\Omega(n \log n)$ comparisons in order to be able to give $n!$ different outputs.

Lower bounds can be even stronger. There are several problems (notably the $EXPTIME$-hard problems) for which there is an exponential lower bound. Problems in this class include computing optimal strategies for games such as (generalized) chess, checkers and go. The proof of this is via the Time Hierarchy Theorem, which states (subject to some restrictions on $f$):

Given a function $f$, there exists a computational problem that can be solved in time $O(f(n))$ but can not be solved in time $o(\frac{f(n)}{\log n})$.

So basically, if you can think of a function $f$ there exists a problem that requires that much time to solve.

Finally, another avenue of not necessarily proving a time lower bound but something even stronger is showing the undecidability of a problem (e.g. halting, post correspondence).

Answer 2

Yes, it's possible. The classic example is the fact that any comparison-based sorting algorithm requires $\Omega(n\log n)$ comparisons to sort a list of length $n$.

However, lower bounds seem to be much harder to prove than upper bounds. To prove that there's a sorting algorithm that requires $O(n\log n)$ comparisons, you just need to exhibit such an algorithm (merge sort – voila!). But for a lower bound, you somehow need to show that no algorithm in a particular class can solve your problem. The difficulty of doing that is illustrated by the fact that we only know that $$\mathrm{L}\subseteq \mathrm{NL}\subseteq \mathrm{P}\subseteq \mathrm{NP} \subseteq\mathrm{PSPACE}\,,$$ even though we know that at least one of those inclusions is strict ($\mathrm{L}\subset \mathrm{PSPACE}$ by the space hierarchy theorem) and most people think they are all strict.

On the other hand, Ryan Williams has a nice paper (and talk, which I've heard a couple of times) called Algorithms for Circuits and Circuits for Algorithms, in which he argues that finding lower bounds and finding algorithms aren't fundamentally all that different. For example, he cites the proof of the undecidability of the halting problem as an example of an algorithm (the universal Turing machine) being used exactly to prove a lower bound (undecidability).

Answer 3

To take a trivial example, you cannot find the largest number from a set of $n$ numbers without checking them all which requires linear time. No big proof. But there are algorithms that do not always require reading all the data. A nice example is searching all occurences of a pattern in a string, which may not require reading the whole string (Boyer-Moore algorithm). But I should not repeat what is already answered, probably better than I would.

However, there is a point in the question that calls for some more remarks regarding lower bound (or complexity bounds in general).

Actually the choice of what is a single computational step is irrelevant, as long as computational steps can be considered as having a constant upperbound (and lower bound). The complexity result will be the same since it is defined up to a constant. Taking 3 comparisons as unit operations, or only a single one, makes no difference.

The same is true regarding the size of data which serves as reference to evaluate the cost of the computation. Taking a single integer, or two integers as unit of size makes no difference.

However, the two choices must be related.

You may consider that a number $n$ is a single unit of data if you consider that operations on integers, such as addition or comparison, are unit operations. You may also consider that it is $\log n$ units of data since it takes $O(\log n)$ digits to represent a number. Then, of course, addition and comparison are no longer unit operations, and their cost depends on the values of the operands.

Whether an operation can be considered having unit cost is tightly related to what data can be considered as having unit size. And that depends on what level of abstraction you choose for your model of computation.