0

I had a friendly discussion with someone about closed form solutions. They contended that the backpropagation algorithm used in calculating the gradients of deep neural networks can't be called a closed form expression in itself, though it is computing a closed form and can be easily converted to one. I pointed them to the Wikipedia entry on closed forms which says -

"In mathematics, a closed-form expression is a mathematical expression that can be evaluated in a finite number of operations. It may contain constants, variables, certain "well-known" operations (e.g., + − × ÷), and functions (e.g., nth root, exponent, logarithm, trigonometric functions, and inverse hyperbolic functions), but usually no limit."

On the one hand, backpropagation can be completed in a finite number of steps. But the next question becomes, is it a "Mathematical expression".

From the Wikipedia entry, it isn't clear if function composition should be an operation that qualifies.

For me, backpropagation should certainly be qualified as a "closed form solution" but I can't come up with a water-tight argument for why.

Also, other examples came up in the discussion. Let's say I have an algorithm for finding the smallest prime number larger than some integer. This involves checking each number greater than that integer and stopping when I find a prime. This algorithm can be completed in a finite number of steps and calculates the solution exactly. But can we say there is "no Mathematical expression" here and hence no closed form?

EDIT: My question is different from the linked question since it focuses on these two cases in particular. The first one being backpropagation for deep neural networks. This basically involves the following:

In general we get a loss function that looks like - $L = f(g(h(x)))$ and then we take the derivative and get - $L' = f'(g(h(x))g'(h(x))h'(x)$. The argument being that because we calculate $h(x)$ separately and then substitute it, this is not a closed form.

The second example being the algorithm for finding the smallest prime number larger than an integer.

Rohit Pandey
  • 7,547
  • As far as I am aware, "closed-form" doesn't have a widely accepted precise definition. People would probably say that $\frac{1}{1-x}$ is closed form, but $\sum_{n=0}^\infty x^n$ is not, even though they compute the same function (for $|x|<1$). I don't know what backpropogation is, but I expect that most algorithms would not be considered closed form. It isn't just the finiteness that is important; you need to be able to express the process in terms of elementary functions and composition too. – James Mar 10 '19 at 19:17
  • @James - backpropagation is just a method to find the gradient of the objective function. In general we get a loss function that looks like - $L = f(g(h(x)))$ and then we take the derivative and get - $L' = f'(g(h(x))g'(h(x))h'(x)$. The argument being that because we calculate $h(x)$ separately and then substitute it, this is not a closed form. – Rohit Pandey Mar 10 '19 at 19:21
  • 1
    The point of naming certain (limited) classes of functions is that may allow us to prove nice things about them, e.g. Risch's algorithm for deciding if an elementary integral exists. Generally the more you widen the class of functions the less you can prove. – Bill Dubuque Mar 10 '19 at 20:54
  • @James "Computing the same function" can't ever matter for defining "closed-form", otherwise there would be no meaning to finding a closed form for any expression. – Misha Lavrov Mar 11 '19 at 04:16
  • see also https://math.stackexchange.com/questions/708787/why-we-say-this-function-have-closed-form-while-the-other-doesnt – IV_ Jul 21 '22 at 13:40

0 Answers0