22

I know that the Lagrange multiplier method helps us evaluate critical points of $f$ on the closed boundary of the restriction.

In other words we solve:$$\nabla f=\lambda \nabla g$$

But why does this method actually work? Can someone please give me an explanation. Thank you!

CivilSigma
  • 1,117
  • 1
    If $\nabla f=\lambda \nabla g$, it means that the two gradients are scalar multiples of each other and therefore parallel. So when the gradients of the surface ($f$) and the constraint ($g$) align, any movement perpendicular to the gradients will throw you off that critical point. – user3932000 Jan 05 '16 at 15:44

2 Answers2

37

$\nabla f$ is the direction you need to go (to increase the value of $f$), and $\nabla g$ is the direction you can't go (because that's the direction off the surface, and you need to stay on). When these point in the same direction, no possible movement makes any improvement.

I hope it's obvious that this is just a sloppy heap of intuition. Maybe it's a good exercise to figure out the qualifications that need to be made to make it closer to actually correct.

  • "When these point in the same direction, no possible movement makes any improvement." What do you mean by that? thanks – CivilSigma May 20 '15 at 02:22
  • "Possible" movements are the ones allowed by the constraint that you have to stay in the surface $g=0$. (Those movements are in directions $u$ such that $\langle u,\nabla g\rangle=0$.) "Improvement" means increasing the value of the objective function $f$. (Movements making improvements are in directions $u$ such that $\langle u,\nabla f\rangle > 0$.) –  May 20 '15 at 02:25
  • Sorry to bother you again Steven, but does $\lambda$ need to be constant? can it vary by solving each of the x,y or z components of the posted equation? – CivilSigma May 20 '15 at 02:55
  • 1
    I'm not sure what you mean. If you're writing $\nabla f = \lambda \nabla g$ as three equations, one in each coordinate, then no, $\lambda$ must be the same in all three equations. Otherwise $\lambda\nabla g$ isn't a scalar multiple of $\nabla g$, so it's not "the same direction". –  May 20 '15 at 03:10
  • 1
    But if you're thinking about $\nabla f = \lambda\nabla g$ as an equation that might be true or not true at different points on the surface, and wondering whether $\lambda$ has to be the same at each point, then: it's enough that $\nabla f$ and $\nabla g$ be co-directional at each critical point, without there being any particular relationship between their lengths at different critical points. –  May 20 '15 at 03:16
15

So this is largely extracted from the wikipedia article on the above (http://en.wikipedia.org/wiki/Lagrange_multiplier) but hopefully I can make it clear.

I find lagrange multipliers easiest to understand in terms of vectors, so the concepts described apply mainly to functions of 2 variables but I imagine the concepts generalise readily to higher dimensions. So for the below discussion pretend the $\nabla$ is $(\partial/\partial x,\partial/\partial y)$

If you have a function $f(x,y)$ describing a surface over the x,y plane, one possible representation would be as a series of contour lines, curves of constant $f$. Similarly, any constraint on the variable $x,y$ can be rewritten as $g(x,y) = c$. This equation describes a single contour line of $g(x,y)$.

In order to find a minima of $f(x,y)$ subject to a constraint $g$ we are attempting to find the position on $g$ which lies lowest on f. If you calculate values of $f$ moving along the contour $g$, $f$ must decrease as you approach the minima, and increase afterwards. At the minima the change in $f$ must be zero. For this to be true the contour represented by $g$ must be parallel to a contour of $f$, such that infinitesimally any motion along $g$ is along the contour of $f$ and doesn't change $f$'s value. Similar considerations apply to maxima.

Now we need to translate the above statements into something mathematical: the quantity $\nabla f$ points along the direction of fastest change in $f$ (http://en.wikipedia.org/wiki/Gradient), so must be a vector normal to contours of $f$ (otherwise a component of the vector would lie along a contour, where by definition f does not change). Similarly the quantity $\nabla g$ is normal to contours of $g$, and hence to the contour given by our constraint $g(x,y)=c$. Finding the point where contours of $g$ and $f$ are tangential is now easy. We just need to find the points where the normals to $f$ and $g$ are parallel $\implies \nabla f = \lambda \nabla g$ where $\lambda \in R$. Note that if there are multiple minima or maxima this should just fall out of the maths.

Scott Holmes
  • 161