8

Related: Why is this function, related to SVM derivation, non-convex?

I am studying notes which cover the derivation of SVM. The intuition is the geometric margin should be maximized in order to result in more confident predictions. The notes pose the following optimization problem:

\begin{equation*} \begin{aligned} & \max_{\gamma,w,b} & & \gamma \\ & \text{s.t.} & & y^{(i)}(w^Tx^{(i)} + b) \geq \gamma; i = 1, \ldots, m.\\ &&& \|w\| = 1. \end{aligned} \end{equation*}

$\gamma = y((\frac{w}{\|w\|})^Tx + \frac{b}{\|w\|})$

If $\|w\| = 1$ then the geometric margin equals the functional margin. However, it is stated that the constraint $\|w\| = 1$ is non-convex. I cannot intuitively see why this constraint is non-convex.

I tried graphing the function $\|w\| - 1 = 0, w \in \mathbb{R^1}$ and plugging in values to the definition of a convex function. I have not found any combination that would show this function to be non-convex.

Clearly I am lacking a fundamental understanding of this problem. Why is $\|w\| = 1$ a non-convex constraint?

Rodrigo de Azevedo
  • 23,223
Alex
  • 285

3 Answers3

20

What makes a constraint convex is not the convexity (or otherwise) of the functions involved, but rather the convexity of the set of points that satisfy the constraint.

The set of points satisfying $\|w\|=1$ is the surface of a norm ball. For the Euclidean norm $\|w\|=\langle w, w\rangle^{1/2}$, it is the surface of an $n$-sphere. These points do not form a convex set. To see this, consider the line segment between any point in the set $w$ and its negative $-w$, also on the surface of the ball. The midpoint of this segment is the origin, which of course is not in the set. This contradicts a claim of convexity, which requires that the entire segment lie within the set. (In fact, when $n=1$, the set consists of just two points!)

The set $\|w\|\leq 1$, on the other hand, is the entire norm ball, including the interior. This is a convex set; so it is a convex constraint.

In practice, equality constraints involving nonlinear functions are almost never convex. The exceptions tend to be trivial to reduce to an equivalent set of linear equations.

Michael Grant
  • 20,110
  • 2
    I can see now that my fixation on convex functions was getting me nowhere. Since $||w|| = 1$ represents a hollow ball, it is not convex. I can imagine trying to connect a point from one side of the sphere to another on the opposite end of the sphere. The line would pass through the hollow space that is not apart of the set. Therefore, it is not convex.

    This may be a hard question to answer, but do you have any suggestions for books relating to this topic that would bolster my understanding? That is very vague, but any recommendations would be great.

     – Alex May 27 '15 at 20:46
  • 1
    That's exactly right. Convex optimization deals in both convex sets and convex functions. In fact, I would argue that it's really the convex sets that matter the most. After all, we can without loss of generality require that a convex optimization problem have a linear objective, so only the constraints have nonlinear functions. And if that's the case, it's only the sets that those constraints make that need to be considered for convexity. – Michael Grant May 27 '15 at 20:48
  • I made the previous edit after you had posted your reply. Could you point me towards other resources that would help me build a better foundation in this topic? – Alex May 27 '15 at 23:35
  • 1
    Indeed: Convex Optimization by Boyd & Vandenberghe. You can even watch Prof. Boyd's lectures online. – Michael Grant May 28 '15 at 00:39
4

Consider $x$ such that $\|x\| = 1$. Then you also have that $\|-x\| = 1$ but what about $\frac{1}{2}(x+(-x))$? This has norm $0$ and so $\frac{1}{2}(x+(-x))$ is not in the set, i.e. the unit sphere is not closed under convex combinations.

Cameron L. Williams
  • 30,485
1

Imagine this equation represents the surface of a sphere or even a circle (W1^2 + W2^2 + W3^2=1 or W1^2 + W2^2=1), now if you join any two random points by a line, none of the points in that line belongs to the sphere or circle, but by definition of the convex set it should be.

explorer
  • 77