2

How do I prove that the Perceptron bound for mistakes is tight? I need to prove that for any amount of given data points, the total amount of updates (mistakes) that the algorithm will make is $\frac{1}{\gamma ^2}$, where $\gamma$ is the margin. I know that I'll have to make it such that for a given $m$ number of data points, I'll make it such that $\gamma = \frac{1}{\sqrt{m}}$, but I'm not really sure how to continue and would appreciate a hint in the right direction.

Yuval Filmus
  • 280,205
  • 27
  • 317
  • 514
Robert H
  • 21
  • 2

1 Answers1

1

This is also given as Exercise 9.3 in Shai Shalev-Shwartz and Shai Ben-David's Understanding Machine Learning: From Theory to Algorithms, where they provide the following hint: choose the dimension to be $d=m$ and the points to be $x_i=e_i$ (the canonical vectors $(e_i)_j=\mathbf 1_{[j=i]}$).

Observe that all $x_i$s have norm 1. You can first prove that there is a vector $w^*$ with norm $\|w^*\|^2=m$ such that $y_i\langle x_i,w^*\rangle\geq 1$, realising the margin $\gamma=\frac 1{\sqrt m}$ and giving the Perceptron bound $m$. Finally, you can prove that at least $m$ updates are required.