Computer Science Stack Exchange
Computer Science Stack Exchange

Questions tagged [vc-dimension]

The VC dimension (for Vapnik–Chervonenkis dimension) is a measure of the capacity (complexity, expressive power, richness, or flexibility) of a statistical classification algorithm, defined as the cardinality of the largest set of points that the algorithm can shatter.

49 questions
104
votes
7 answers

Why is deep learning hyped despite bad VC dimension?

The Vapnik–Chervonenkis (VC)-dimension formula for neural networks ranges from $O(E)$ to $O(E^2)$, with $O(E^2V^2)$ in the worst case, where $E$ is the number of edges and $V$ is the number of nodes. The number of training samples needed to have a…
yters
  • 1,457
  • 2
  • 13
  • 21
19
votes
1 answer

Efficiently computing or approximating the VC-dimension of a neural network

My goal is to solve the following problem, which I have described by its input and output: Input: A directed acyclic graph $G$ with $m$ nodes, $n$ sources, and $1$ sink ($m > n \geq 1$). Output: The VC-dimension (or an approximation of it) for the…
Artem Kaznatcheev
  • 4,872
  • 2
  • 28
  • 57
8
votes
1 answer

Vapnik-Chervonenkis Dimension: why cannot four points on a line be shattered by rectangles?

So I'm reading "Introduction to Machine Learning" 2nd edition, by Bishop, et. all. On page 27 they discuss the Vapnik-Chervonenkis Dimension which is, "The maximum number of points that can be shattered by H [the hypothesis class] is called the…
BrotherJack
  • 1,115
  • 1
  • 10
  • 23
8
votes
2 answers

VC dimension of linear separator in 3D

I am confused about the Vapnik-Chervonenkis dimension of a linear separator in 3 dimensions. In three dimensions, a linear separator would be a plane, and the classification model would be "everything on one side of a plane." It's apparently…
Jason Smith
  • 81
  • 1
  • 2
6
votes
2 answers

What is usually the next step after showing the VC dimension?

I am new to statistical learning. I have a structure $X$ where I showed its hypothesis class $H$ has VC dimension $d$. All I know now is that I can bound the number of examples by $m\geq \frac{1}{\epsilon}ln \frac{d}{\delta}$ and with probability at…
seteropere
  • 677
  • 1
  • 4
  • 12
5
votes
1 answer

VC dimensions: Let ${x_1, \ldots, x_N}$ be $N$ labelled points on $\mathbb{R}$, then there exists a sinusoid that separates these points

(Proposition, pg 26): Let ${x_1, \ldots, x_N}$ be $N$ points on $\mathbb{R}$, $N \in \mathbb{Z}$, labelled either $+1$ or $-1$ , then there exists a function from the set $\{t \mapsto \sin(\omega t)| \omega \in \mathbb{R}\}$ that separates these…
Your neighbor Todorovich
  • 153
  • 4
4
votes
2 answers

Is it a problem that "successful" machine learning algorithms have large VC dimension?

In my limited exposure, it appears that "successful" machine learning algorithms tend to have very large VC dimension. For example, XGBoost is famous for being used to win the Higgs Boson Kaggle competition, and Deep Learning has made many…
yters
  • 1,457
  • 2
  • 13
  • 21
4
votes
0 answers

VC dimension and binary operations

There are two classes of binary functions, $F_1,F_2$. The VC-dimension of $F_i$ is $d_i$. Is there any simple formula for the VC-dimension of the following classes? $F_\lor := \{ f_1(x) \lor f_2(x) | f_1\in F_1 \text{ and } f_2\in F_2\} $ $F_\land…
Erel Segal-Halevi
  • 6,088
  • 1
  • 25
  • 60
4
votes
1 answer

VC dimension of complement

Let $C\subseteq 2^X$ be a concept class over $X$ and let $\bar{C}:=\{X\setminus c\mid c\in C\}$ be the complement. Show that $VCdim(C)=VCdim(\bar{C})$. Proof: Let $d:=VC_{dim}(C)$, then there exists $S\subseteq X$, $|S|=d$, s.t. $S$ is shattered by…
user14600
  • 51
  • 3
4
votes
1 answer

VC dimension and optimal mistake bound

I have a question regarding optimal mistake bound for learning algorithm There is a famous fact that $VC(C) \leq Opt(C)$, where $C$ - set of learning concepts, VC(C) - VC dimension of C, $Opt(C)$ - the smallest mistake bound (of the best learning…
user16168
  • 317
  • 1
  • 8
3
votes
3 answers

VC Dimension of Origin-Centered Circle

Is the VC dimension of an origin-centered circle 1 or 2? It seems to me that the VC dimension of an origin centered circle should be 1, because for two points with distances from the origin r1 <= r2, r2 will never be able to be labeled 1 without r1…
rfj001
  • 133
  • 1
  • 5
3
votes
2 answers

Do I have the right definition of VC dimension?

I am having some trouble understanding the notion of the VC dimension. The definition I have is the following: The VC dimension of a set of hypothesis functions $H$ is the cardinality of the largest set which $H$ can shatter. We say that $H$…
user27182
  • 133
  • 3
3
votes
1 answer

The VC dimension when the samples are fixed

The VC dimension is usually used in the following way. There is a space of hypotheses. There is an unknown probability distribution. We sample some training-samples from this distribution. We find the hypothesis that scores best on the…
Erel Segal-Halevi
  • 6,088
  • 1
  • 25
  • 60
3
votes
2 answers

How to determine the size of training data using VC dimension?

I want to determine the size of training data ($m$) when I know the parameters $VC(H)$, $δ$ and $e$. As I know the $VC$ bound satisfy this equation: $$ \mathrm{error}_{\mathrm{true}}(h) \le \mathrm{error}_{\mathrm{train}}(h) + \sqrt\frac{VC(H)…
Hadi
  • 141
  • 6
3
votes
0 answers

How to find the shattered set size for unknown hypothesis target

My aim is to prove a VC-dimension $d$ for different problems. All the problems I have do not have a target function (or concept) explicitly stated. This unlike most of the examples I came through. For example in the interval problem, the target…
seteropere
  • 677
  • 1
  • 4
  • 12
1
2 3 4