Reproducing kernel Hilbert space induced by $k(x, y) = \delta_{x, y}$, where $\delta$ is the Kronecker delta

Question

I am trying to find the reproducing kernel Hilbert space induced by the symmetric positive definite (and bounded and measurable) kernel $$ k \colon X \times X \to \{ 0, 1 \}, \qquad (x, y) \mapsto %\delta_{x, y} := \begin{cases} 1, & \text{if } x = y, \\ 0, & \text{if } x \ne y \end{cases}, $$ where $X$ is a uncountably infinite set, e.g. $X = [0, 1]$ or $X = \mathbb R$ (let us assume that $X$ is second countable and equipped with the Borel sigma algebra).

My work so far: The set $H_{\text{pre}} := \text{span}\{ k(x, \cdot): x \in X \}$ becomes an infinite-dimensional inner product space with respect to the inner product $$ \left\langle \sum_{i = 1}^{n} a_i k(x_i, \cdot), \sum_{j = 1}^{m} b_j k(y_j, \cdot) \right\rangle_{H_{\text{pre}}} := \sum_{i = 1}^{n} \sum_{j = 1}^{m} a_i b_j k(x_i, y_j). $$ In order to find the completion of $H_{\text{pre}}$ with respect to this inner product, I want to find $H := \overline{\iota(H_{\text{pre}})}$, where $\iota \colon H_{\text{pre}} \to H_{\text{pre}}^{**}$, $f \mapsto \Lambda_f$ is the canonical injection into the topological bidual space and $\Lambda_f(\Phi) := \Phi(f)$ for $\Phi \in H_{\text{pre}}^*$ (this is mentioned in this answer by Daniel Fisher). The inner product on $\iota(H_{\text{pre}})$ is $\langle \Lambda_f, \Lambda_g \rangle := \langle f, g \rangle_{H_{\text{pre}}}$, where $f, g \in H_{\text{pre}}$, which can be extended by continuity to $H$.

The set $\mathcal B := \{ k(x, \cdot): x \in X \}$ is an orthonormal basis for $H_{\text{pre}}$. In order to find the topological dual space ${H_{\text{pre}}}^*$, I consider the dual set $$ \mathcal B^* := \{ h \mapsto \langle k(x, \cdot), h \rangle_{H_{\text{pre}}}: x \in H_{\text{pre}} \}. $$ If I am not mistaken, $\mathcal B^*$ is also orthonormal. Note that $\langle k(x, \cdot), h \rangle_{H_{\text{pre}}} = h(x)$ for $h \in H_{\text{pre}}$, so that $\mathcal B^* := \{ \text{ev}_x: x \in X \}$, where $\text{ev}_x \colon H_{\text{pre}} \to \mathbb R$, $h \mapsto h(x)$ is the evaluation functional at $x$. Since $\text{ev}_x(k(y, \cdot)) = k(x, y) = \delta_{x, y}$, $(\mathcal B, \mathcal B^*)$ form an biorthogonal system.

Writing $x = \sum_{i = 1}^{n} a_i k(x_i, \cdot) \in H_{\text{pre}}$ and $f = \sum_{j = 1}^{m} b_j \text{ev}_{y_j}$ we have $$ \iota(x)(f) = \sum_{i = 1}^{n} \sum_{j = 1}^{m} a_i b_j k(x_i, y_j). $$

My questions

Is $\mathcal B^*$ a basis of ${H_{\text{pre}}}^*$, that is, is every functional of $H_{\text{pre}}$ a linear combination of evaluation functionals? Is there a neat representation for the RKHS $H$, e.g. $H = \{ \sum_{k = 1}^{\infty} a_k k(x_k, \cdot): (x_k)_{k = 1}^{\infty} \subset X, (a_k)_{k = 1}^{\infty} \in \ell^1 \}$ in the case that $X \subset \mathbb R^d$?

---

Proof (that $\mathcal B$ is an ONB). For $x, y \in X$ we have $\langle k(x, \cdot), k(y, \cdot) \rangle_{H_{\text{pre}}} = k(x, y) = \delta_{x, y}$. Lastly, $\mathcal B$ is complete: if $\ell \in H_{\text{pre}}$ is in $\mathcal B^{\perp}$, then $\ell(k(x, \cdot)) = 0$ for all $x \in X$ and thus $\ell(f) = 0$ for all $f \in H_{\text{pre}}$. $\square$

---

By @porridgemathematics's comment we know that $$ H = \left\{ \sum_{k = 1}^{\infty} a_k k(x_k, \cdot): \lim_{n \to \infty} \sup_{p \in \mathbb N_{\ge 0}} \sqrt{\sum_{k = n}^{n + p} a_k^2} = 0 \right\}. $$ Since $p \mapsto \sqrt{\sum_{k = n}^{n + p} a_k^2}$ is non-decreasing, I think we can rewrite the condition as $$ \lim_{n \to \infty} \sqrt{\sum_{k = n}^{\infty} a_k^2} = 0, $$ so $H \subset \ell^2(\mathbb N; X)$ if $X$ is nice enough.

Answer 1

For the reverse inclusion (see the discussion in comments), I think everything works out by comparison.

That is, we wish to prove that if $(a_i)_{i = 1}^{\infty} \in \ell^2$ then it satisfies $\lim_{n \rightarrow \infty} \sup_{p \geq 0} || \sum_{i=n}^{n+p} a_i k(x_i,.) ||_{H_{\text{pre}}} = 0$.

One has that $\sup_{p \geq 0} ||\sum_{i=n}^{n+p} a_ik(x_i,.)||_{H_{\text{pre}}} = \sup_{p \geq 0} \sqrt{\sum_{i=n}^{n+p} |a_i|^2}$ since $\langle k(x_i,.) , k(x_j,.) \rangle = k(x_i,x_j) = \delta_{ij}$.

Since $(a_i)_{i=1}^{\infty} \in \ell^2$, we necessarily have that $\lim_{n \rightarrow \infty} \sum_{i=n}^{\infty} |a_i|^2 = 0$, i.e. $\lim_{n \rightarrow \infty} \sqrt{\sum_{i=n}^{\infty} |a_i|^2} = 0$ hence:

$\sup_{p \geq 0} \sqrt{ \sum_{i=n}^{n+p} |a_i|^2 }\leq \sqrt{ \sum_{i=n}^{\infty} |a_i|^2}$, and taking limits as $n \rightarrow \infty$ on both sides yields $\lim_{n \rightarrow \infty} \sup_{p \geq 0} || \sum_{i=n}^{n+p} a_i k(x_i,.) ||_{H_{\text{pre}}} = 0$ as desired.

$\textbf{Edit}$: Therefore $H = \{ \sum_{i=1}^{\infty} a_i k(x_i,.) : (a_i)_{i=1}^{\infty} \in \ell^2(\mathbb{N}) ; a_i \neq 0 \forall i ; (x_i)_{i=1}^{\infty} \text{ any distinct sequence in } X \}$, which are all the square summable (i.e. square integrable with respect to the counting measure on $X$) functions $f : X \rightarrow \mathbb{C}$, i.e. $H = \ell^2(X)$

Answer 2

The attempt in the question as well as the answers seem to obscure the main point a bit (in part because the Wikipedia page does not have the best exposition). In particular, $\lim_{n\to\infty}\sup_{p\in\mathbb N_0}\sqrt{\sum_{k=n}^{n+p}a_k^2}=0$ is just a complicated way of writing $\sum_{k=1}^\infty a_k^2<\infty$.

In general, the RKHS induced by the kernel $k$ is the completion of $\{\sum_x f(x) k(x,\cdot)\mid f\in c_c(X)\}$ with respect to the inner product given by $$ \left\langle\sum_x f(x)k(x,\cdot),\sum_y g(y)k(y,\cdot)\right\rangle=\sum_{x,y}k(x,y)f(x)g(y). $$ Here $c_c(X)$ denotes the set of all finitely supported functions from $X$ to $\mathbb R$.

In the example from the question, $k(x,\cdot)=1_x$ and thus $\sum_x f(x)k(x,\cdot)=f$. Thus the RKHS $H$ is the completion of $c_c(X)$ with respect to the inner product $\langle f,g\rangle=\sum_x f(x)g(x)$. Hence $H$ is contained in $\ell^2(X)=\{f\colon X\to\mathbb R\mid \sum_x f(x)^2<\infty\}$, and since $c_c(X)$ is dense in $\ell^2(X)$, it coincides with $\ell^2(X)$.

Answer 3

Suppose $h_n = \sum_{i=1}^{\infty} a_{n x_{ni}} K(x_{ni},:)$ with$ \sum_{i = 1}^{\infty} (a_{n x_{ni}})^2 < \infty$ is a cauchy sequence then

$||h_n - h_m|| \leq \epsilon$ for all $n,m \geq N$

Let $S = \cup_n \{(x_{ni}): i\}$. The set $S$ is countable. If $x \in S$ does not appear in $\{x_{ni} : i\}$, we set $a_{nx} = 0$.

Hence, $$||h_n - h_m|| = \sum_{x \in S} (a_{nx} - a_{mx})^2 \leq \epsilon$$

We also have $||h_n|| < \infty \implies \sum_{x \in S} (a_{nx})^2 < \infty$.

Hence, $(a_nx : x \in S)$ is a cauchy $\ell_2$ sequence and completeness follows from completeness of $\ell_2$. Let $(a_{nx} : x \in S) \rightarrow (b_x: x \in S)$

Hence $\lim_n h_n = \sum_{x \in S} b_x K(x,:)$.

Hence as other answer pointed out the space: $H = \{\sum_{x \in S} b_x K(x,:), S \ countable, (b_x : x \in S) \in \ell_2\}$ is a complete space and is the smallest complete space containing $H_{pre}$.

Let Assuming you are looking at continuous linear functionals in $H^*$, we have $L \in H^*$ implies $L(f) = <\phi_L,f>$ for a unique $\phi_L \in H$ and for every $f \in H$ by Riesz Representation theorem.

let $f(.) = \sum_i a_i K(x_i,:)$, $\phi_L(.) = \sum_i b_i K(y_i,:)$.

$L(f) = <\sum_i b_i K(y_i,:), \sum_i a_i K(x_i,:)> = \sum_{i,j} a_i b_j K(x_i,y_j)$

and

$\phi_L(.) = \sum_i b_i ev_{y_i}$.

Let $\hat{B} = \{\sum_{x \in S} b_x ev_x, S \ countable, (b_x : x \in S) \in \ell_2\}$.

Hence $\mathcal{\hat{B}}^* = H^*$ as a consequence of Riesz Representation theorem.

But since $H_{pre}$ is not complete, Riesz Representation theorem may not apply in full strength. Look into Riesz representation theorem on where completeness is getting used and then decide.

Note that, $\mathcal{\hat{B}}^* = H^* \subseteq H_{pre}^*$. Hence you have lot more elements than finite linear combination of elements in $\mathcal{B}^*$.

Cheers !