Binary classification, Bayes classifier, Bayes decision boundary

Question

I have recently come across this problem from a friend I help with stats occasionally. This however stumped me completely. I have looked online on basically every single website you can find but what I did find either I did not fully understand or I didn't fully understand well enough to explain to my friend. In my last resort I have made an account here hoping for some help.

From what I have read and understood I know the Bayes boundary is a kind of squiggly line you find which separates between classifying an observation on either end. However I don't understand how one comes to finding how to get one. Furthermore, bayesian stats is very new to me so I am struggling and therefore my friend is too. Thank you for reading and I hope someone who understands this well can explain it to me in a simple concise and easy way

Consider a binary classification problem $Y \in \{0, 1\}$ with one predictor $X$. The prior probability of being in class 0 is $Pr(Y = 0) = \pi_0= 0.69$ and the density function for $X$ in class 0 is a standard normal
$$f_0(x) = Normal(0, 1) = (1/\sqrt{2\pi})\exp(-0.5x^2).$$

The density function for $X$ in class 1 is also normal, but with $\mu = 1$ and $\sigma^2 = 0.5$, i.e.
$$f_1(x) = Normal(0, 1) = (1/\sqrt{\pi})\exp(-(x-1)^2).$$

(a) Plot $\pi_0f_0(x)$ and $\pi_1f_1(x)$ in the same figure.
(b) Find the Bayes decision boundary.
(c) Using Bayes classifier, classify the observation $X = 3$. Justify your prediction.
(d) What is the probability that an observation with $X = 2$ is in class 1?

Any help at all would be greatly appreciated!

Answer 1

I think I'm interpreting this problem correctly, but I might also just be kidding myself.

Basic approach. Suppose that we observe $X = x$. We are looking for, essentially, $P(Y = y \mid x < X < x+dx), y = 0, 1$. Bayes permits us to rewrite this as

$$ P(Y = y \mid x < X < x+dx) = \frac{P(x < X < x+dx \mid Y = y) P(Y = y)}{P(x < X < x+dx)} $$

We are given that

$$ \pi_y \stackrel{\text{def}}{=} P(Y = y) =\begin{cases} 0.69 & y = 0 \\ 0.31 & y = 1 \end{cases} $$

and conveniently,

$$ f_y(x)\,dx = P(x < X < x+dx \mid Y = y) $$

I would guess that the decision boundary lies wherever

$$ P(Y = 0 \mid x < X < x+dx) = P(Y = 1 \mid x < X < x+dx) = \frac12 $$

On the basis of the foregoing, we can rewrite this as

$$ \pi_0 f_0(x) \, dx = \pi_1 f_1(x) \, dx $$

or, equivalently,

$$ \pi_0 f_0(x) = \pi_1 f_1(x) $$

For this problem, I make it that the boundary consists of two values of $x$; between these values, one of the values of $Y$ is preferred; outside of those values, the other value of $Y$ is preferred.

Answer 2

Given a predictor $X$, you want to know from which gaussian it was most likely sampled. So you want to calculate $\mathbb{P} (Y = 0 | X)$ for instance. Using bayes rule, and the fact that we have access to $\mathbb{P} (X | Y = 0)$: $$ \mathbb{P} (Y = 0 | X)= \frac{\mathbb{P} (X | Y = 0) \pi_0}{\mathbb{P}(X)} $$ and $\mathbb{P}(X) = \pi_0f_0(x) + \pi_1f_1(x)$ by the law of total probability.
In summary : $$ \mathbb{P} (Y = 0 | X)= \frac{\pi_0 f_0(x)}{\pi_0f_0(x) + \pi_1f_1(x)} $$ $$ \mathbb{P} (Y = 1 | X)= \frac{\pi_1 f_1(x)}{\pi_0f_0(x) + \pi_1f_1(x)} $$ And you're going to predict $Y=0$ for $X$ if $$\mathbb{P} (Y = 0 | X) \gt \mathbb{P} (Y = 1 | X) $$ meaning : $$ \pi_0 f_0(x) \gt \pi_1 f_1(x) $$ And the decision boundary is the $x$ solution to: $\pi_0 f_0(x) = \pi_1 f_1(x)$. I'll leave the calculations to you because it's pretty basic.

The intuition behind this is that : If you have two gaussians $G_1$ (red) and $G_2$ (blue) that have same probability $1/2$ (like in the figure below), you're going to predict $G_1$ for $X$ if $X \leq 0$, and $G_2$ otherwise. Here the decision boundary is the intersection between the two gaussians. In a more general case where the gaussians don't have the same probability and same variance, you're going to have a decision boundary that will obviously depend on the variances, the means and the probabilities. I suggest that you plot other examples to get more intuition.