7

We know that for a density function $f(x\mid\theta)$ we have $$\int^{\infty}_{-\infty}f(x\mid\theta)\, dx=1$$ Do we also have for the likelihood function $L(\theta\mid x)$ that $$\int^{\infty}_{-\infty}L(\theta\mid x)\, d\theta=1?$$ My guess is yes, since $L(\theta\mid x)$ is some kind of "possibility" of the parameter that under different data.

Thanks~

TheSimpliFire
  • 28,020
Matata
  • 2,138
  • The likelihood, a function of the parameter given the sample observation, need not integrate to $1$ as it is not supposed to behave like a probability density. The likelihood function does not even always equal the joint pdf, it is only supppsed to be non-negative. – StubbornAtom Jun 19 '18 at 19:54
  • @StubbornAtom Thanks. So according to my textbook, the likelihood function is defined as the joint pdf, so why they do not need to be the same then? – Matata Jun 19 '18 at 20:21
  • See the answers below. – StubbornAtom Jun 19 '18 at 20:32

3 Answers3

3

A biased coins turns up "heads" $2/3$ of the time; a fair coin $1/2.$

Given a "head", the likelihood function is \begin{align} & L(\text{fair}) = 1/2, \\ & L(\text{biased}) = 2/3. \end{align} Do these add up to $1$? Not if we assign weight $1$ to each possible coin.

And assigning such weights does not appear to make sense except when they are prior probabilities. Suppose we have prior probabilities $\Pr(\text{fair}) = 0.9 = 1 - \Pr(\text{biased}) = 1 - 0.1\,.$ Then integrating our likelihood yields $$ (0.9)(2/3) + (0.1)(1/2) = 0.65 $$ and that is not $1,$ but rather it is the marginal probability of the observed data, i.e. the marginal probability of "heads".

Michael Hardy
  • 1
1

Take the Poisson distribution example, with two data, say $x_1=1,\,x_2=2$.

We have $$f(x\mid \mu)=\frac{\mu^{x}e^{-\mu}}{x!}$$ so $$L(\mu\mid x_1, x_2)=\prod_{i=1}^2\frac{\mu^{x_i}e^{-\mu}}{x_i!}=\frac{\mu^3e^{-2\mu}}{2}$$ Then $$\int_{-\infty}^\infty L(\mu\mid x_1,x_2)\,d\mu=\frac12\int_0^\infty\mu^3e^{-2\mu}\,d\mu=\frac12\cdot\frac38\neq1.$$

TheSimpliFire
  • 28,020
  • 3
    I find this flawed: For one thing, the parameter space is $[0,+\infty),$ but you integrated over $(-\infty,+\infty).$ The integral over $[0,\infty)$ does converge. The other problem is that there's no measure with respect to which you can integrate, unless perhaps there's a prior probability distribution of $\theta.$ (And in that case, the integral is not $1$ but instead is the marginal probability of the observed data. $\qquad$ – Michael Hardy Jun 19 '18 at 19:42
0

The meaningful way to turn a likelihood into a probability is to integrate it against a prior probability distribution for $\theta$. Good choices of a prior are more art than science. But in particular, the Lebesgue measure on $\mathbb{R}$ is not admissible for this purpose, because it's not a probability distribution. However, if you have $\theta$ confined to $[a,b]$, then you can use the uniform distribution on $[a,b]$ as a "naive prior". You could do the same if $\theta$ is confined to a finite set.

When you do this, you still don't get $1$, though, you get the probability of observing your particular data set under your prior distribution for $\theta$. For example, consider a Bernoulli($p$) data point $x_1=1$ and the naive prior for $p$. The likelihood function is $L(p \mid \{ 1 \})=p$, and the integral of this against the naive prior is $\int_0^1 p \, dp = 1/2$.

Michael Hardy
  • 1
Ian
  • 104,572
  • The Lebesgue measure on $\mathbb R$ may not be not admissible, but it is not unknown for an improper uniform prior on $\mathbb R$ to be used – Henry Jun 19 '18 at 22:04
  • @Henry True, but then you may not be able to integrate the likelihood against it. – Ian Jun 19 '18 at 22:08