Tell me about this $p$-value in relatively simple but not too simple terms.
I've been reading many academic publications of late and come across it quite a few times without having a clear idea what it means. It's something about probability, isn't it? If it's less than, I believe, $0.05$, then it's, whatever it is, not a simple coincidence but a real correlation, right?
The $p$-value is the probability of obtaining the value of the Statistic, or, as pointed out below, a value as extreme as the one you obtained that you obtained while assuming your $H_0$; Null Hypothesis is correct.
In Inferential Statistics you are interested in estimating a population parameter, say the average age of your (human) population. You believe the average is less than, say $35$. This becomes your Null Hypothesis $H_0$. In order to test $H_0$ , you use a statistic ( A function of randomly-selected data. here the sample mean) to obtain a value $\mu$ from your data. Your statistic; here sample mean, has a distribution function $f$ corresponding to it. Using $f$, you can compute the probability of finding a value like $\mu$ or lower. This last is your $p$-value.
The p-value is the smallest level of significance $\alpha_0$ that I would reject the null hypothesis with the observed data.
In olden times people simply reported a result, "rejected the null" or "did not reject the null." This doesn't tell me how close I was to not rejecting the null hypothesis. If a researcher rejected the null hypothesis at significance level $0.05$, this doesn't tell other researchers if they would reject the null hypothesis at significance level $0.01$, which is a stricter condition. Hence, it has become common practice to report all $\alpha_0$'s at which $H_0$ would be rejected.
If the test $\delta$ is of the form "Reject $H_0$ if $T\ge c$" for a test statistic T, and a value of $T=t$ is observed, then the p-value equals the size of the test $\delta_t$ with $\delta_t$ being the test that rejects $H_0$ if $T\ge t$:
$$\text{p-value}=\alpha(\delta_t)=\sup_{\theta\in\Omega_0}\pi(\theta|\delta_t)=\sup_{\theta\in\Omega_0}\Pr(T\ge t|\theta)$$
It's often called the chance of observing a dataset as extreme as the one observed if the null hypothesis is true, which is true if the null hypothesis is simple (contains only one point) or if the power is maximized on a boundary point of $\Omega_0$ (which is often the case).
