Presumably $P$ is a probability measure on $(\Omega,\mathscr{F})$. In any case, in what follows, it suffices to assume that $P$ is a finite measure.
$\mathcal{L}_0(\Omega,\mathscr{F},P)$, as a set, is the space of all $\mathscr{F}$-measurable functions $f:(\Omega,\mathscr{F})\rightarrow(\overline{R},\mathscr{B}(\overline{R}))$ such that $P(|f|=\infty)=0$ (here $\overline{R}=\mathbb{R}\cup\{\pm\infty\}$). The space $\mathcal{L}_0(\Omega,\mathscr{F},P)$ admits a pseudo-metric $d$ under which convergence in $d$ is the same as convergence in probability.
Once $P$-almost surely equal functions are identified ($f\sim g$ iff $P(f=g)=1$) the resulting space $(L_0,d)$ becomes a (metrizable) topological vector space, that is, sum $(f,g)\mapsto f+g$ and scalar product $(r, f)\mapsto rf$ are continuous as maps from $(L_0,d)\times(L_0,d)$ into $(L_0,d)$ and from $(\mathbb{R},|\,|)\times(L_0,d)$ into $(L_0,d)$ respectively. With some additional effort one can show that $(L_0,d)$ is a complete metric linear space.
Here is one way to metrize $L_0$ to obtain the so called convergence in measure topology. Define
$$\|f\|_0:=\inf\{\varepsilon>0: \mu(|f|>\varepsilon)\leq\varepsilon\}$$
It is easy to check that
- $P(|f|>\|f\|_0)\leq \|f\|_0\leq P(\Omega)$
- If $P(f=\pm\infty,g=\mp\infty)=0$, then $\|f+g\|_0\leq\|f\|_0+\|g\|_0$
- If $|f|\leq |g|$, then $\|f\|_0\leq \|g\|_0$
- $\|r f\|_0\leq (r\vee1)\|f\|_0$ for all $r\in[0,\infty)$
- If $P(|f|=\infty)=0$, then $\lim_{r\rightarrow0}\|rf\|_0=0$
All these relations show that $d(f,g)=\|f-g\|_0$ is a (translation invariant) pseudo-metric, that $\|f\|_0=0$ iff $f=0$ $P$-almost surely. Furthermore, after identification of $P$-almost surely functions, the sum and the scalar multiplication operations are continuous.
The most important part of this discussion is that of convergence in measure.
$f_n\rightarrow f$ in measure iff $\|f_n-f\|_0\xrightarrow{n\rightarrow\infty}0$.
Proof: Suppose $\|f_n-f\|_0\rightarrow0$ as $n\rightarrow\infty$. Let $\delta>0$ be fixed. For any $0<\varepsilon\leq\delta $, there is an integer $N$ such that $\|f_n-f\|_0<\varepsilon$ whenever $n\geq N$, and so
\begin{align}
P\big(|f_n-f|>\delta\big)\leq P(|f_n-f|>\varepsilon)\leq
P\big(|f_n-f|>\|f_n-f\|_0\big)\leq\|f_n-f\|_0<\varepsilon
\end{align}
for all $n\geq N$. Hence $f_n$ converges to $f$ in $P$-measure.
Conversely, suppose $f_n$ converges to $f$ in $P$-measure. Then, for any $\varepsilon>0$ there exists and integer $N$ such that $P(|f_n-f|>\varepsilon)<\varepsilon$ for all $n\geq N$. Hence, $\|f_n-f\|_0\leq\varepsilon$ whenever $n\geq N$.
- To show that $d$ is complete, suppose $\{f_n:n\in\mathbb{N}\}$ is a Cauchy sequence. There is an increasing sequence of integers $n_\ell$ such that
\begin{align}
\sup_{m,k\geq n_\ell}\mu(|f_k-f_m|>2^{-\ell})\leq \sup_{m,k\geq n_\ell}\|f_m-f_k\|_0< 2^{-\ell}.
\end{align}
Hence, $\sum_l\mu\big(|f_{n_{\ell+1}}-f_{n_\ell}|>2^{-\ell}\big)<\infty$, and by the Borel--Cantelli theorem, for $\mu$-almost all $\omega\in\Omega$, there is an integer $N$ (depending on $\omega$) such that
$|f_{n_{\ell+1}}(\omega)-f_{n_\ell}(\omega)|\leq 2^{-\ell}$ for all $\ell\geq N$. Consequently, $f:=\lim_{\ell\rightarrow\infty}f_{n_\ell}=\sum_k(f_{n_k}-f_{n_{k-1}}) +f_{n_1}$ exists $\mu$--a.s. This means that $\{f_n\}$ has subsequence $\{f_{n_\ell}\}$ that converges $\mu$--a.s, and so in $L_0$.
There are other equivalent metrics on $L_0$. For any bounded metric $\rho$ on $\mathbb{R}$, $d_\rho(f,g)=E[\rho(f,g)]$ also defines a metric on $L_0$ that coincides with the convergence in measure topology. One of the most common metrics used is $d'(f,g):=E[|f-g|\wedge1]$.
One big difference between $L_0$ and $L_p$ for $p\geq1$ every such $L_p$, being a Banach space, is locally convex. $L_0$ is not in general locally convex. In this respect, $L_0$ has more similarities with the $L_p$ spaces with $0<p<1$.
If $P$ has no atoms, then the dual of $L_0$ is $\{0\}$; the dual space of $L_p$, for $p\geq1$ is $L_q$ with $\frac1p+\frac1q=1$.