$\newcommand{\SLAG}{\mathsf{SmLAG}}$$\newcommand{\guess}{\mathrm{guess}}$ $\newcommand{\ov}[1]{\overline{#1}}$ $\newcommand{\bb}[1]{\mathbb{#1}}$$\newcommand{\Spec}{\mathrm{Spec}}$$\newcommand{\et}{\mathrm{et}}$$\newcommand{\Et}{\mathrm{Et}}$
Question 1: Let us assume that $k$ is perfect (this is not so important) that we're working in the category of smooth (this is important) linear algebraic groups which we denote $\mathsf{SmLAG}_k$. What is the issue with defining the quotient $G/N$ where $G$ and $N$ are objects of $\mathsf{SmLAG}_k$ and $N$ is normal in $G$?
Well, if we're trying to define $G/N$ as a group variety, the first step is to even give it the structure of a $k$-scheme. Now, d'apres Grothendieck there are two ways to think about a $k$-variety:
- We can think of it as a locally ringed space (a topological space with a sheaf of rings whose stalks are local rings) satisfying some special properties.
- We can think of it as a presheaf on the category of affine $k$-schemes satisfying some special properties.
Now, then, an obvious guess of how to define $G/N$ is to think of $G$ and $N$ as belonging to either one of the above two perspectives and trying to take the 'quotient' there. For the sake of simplicity, let's start with perspective 2.
So then, to define $G/N$ as a presheaf on affine $k$-algebras $\mathrm{Spec}(R)$, we need to say what $(G/N)(R)$ is for every $k$-algebra $R$. The obvious guess is to make the following definition
$$(G/N)^\guess (R) := G(R)/N(R)$$
But, this comes with one huge and somewhat obvious issue. Namely, in perspective 2. the devil is in the details of "...satisfying some special properties." Why do we expect that $(G/N)^\guess$ satisfies these special properties?
Well, what are these special properties? For one, note that any $k$-variety $X$ one has that the natural map
$$X(k)\to X(\overline{k})$$
has image precisely $X(\overline{k})^{\Gamma_k}$ (where we set $\Gamma_k:=\mathrm{Gal}(\overline{k}/k)$). Said less obnoxiously, we have that $X(k)=X(\overline{k})^{\Gamma_k}$. In fact, for any finite extension $L$ of $k$ one has that
$$X(L)=X(\overline{k})^{\Gamma_L}$$
So for $(G/N)^\guess$ to have any chance of actually satisfying these `special properties' we better have
$$(G/N)^\guess(L)\overset{?}{=}(G/N)^\guess(\overline{k})^{\Gamma_L}$$
But, what does this explicitly say? This says that the map
$$G(L)/N(L)\to \left(G(\overline{k})/N(\overline{k})\right)^{\Gamma_L}$$
is a bijection. This is just not true.
Example 1: Let us think of the simplest possible case. Namely, let's consider $G=\mathrm{SL}_2$ and $N=Z(\mathrm{SL}_2)=\mu_2$. Then, we're asking whether the map
$$\mathrm{SL}_2(L)/\mu_2(L)\to (\mathrm{SL}_2(\overline{k})/\mu_2(\overline{k}))^{\Gamma_L}$$
is a bijection. But, let us consider a concrete example when $k=\mathbb{Q}=L$. Then, we're asking whether the map
$$\mathrm{SL}_2(\mathbb{Q})/\mu_2(\mathbb{Q})\to (\mathrm{SL}_2(\overline{\mathbb{Q}})/\mu_2(\overline{\mathbb{Q}}))^{\Gamma_\mathbb{Q}}$$
is a bijection. Indeed, let us note that $\begin{pmatrix}0 & i\\ i &0\end{pmatrix}\mu_2(\overline{\bb{Q}})$ is in $(\mathrm{SL}_2(\overline{\mathbb{Q}})/\mu_2(\overline{\mathbb{Q}}))^{\Gamma_\mathbb{Q}}$. Indeed, the $\Gamma_\bb{Q}$-action on $\begin{pmatrix}0 & i\\ i &0\end{pmatrix}$ factors through $\mathrm{Gal}(\mathbb{Q}(i)/\bb{Q})$ and the non-trivial element there acts by $\pm 1$. That said, evidently $\begin{pmatrix}0 & i\\ i &0\end{pmatrix}\mu_2(\overline{\bb{Q}})$ is not in the image of the map
$$\mathrm{SL}_2(\mathbb{Q})/\mu_2(\mathbb{Q})\to (\mathrm{SL}_2(\overline{\mathbb{Q}})/\mu_2(\overline{\mathbb{Q}}))^{\Gamma_\mathbb{Q}}$$
Thus, this map is not surjective. $\blacksquare$
So, we see that $(G/N)^\guess$ is kaput as a possible quotient of $G/N$ in $\SLAG_k$. But, the above also gives us an indication of how to fix $(G/N)^\guess$. Namely, it seems that a better idea might be to define, at least for finite extensions $L$ of $k$, the following:
$$(G/N)(L):=(G(\overline{k})/N(\overline{k}))^{\Gamma_L}$$
Of course, we have an obvious issue. While this seems like it fixes the issue of $(G/N)^\guess$ on finite extensions $L$ of $k$, we need to define $(G/N)(R)$ for all $k$-algebras $R$, not just the finite extension of $k$. That said, if we examine the relationship between $(G/N)^\guess$ and our $G/N$, on the category of finite extensions $L$ of $k$, we'll see what's going on.
Namely, let's suppose that we have a presheaf
$$\mathcal{F}:\left\{\begin{matrix}\text{Finite extensions}\\ L\text{ of }k\end{matrix}\right\}\to \mathsf{Set}$$
we can then extend this to a presheaf
$$\mathcal{F}:\left\{\bigsqcup_i \Spec(L_i)\right\}\to\mathsf{Set}$$
by setting
$$\mathcal{F}\left(\bigsqcup_i \Spec(L_i)\right):=\prod_i \mathcal{F}(L_i)$$
and this gives us an embedding of presheaves
$$P:\mathrm{PSh}\left(\left\{\begin{matrix}\text{Finite extensions}\\ L\text{ of }k\end{matrix}\right\}\right)\to \mathrm{PSh}\left(\left\{\bigsqcup_i \Spec(L_i)\right\}\right)$$
(in fact, it's image is exactly the 'Zariski sheaves').
We then described a process in our fixing of $(G/N)^\guess$ that can be applied to this general situation. Namely, we see that if we take a presheaf
$$\mathcal{F}:\left\{\begin{matrix}\text{Finite extensions}\\ L\text{ of }k\end{matrix}\right\}\to \mathsf{Set}$$
we can define another presheaf $\mathcal{F}^\sharp$ as follows:
$$\mathcal{F}^\sharp(L):=\mathcal{F}(\overline{k})^{\Gamma_L}$$
and this process has a very natural interpretation in
$$\mathrm{PSh}\left(\left\{\bigsqcup_i \Spec(L_i)\right\}\right)$$
Namely, one can check that
$$\left\{\bigsqcup_i \Spec(L_i)\right\}=\mathrm{Spec}(k)_\mathrm{et}$$
where the right hand side is the small etale site of $\mathrm{Spec}(k)$, and one can check that
$$P(\mathcal{F}^\sharp)=P(\mathcal{F})^\sharp$$
where the $\sharp$ on the right-hand side denotes the sheafification in the etale topology.
Thus, we see that the way to fix $(G/N)^\guess$, at least on $\Spec(k)_\et$, is to sheafify for the etale topology.
So then, one can interpret the issue of passing from needing to extend our fix
$$(G/N)(L):=((G/N)^\guess)^\sharp(L)$$
to a fix for $(G/N)^\guess$ on all $k$-algebras $R$ as a means to extend the sheafification on the small etale site to an operation on the big etale site. But, of course, we can just sheafify on the big etale site!
So, if $\Spec(k)_\Et$ denotes the big etale site on $\Spec(k)$, then we can interpret $(G/N)^\guess$ as a presheaf on $\Spec(k)_\Et$. We can then extend our fix of $(G/N)^\guess$ on $\Spec(k)_\et$ to $(G/N)^\guess$ on $\Spec(k)_\Et$ by declaring that $G/N$ is defined to be sheafification of $(G/N)^\guess$ on $\Spec(k)_\Et$.
It turns out that this definition does in fact produce a presheaf on $\Spec(k)_\Et$ which is representable. So, we define $G/N$ to be the scheme representing this presheaf!
Now, where does fppf come in to play? Well, it turns out that if one wants to extend the above construction to $\mathsf{LAG}_k$ to all linear algebraic groups over $k$ (i.e. removing the smoothness condition), then sheafifying with respect to the etale topology is the correct thing to do:
Example 2: Let $k$ denote a separably closed but not algebraically closed field of characteristic $p$. Consider the quotient sheaf $\mathcal{F}$ of $\mathbb{G}_{m,k}$ by $\mu_{p,k}$ for big etale topology. Note then that we have an exact sequence
$$0\to \mu_p(k)\to\mathbb{G}_{m,k}(k)\to \mathcal{F}(k)\to H^1_\et(\Spec(k),\mu_{p,k})$$
Since $k$ is sparably closed one hs that $H^1_\et(\Spec(k),\mu_p)=0$ and so $\mathbb{G}_{m,k}(k)\to \mathcal{F}(k)$ is surjective.
That said, let $\mathcal{G}$ denote the sheafification of the quotient presheaf for $\mathbb{G}_{m,k}$ by $\mu_{p,k}$ for the big fppf topology. Then, we have an exact sequence
$$1\to \mu_p(k)\to \mathbb{G}_{m,k}(k)\to \mathcal{G}(k)\to H^1_\text{fppf}(\Spec(k),\mu_{p,k})\to H^1_\mathrm{fppf}(\Spec(k),\bb{G}_{m,k})$$
and since
$$H^1_\text{fppf}(\Spec(k),\mu_{p,k})=k^\times/(k^\times)^p\ne 0,\qquad H^1_\text{fppf}(\Spec(k),\mathbb{G}_{m,k})=0$$
we see that the map $\mathbb{G}_{m,k}(k)\to\mathcal{G}(k)$ is not surjective.
Since schemes are sheaves for the big fppf topology, we see that $\mathcal{G}$ is the more correct object to take. $\blacksquare$
Thus, all in all, we see that the fppf sheafification is the correct object to consider if one thinks naively of taking quotients in the perspective 2. of schemes and tries to fix the 'obvious issue' to make the quotient presheaf satisfy the 'special properties' to be a scheme. The fact that this fppf sheafification is enough to actually give all all the 'special properties' is the non-trivial result.
To explicitly answer the question you're asking, the quotient that is considered in [1] is precisely the quotient in the category of fppf sheaves. I think you yourself can see that. If you want to see some more discussion to this end, see the introduction to [2, Chapter 1].
Let me just say one word about trying to find quotients in the sense of 1. Namely, trying to take quotients in the category of locally ringed spaces. Such quotients exist in some situations, but aren't always schemes. One good situation one can consider is when $G$ is affine and $N$ is reductive. Then, $G/N$ exists in the sense of GIT quotients (e.g. see [3, Chapter 0]) which is stronger than in the sense of locally ringed spaces, and in fact is just $\Spec(O(G)^N)$ where
$$O(G)^N:=\left\{a\in O(G):\begin{matrix}a\text{ maps to }a\otimes 1\text{ under the comultiplication}\\\text{map }O(G)\to O(G)\otimes_k O(N)\end{matrix}\right\}$$
and this agrees with the definition of fppf quotient $G/N$ as in [1] (e.g. see [6, Theorem 4.35]).
Question 2: Let us recall that a morphism of schemes $f:X\to Y$ is called faithfully flat if it is flat and surjective. If $X$ and $Y$ are affine, then $f$ is faithfully flat if and only if $f^\ast:\mathcal{O}(Y)\to \mathcal{O}(X)$ is faithfully flat (e.g. see [4, Tag02JZ]).
We then have the following:
Fact: Let $f:X\to Y$ be flat map of finite type $k$-schemes. Then, $f$ is faithfully flat if and only if $f$ is surjective as a map of sheaves on $\Spec(k)_\mathrm{FPPF}$ (the big fppf site on $\Spec(k)$.
Proof: Suppose first that $f:X\to Y$ is faithfully flat. Let $S$ be any $k$-scheme. We then need to show that for all element $s:Y\to S$, which is an element $Y(S)$ there exists a faithfully flat finitely presented map $S'\to S$ such that the image of $s$ in $Y(S')$ is in the image of $X(S')$. But, note that if we set $S':=X\times_Y S$ then $S'\to S$ is faithfully flat and finitely presented since $X\to Y$ is. Moreover the image of $s$ under the map $Y(S)\to Y(S')$ is the composition
$$S'\to S\to Y$$
but, by definition if fibered product, this is equal to the composition
$$S'\to X\to Y$$
and thus is in the image of $X(S')\to Y(S')$.
Conversely, suppose that $f:X\to Y$ is surjective as a map of sheaves on $\Spec(k)_\text{FPPF}$. We need to show that $f$ is surjective. But, let $y$ be a point of $Y$. Then, $y$ naturally defines an element of $Y(k(y))$. Thus, by assumption, there exists a faithfully flat finitely presented cover $S'\to \Spec(k(y))$ such that the image of $y$ under $Y(k(y))\to Y(S')$ is in the image of $X(S')\to Y(S)$. But, note that this implies that there is a map
$$g:S'\to X$$
such that the diagram
$$\begin{matrix}S' & \to & X\\ \downarrow & & \downarrow \\ \Spec(k(y)) & \to & Y\end{matrix}$$
commutes. In particular, it's easy to see that this implies that $f(g(s'))=y$ for any point $s'$ of $S$. Thus, $f$ is surjective. $\blacksquare$
This is the relationship between faithfully flat maps and maps of fppf sheaves. Note that perhaps the assumption that $f:X\to Y$ in the above is flat annoys you, but note that if $f:G\to H$ is surjective homomorphism of smooth finite type $k$-group schemes, then it's automatically flat. The reason is that by Miracle Flatness (e.g. see [4, Tag00R4]) one needs only check that $f^{-1}(h)$ is equi-dimensional for all $h\in H(\overline{k})$. But, one can check that if $g\in f^{-1}(h)$ then the translation map
$$t_g:(\ker f)_{\overline{k}}\to f^{-1}(h)$$
is an isomorphism.
Question 3: The correct notion in general is that of the fppf quotient of sheaves. By Chevalley's theorem (e.g. see [5]) every finite type group scheme is an extension of an affine group scheme by an abelian variety so, at least morally, the only other class of group schemes of interest over fields are abelian varieties, and the notion of quotient there is just the fppf quotient which, by the remark at the end of the answer to Question 2, is just a surjective map. If you want to check this, I suggest reading the contents of [6, Chapter 4].
One also gets that, with this notion of quotient, that the category of all commutative finite type $k$-group schemes is abelian (e.g. see [6, Theorem 4.41]).
References:
[1] Milne, J.S., 2017. Algebraic groups: The theory of group schemes of finite type over a field (Vol. 170). Cambridge University Press.
[2] Conrad, B., 2014. Reductive group schemes. Autour des schémas en groupes, 1, pp.93-444.
[3] Mumford, D., Fogarty, J. and Kirwan, F., 1994. Geometric invariant theory (Vol. 34). Springer Science & Business Media.
[4] Various authors, 2020. Stacks project. https://stacks.math.columbia.edu/
[5] Conrad, B., 2002. A modern proof of Chevalley's theorem on algebraic groups. JOURNAL-RAMANUJAN MATHEMATICAL SOCIETY, 17(1), pp.1-18.
[6] Edixhoven, Bas and van der Geer, Gerard and Moonen, Ben. *Abelian varieties. http://gerard.vdgeer.net/AV.pdf
