Is it possible to use abstract groups to generalize DSA, ECDSA and EdDSA signature creation and verification?
Yes. But towards that it is essential to systematically distinguish:
- the abstract finite group used, that I'll note $(\mathbb G,\mathbin{\dot+})$, of prime order $q$, with elements including a generator $G$ and neutral element $\mathcal O$.
- the well-known finite field of integers modulo $q$ that I'll note $(\mathbb F_q,+,\cdot)$.
The group's internal law $\mathbin{\dot+}$ is the question's Operation(). On top of that primitive the question's Scale(), which is known as scalar multiplication, can be defined and efficiently implemented as: $\forall\,i\in\mathbb F_q$ and $\forall\,U\in\mathbb G$
$$i*U=\begin{cases}
\mathcal O&\text{for }i=0\\
U&\text{for }i=1\\
V\mathbin{\dot+}V\text{ with }V=(i/2)*U&\text{for other even }i\\
((i-1)*U)\mathbin{\dot+}U&\text{for other }i\\
\end{cases}$$
The additive subgroup $(\mathbb F_q,+)$ is isomorphic to $(\mathbb G,\mathbin{\dot+})$ thru transforming $i\in\mathbb F_q$ into $i*G\in\mathbb G$. This bijection can be computed in time polynomial in $\log q$ only in the forward direction. The DLP and CDH assumptions are assumed to hold in $\mathbb G$, but not in any subgroup (additive or multiplicative) of $\mathbb F_q$. It holds $\forall\,i,j\in\mathbb F_q$ and $\forall\,U,V\in\mathbb G$
$$\begin{align}
(i+j)*U&=(i*U)\mathbin{\dot+}(i*U)\\
j*(U\mathbin{\dot+}V)&=(j*U)\mathbin{\dot+}(j*V)\\
(i\cdot j)*U&=i*(j*U)
\end{align}$$
And, depending on the algorithm, further building blocks are necessary.
Looking at these algorithms I cannot find any similarities that I could abstract over algebraic groups
That's possible for DSA and ECDSA, within the above framework. They differ only in the choice of the group $\mathbb G$, and of an ad-hoc function $f:\,\mathbb G\setminus\{\mathcal O\}\to\mathbb F_q$.
- DSA: the group is a Schnorr group of prime order $q$, obtained as the subgroup of order $q$ of the multiplicative group modulo a prime $p$, with $q$ dividing $p-1$, and $p\gg q$. Elements $U\in\mathbb G$ are $U\in\mathbb F_p$ satisfying $U^q=1$ computed in $\mathbb F_p$, and the group law for $\mathbb G$ is defined as $U\mathbin{\dot+}V=U\cdot V$ with the right-hand size computed in $\mathbb F_p$ (this is why DSA's group is typically noted multiplicatively as in the question, when here we note it as $\dot+\,$ for the purpose of abstracting the group). The ad-hoc function $f$ is $f(U)=U\bmod q$ with the integer representative# of $U$ in $[0,p)$.
- ECDSA: the group is an Elliptic Curve group, obtained as the subgroup of prime order $q$ of an Elliptic Curve over some finite base field. See sec1 for the construction of the group and it's "point addition" internal law $\mathbin{\dot+}$, sec2 for some common parameters. Restricting our description of ECDSA to the base field being a prime field $\mathbb F_p$ for some suitable $p$, an element of the full elliptic curve other than $\mathcal O$ is a pair $(X_U,Y_U)$ of integers in $\mathbb F_p$ satisfying the curve's equation, and those in $\mathbb G$ are those with $q*U=\mathcal O$. The ad-hoc function $f$ is $f(U)=X_U\bmod q$ with the integer representative# of $X_U$ in $[0,p)$.
DSA and ECDSA can then be described in a unified manner as:
- Signer and verifier agree on the group $(\mathbb G,\mathbin{\dot+})\,$: nature, parameters including order $q$, generator $G$, representation of an element of $\mathbb G$ (for the public key) and of $\mathbb F_q$ (for signature components); an ad-hoc function $f:\,\mathbb G\setminus\{\mathcal O\}\to\mathbb F_q\,$; and a hash $\operatorname{Hash}:\,\{0,1\}^*\to\mathbb F_q$.
- Signer generates (private, public) keys pair $(x,Y)$ with $x$ a secret random in $\mathbb F_q\setminus\{0\}$, $Y=x*G$, and communicates $Y$ to verifier with integrity.
- For each message $m\in\{0,1\}^*$, signer
- computes $e=\operatorname{Hash}(m)$
- generates uniformly random secret $k\in\mathbb F_q\setminus\{0\}$
- computes $r=f(k*G)$ in $\mathbb F_q$ and if $r=0$ reiterates from 2
- computes $s=k^{-1}\cdot(e+r\cdot x)$ in $\mathbb F_q$ and if $s=0$ reiterates from 2
- outputs signature $(r,s)$ and sends it to verifier
- For each alleged message $m\in\{0,1\}^*$ and signature $(r,s)$, verifier
- checks $r\in\mathbb F_q\setminus\{0\}$ and $s\in\mathbb F_q\setminus\{0\}$, rejects signature otherwise
- computes $e=\operatorname{Hash}(m)$ in $\mathbb F_q$
- computes $w=s^{-1}$ in $\mathbb F_q$
- computes $a=w\cdot e$ and $b=w\cdot r$ in $\mathbb F_q$
- computes $X=(a*G)\mathbin{\dot+}(b*Y)$ in $\mathbb G$ and if $X=\mathcal O$ rejects signature
- accepts or rejects signature w.r.t. $m$ according to if $f(X)=r$ or not.
Argument that valid signatures are always accepted: in verifier's step 6, expanding:
$$\begin{align}
f(X)&=f\bigl((a*G)\mathbin{\dot+}(b*Y)\bigr)&&\text{per ver. 5}\\
&=f\bigl(((w\cdot e)*G)\mathbin{\dot+}((w\cdot r)*(x*G))\bigr)&&\text{per ver. 4 and gen.}\\
&=f\bigl((w\cdot(e+r\cdot x))*G\bigr)&&\text{per algebra in }\mathbb F_q\text{ and }\mathbb G\\
&=f\bigl((s^{-1}\cdot(e+r\cdot x))*G\bigr)&&\text{per ver. 3}\\
&=f\bigl(((k^{-1}\cdot(e+r\cdot x))^{-1}\cdot(e+r\cdot x))*G\bigr)&&\text{per sig. 4}\\
&=f\bigl(k*G\bigr)&&\text{per algebra in }\mathbb F_q\\
&=r&&\text{per sig. 3}
\end{align}$$
This proof is independent of what the ad-hoc function $f$ is! Any function $f$ making $g:\,\mathbb F_q\setminus\{0\}\to\mathbb F_q$ would do from the standpoint of allowing verification. Thus we could construct $f$ by hashing to $\mathbb F_q$.
With $f$ modeled as a hash and if DDH holds in $\mathbb G$, it can be proven that the signature system is sEUF-CMA. That appears to stand for DSA, however ECDSA is not sEUF-CMA (but still appears to be EUF-CMA), which illustrates the limited confidence we should have in the conclusion of such proof, because it relies on an unwarranted assumption about $f$. Problem is that the restriction to X coordinate in the ad-hoc function $f$ of ECDSA implies $f(\dot-U)=U$, which makes it trivial to break second preimage of $f$. There are other regularities in the ad-hoc functions $f$ used in DSA and ECDSA, and arguments that they can't harm fly above my head (that's not to suggest they may not hold, and I'm even ready to accept that the regularity there is in the ad-hoc functions $f$ may improve security).
Especially interested in how can I get these $X_x$ or $R_x$ in ECDSA
You don't get that straight from algebra. That's part of the construction of the ad-hoc function $f:\,\mathbb G\setminus\{\mathcal O\}\to\mathbb F_q$.
EdDSA differs: a reasonably accurate description is possible in term of $\mathbb G$ with a representation of an element $U$ as a bitstring $\underline U$, the field $\mathbb F_q$, and a hash, without resorting to an ad-hoc function $f$. EdDSA internally builds an analog to $f$ on top of that $U\mapsto\underline U$ transformation and a hash; also this analog has additional inputs.
Another view is that EdDSA is a modern-textbook Schnorr signature (in the variant where the signature contains a representation of a group element rather than a short hash as is the case in the original Schnoor signature), instantiated in a subgroup of prime order $q$ of a twisted Edwards elliptic curve group on a prime field.
Note: The question does not accurately describe EdDSA (for example the creation of the public key actually involves a hash). For a mathematically-oriented description of the original EdDSA, see this (caveat: it uses $\ell$ where we have $q$). For the "standard", see RFC 8032.
# $\mathbb F_p$ is $\mathbb Z/p\mathbb Z$. That is, an element $U$ of field $\mathbb F_p$ for prime $p$ is an equivalence class of the equivalence relation on the ring of integers $\mathbb Z$ defined by: $u$ is equivalent to $u'$ when $u-u'$ is a multiple of $p$, noted $u\equiv u'\pmod p$. These $u$ and $u'$ are called integer representatives of $U$. They are elements of the set $U$, which contains infinetly many. To define $U\bmod q$ for $q\ne p$, DSA and ECDSA make it $u\bmod q$ for the uniquely defined $u\in\,[0,p)\cap U$. Using $u\in\,[(1-p)/2,(p-1)/2]\cap U$ leads to a different ad-hoc function $f$.
