I will use "inner product" rather than "scalar product." (Edit: now fixed) The "scalar product" is often the name given to the map $K\times V\to V$ that is part of the definition of a vector space.
I will also note that I disagree with your definition of "preserves inner product". (Edit: Now fixed to “preserves norm”). In my opinion, it should be
$$\langle Ox,Oy\rangle = \langle x,y\rangle\qquad\text{for all }x,y\in V.$$
Instead, what you have is "preserves norm": since your equations amount to $\lVert Ox\rVert^2 = \lVert x\rVert^2$, and this yields $\lVert Ox\rVert = \lVert x\rVert$.
That said: I will treat both orthogonal and unitary together. I will use $U^*$ to denote the adjoint of $U$; the matrix is unitary if we are working over $\mathbb{C}$ and $U^*U = UU^* = I$, and is orthogonal if we are working over $\mathbb{R}$ and $U^*U=UU^*=I$. For matrices, the operation amounts to conjugate transpose over $\mathbb{C}$ and to transpose over $\mathbb{R}$.
Theorem. Let $V$ be a finite dimensional inner product space, and let $U$ be an operator on $V$. The following are equivalent:
- $UU^*=U^*U=I$.
- $\lVert Ux\rVert = \lVert x\rVert$ for all $x\in V$.
- $\langle Ux,Uy\rangle = \langle x,y\rangle$ for all $x,y\in V$.
- For any orthonormal basis $\beta$ of $V$, $U(\beta)= \{Ux\mid x\in\beta\}$ is an orthonormal basis of $V$.
- There exists an orthonormal basis $\gamma$ of $V$ such that $U(\gamma) = \{Ux\mid x\in \gamma\}$ is an orthonormal basis for $V$.
Proof. $1\implies 2$: $\lVert Ux\rVert^2 = \langle Ux,Ux\rangle = \langle x,U^*Ux\rangle = \langle x,x\rangle = \lVert x\rVert^2$, hence $\lVert Ux\rVert = \lVert x\rVert$.
$1\implies 3$: as above, $\langle Ux,Uy\rangle = \langle x,U^*Uy\rangle = \langle x,y\rangle$.
$2\implies 3$: Let $x$ and $y$ be arbitrary vectors. Then $\lVert A(x+y)\rVert^2 = \lVert x+y\rVert^2$. We therefore have:
$$\begin{align*}
\langle A(x+y),A(x+y)\rangle &= \langle Ax,Ax\rangle + \langle Ax,Ay\rangle + \langle Ay,Ax\rangle + \langle Ay,Ay\rangle\\
&= \lVert Ax\rVert^2 + \lVert Ay\rVert^2 + \langle Ax,Ay\rangle + \overline{\langle Ax,Ay\rangle}.\\
\langle x+y,x+y\rangle &=\langle x,x + \langle x,y
\rangle + \langle y,x\rangle + \langle y,y\rangle\\
&= \lVert x\rVert^2 + \lVert y\rVert^2 + \langle x,y\rangle + \overline{\langle x,y\rangle}.
\end{align*}$$
Now, since $\lVert Ax\rVert = \lVert x\rVert$ and $\lVert Ay\rVert = \lVert y\rVert$, we conclude that
$$\langle Ax,Ay\rangle + \overline{\langle Ax,Ay\rangle} = \langle x,y\rangle + \overline{\langle x,y\rangle}.$$
If we are working over the real numbers, this gives $2\langle Ax,Ay\rangle = 2\langle x,y\rangle$, and we conclude $\langle Ax,Ay\rangle = \langle x,y\rangle$, as desired.
If we are working over the complex numbers we conclude that $2\mathrm{Re}(\langle Ax,Ay\rangle) = 2\mathrm{Re}(\langle x,y\rangle)$, so the real parts are equal.
Now, replace $x$ with $ix$, we get
$$i\langle Ax,Ay\rangle + \overline{i\langle Ax,Ay\rangle} = i\langle x,y\rangle + \overline{i\langle x,y\rangle}$$
and therefore,
$$i\left(\langle Ax,Ay\rangle - \overline{\langle Ax,Ay\rangle}\right) = i\left(\langle x,y\rangle - \overline{\langle x,y\rangle}\right),$$
and hence $-2\mathrm{Im}(\langle Ax,Ay\rangle) = -2\mathrm{Im}(\langle x,y\rangle)$. Thus, the imaginary parts of $\langle Ax,Ay\rangle$ and $\langle x,y\rangle$ also agree, and we conclude that $\langle Ax,Ay\rangle = \langle x,y\rangle$, as desired.
$3\implies 2$: immediate.
$3\implies 1$: We consider the operator $I-U^*U$. Let $y\in V$. Then for any $x\in V$ we have
$$\begin{align*}
\langle x,(I-U^*U)y\rangle &= \langle x,y\rangle - \langle x,U^*Uy\rangle\\
&= \langle x,y\rangle - \langle Ux,Uy\rangle\\
&= \langle x,y\rangle - \langle x,y\rangle\\
&= 0.
\end{align*}$$
As this holds for all $x\in V$, it follows that $(I-U^*U)y = 0$. This holds for each $y\in V$, so $I=U^*U$.
Now, this means that $U$ is one-to-one, and hence invertible; therefore, $U^*=U^{-1}$, so $UU^* = I$ as well.
So now we know that 1, 2, and 3 are equivalent. This already answers your question.
$3\implies 4$: Note that 2 implies that $U$ is one-to-one, and since 3 implies 2, we know that $U$ is one-to-one. Thus, $U(\beta)$ has $|\beta|=\dim(V)$ elements.
If $x,y\in\beta$, then $\langle x,y\rangle = \delta_{xy}$ (Kronecker's delta). Therefore, for $Ux,Uy\in U(\beta)$, $\langle Ux,Uy\rangle = \langle x,y\rangle = \delta_{xy} = \delta_{U(x),U(y)}$. Thus, $U(\beta)$ is an orthonormal subset of $V$ with $\dim(V)$ elements, and hence an orthonormal basis.
$4\implies 5$: immediate.
$5\implies 3$: Let $\gamma = [v_1,\ldots,v_n]$ and express $x$ and $y$ in terms of $\gamma$. Say
$$x = \sum_{i=1}^n \alpha_iv_i,\qquad y=\sum_{i=1}^{n}\beta_iv_i.$$
Then
$$\langle x,y\rangle = \left\langle \sum_{i=1}^n\alpha_iv_i ,\sum_{j=1}^n \beta_jv_j\right\rangle = \sum_{i=1}^n\alpha_i\overline{\beta_j},$$
where $\overline{\beta_j}$ is the complex conjugate of $\beta_j$ if $V$ is complex, and $\beta_j$ if $V$ is real.
On the other hand, we have
$$\begin{align*}
\langle Ux,Uy\rangle &= \left\langle \sum_{i=1}^n\alpha_iUv_i,\sum_{j=1}^n\beta_jUv_j\right\rangle\\
&= \sum_{i=1}^n\sum_{j=1}^n \alpha_i\overline{\beta_j}\langle Uv_i, Uv_j\rangle\\
&= \sum_{i=1}^n\sum_{j=1}^n \alpha_i\overline{\beta_j}\delta_{ij}\\
&= \sum_{i=1}^n \alpha_i\overline{\beta_i}\\
&= \langle x,y\rangle.
\end{align*}$$
The third equality holds because $U(\gamma)$ is an orthonormal set, so $\langle Uv_i,Uv_j\rangle = \delta_{ij}$.
This proves that 3, 4, and 5 are equivalent. As 3 was already known to be equivalent to 1 and 2, this proves the Theorem. $\Box$
