As explained by a previous answer, what you want is a linear transformation $A:X\rightarrow X$ on finite dimensional inner product space $X$ that preserves amplitude of vectors, and angle between vectors. Namely,
- $\|Ax\|=\|x\|,\ \forall x\in X$
- $\langle Ax, Ay\rangle=\langle x,y\rangle,\ \forall x,y\in X$
In fact, we can introduce the following additional conditions,
- $A^\ast A=I$
- $AA^\ast=I$
and show that 1, 2, 3 and 4 are equivalent (see Theorem 2.9 in Functional Analysis by Bachman and Narici, Dover Publication, 2000).
This happen to be the definition of the unitary matrix. Note that the determinant of unitary matrix could be $1$ or $-1$. The only extra condition to throw in to exclude “flipping” so that we are left with “rotation” only is
- $\det(A)=1$.
So back to your question, how can one generate a matrix that rotates the underlying vector space by an arbitrary angle? Here is an idea.
Start with the identity matrix $I$, and suppose the desired rotation rotates the $j^\text{th}$ column of $I$ denoted $I_j$ to $v$ (with $\|v\|=1$ obviously). Assuming $\langle I_j, v\rangle\neq 0$, then what you can do is
- Move $I_j$ to the first column and replace it with $v$
- Perform Gram-Schmidt orthogonalization on the resulted matrix, starting from the $2^\text{nd}$ column
- Swap the $1^\text{st}$ column and the $j^\text{th}$ column to resume the original order
The resulted matrix $A$ satisfies $A^\ast A=I$ by the nature of Gram-Schmidt process, therefore is a rotation. And it transforms $I_j$ to $v$, thus giving you the desired rotation.