In Spanier's book of algebraic topology, there is a definition of "categories" which entails "a class of objects".
I realize that the vagueness of the concept of "class of objects" is exactly used instead of "set of sets" because we want to avoid certain paradoxes of set theory.
Still, I am wondering, is there a more formal definition, or axiomatization, of what a "class of objects" mean in the concept of a category?
The notion of a class is defined rigorously in Von Neumann–Bernays–Gödel set theory, which is a conservative extension of ZFC.
Basically, you can form classes of sets using unrestricted comprehension, and you can freely take subclasses, images of classes under functions, and use the Axiom of Choice on classes of sets. However, no proper class is allowed to be an element of anything -- if a class $C$ is an element of a class $D$, then $C$ must be a set. In particular, there is no class of all classes, although there is a class of all sets.
It is just that, a class. That is: Given $A$, it is either true or false that $A\in\operatorname{Obj}(\mathcal C)$. Put differently, we may assume that "is an object of category $\mathcal C$" is a valid predicate.
One way to approach this is to start with $\in $, and build up a "universe" $U$, using a standard naive set theoretic construction. $U$ is defined in such a way so as to ensure that all the usual operations of set theory applied to elements of $U$, always produce elements of $U$.
Now, we say that $u$ is $\textit {small}\Leftrightarrow u\in U$. Otherwise $u$ is said to be $\textit {large}$. Note that there are large sets: $U$ itself is large because of course $U\notin U$.
Once $U$ is defined, then you get most of the properties you want in order to do "ordinary" mathematics, such as $f:u\to v$ is small whenever $u$ and $v$ are small.
A $\textit {class}$ is defined to be any subset $S\subset U$. Note that since, by construction of $U$, $x\in u\in U\Rightarrow x\in U$, every element of $U$ is a subset of $U$ so every small set is a class. The classes that do not belong to $U$ are called $\textit {proper classes}$, $U$ being an example of a proper class.
From here, one defines Cat, the category of small categories, Cat', the category of large categories, etc.
This approach has some disadvantages. For example, if you define Cls to be the category of all classes then the set of objects of Cls is $\mathcal P(U)$ which is not a class, since its cardinality is strictly greater than that of $U$.
There are more sophisticated approaches to defining categories, but I am not expert enough to elaborate on them.
