I'm currently working to grasp the concepts of (continuous) functional calculus, aiming to prove the spectral theorem for bounded self-adjoint operators as outlined in "Introduction to Hilbert space and the theory of spectral multiplicity" by Paul Halmos and "Quantum Theory for Mathematicians" by Brian Hall. The specific definition of continuous functional calculus I'm exploring is as follows:
Theorem 8.3 ("Quantum Theory for Mathematicians" - Brian Hall):
Let $A \in \mathcal{B}(H)$ be a self-adjoint operator. Then there exists a unique bounded linear map from $\mathcal{C}(\sigma(A); \mathbb{R})$ to $\mathcal{B}(H)$, denoted as $f \mapsto f(A)$, such that when $f(\lambda)=\lambda^{m}$, we have $f(A)=A^{m}$. This mapping, denoted as $f \mapsto f(A)$, where $f\in \mathcal{C}(\sigma(A);\mathbb{R})$, is commonly referred to as the (real-valued) functional calculus for the operator $A$."
My questions are mainly concerned with the choice of domain of $f$:
Why is the domain of $f$ restricted to $\sigma(A)$ (the spectrum of $A$)? Why not simply use $\mathbb{R}$ as the domain, given that $A$ is self-adjoint and $\sigma(A) \subset \mathbb{R}$?
In Halmos' book, for a polynomial $p(\lambda)=\sum_{j=0}^{n}\alpha_{j}\lambda^{j}$ and an operator $A$ on a Hilbert space, he defines $p(A)$ to be $p(A)= \sum_{j=0}^{n}\alpha_{j} A^{j}$. To me, this seems acceptable, assuming that the domain and codomain of $p(A)$ and $A$ are the same. However, am I correct that when considering more complex functions like Borel or holomorphic functions, we need to be cautious about our choice of domain? Unfortunately, this aspect is not generally explained.
Update: The proof makes use of the Stone-Weierstrass theorem, i.e. that polynomials are dense in $C(K;\mathbb{R})$, where $K$ is a compact set. Thus we could use any compact $K \supset \sigma(A)$. However, since $\sigma(A)$ is compact, we wouldn't gain anything as pointed out in the answers below.