If you wish, you can regard all physical statements to be only about numbers and units only to be a means to formulate the statements, which do not relate to the mathematics involved. For example, we could reformulate
If we weld a wire of length $x$ to a wire of length $y$, the resulting wire has length $x+y$. (1A)
such that it does not use any arithmetics involving units as follows:
If we have one wire of length $x∈ℝ^+$ in reference to some unit length and another wire of length $y∈ℝ^+$ in reference to the same unit length, and we weld those wires together, the resulting wire has the length $x+y$ in reference to that unit length. (1B)
In another example, we could reformulate:
If a wire has a lenth $x$ at temperature $t$, there is a constant $α$ such that its length at any temperature $s$, the wire has a length of $x·(1+α(s-t))$. (2A)
A wire has a length $x∈ℝ^+$ in reference to some unit length at a temperature $t∈ℝ^+$ in reference to some unit temperature. Then there exists an $α∈ℝ$ (which depends on the unit length and unit temperature, but not on $x$, $s$ or $t$) such that at any temperature $s∈ℝ^+$ in reference to the aforementioned unit temperature, the wire has a length $x·(1+α(s-t))$ in reference to the above. (2B)
Of course the above only works for reasonable definitions of “a wire has length $x∈ℝ^+$ in reference to some unit length” and similar, which complies with the arithmetic of real numbers. For example, we need:
If a length $L$ is $x∈ℝ^+$ in reference to some unit length $A$ and $y∈ℝ^+$ in reference to some unit length $B$, then the length $A$ is $x/y$ in reference to the unit length $B$.
Without this, the statement 1B would not be universally true for all unit lengths.
That such definitions exist is an extremely well established empirical fact, but nothing more. (Historically it’s of course rather that the real numbers were abstractions of the arithmetic properties of physical values and one could take the point of view that mathematics is justified by the very well established empirical observation that the axioms of some prominent algebraic structures comply with reality.)
Making use of this observation, we can introduce an arithmetics for physical values (i.e., real numbers equipped with a unit) and, by equipping constants such as $α$ in 2B to appropriate units, we can arrive at the usual formulation of physics which uses these arithmetics as a notational shortcut, if you so wish.
If we moreover only use units that are expressed in terms of some algebraically independent base units (such as the SI base units), we can now omit writing down the units and can deduce the unit of our results from their dimension. E.g., if we strictly work in the SI system, we know that a velocity has the unit $\frac{\text{m}}{\text{s}}$ (as this is the only unit for velocities in this system). This is prominently done in high-energy physics, where usually $\hbar$, $c$ and $\text{MeV}$ are used as base units and $\hbar$ and $c$ are omitted (which is usually written as the cringeworthy $\hbar=c=1$).