Your question gets at the whole point of choosing a single foundational basis for mathematics, like ZFC set theory.
Let me tell you a little fable. Imagine a world where mathematicians work with separate axiomatic systems for geometry (e.g., Euclid's axioms), real numbers (e.g., the theory of real closed fields), natural numbers (e.g., Peano arithmetic), etc. etc.
Let's say I'm a geometer in this world. I go about my days proving theorems of geometry from the axioms laid down for geometry. But then some things start happening that trouble me.
- My friend the number theorist has proven a theorem about natural numbers from Peano arithmetic that would be useful to prove a theorem in geometry. Is it ok to use their theorem? It follows from a different axiom system...
- My friend who studies real numbers points out that concepts of geometry can be recovered "analytically" by viewing $\mathbb{R}^2$ as the plane. They start proving theorems of geometry from the axioms of real closed fields that I haven't yet proved on the basis of Euclid's axioms. Is my plan "the same" as their plane? Do I trust their theorems? Can we be confident we won't someday prove contradictory results about plane geometry?
- Some of my geometer friends point out that if we replace the parallel postulate by variants of it, we get other geometric theories that seem useful and interesting. Do we have to split geometry into (at least) three branches, which work with different axiomatic systems? Or can we somehow study them all on the same footing?
- When studying geometry, I notice that it's really useful to be able to talk about sets of points and functions between sets of points, and sometimes nontrivial questions arise about how to manipulate sets and functions behave, which just aren't addressed by my axioms of geometry (which are about things like individual points and lines). In fact, the same nontrivial questions are arising for my friends who study other branches of mathematics. Do we need extra assumptions governing these sets and functions? Should all the branches of mathematics adopt the same assumptions, or are different assumptions more suited to different axiom systems?
As the number of subfields of mathematics and kinds of mathematical objects grows, issues like the ones above proliferate and become impossible to ignore.
The benefit of choosing a foundation for mathematics is that all these issues get resolved. Everyone agrees to pick some primitive objects and relations between these objects and some basic axioms governing them. (In the case of ZFC, the primitive objects are called sets and the primitive relation is called $\in$. There are other options, e.g. types, or categories, but I will focus on ZFC.) Now all the other basic objects of mathematics are defined to be particular sets, and the axiomatic systems for other classes of mathematical objects turn into definitions.
For example, in ZFC we define $0$ to be the empty set $\varnothing$, we define an operation $S$ on sets called "successor", we define $1 = S(0)$, $2 = S(S(0))$, etc., and we prove that there is a smallest set called $\mathbb{N}$ containing $0$ and closed under successor. This becomes the set of natural numbers in which number theory happens. It happens that we can prove that $\mathbb{N}$ satisfies all the axioms of Peano arithmetic (PA), but this is a little beside the point: we now can do number theory working from the axioms of ZFC, not from the specialized theory PA.
Similarly, we construct the integers $\mathbb{Z}$ from $\mathbb{N}$, construct the rational numbers $\mathbb{Q}$ from $\mathbb{Z}$, and construct the real numbers $\mathbb{R}$ from $\mathbb{Q}$ (with explicit set-theoretic constructions, e.g., the real numbers is the set of Dedekind cuts in $\mathbb{Q}$). Again $\mathbb{R}$ happens to be a real closed field, but this is a little beside the point: we are no longer constrained by first-order reasoning from the axioms of real-closed fields. Our proofs about $\mathbb{R}$ can use reasoning about arbitrary subsets, arbitrary functions, facts from number theory, etc. etc. because everything lives in the same mathematical universe of sets.
Let me respond now to a few of the things you wrote in the question.
I'm not sure how matrix (or more generally elements of vector space) is rigorously defined in ZFC set theory.
ZFC's success as a foundation for mathematics comes down to the fact that almost every mathematical object people want to work with can be encoded in set theory, and almost every mathematical proof can, in principle, be reduced to the axioms of ZFC. If you want to delve into the details of how this encoding works, you need to pick up a book on set theory!
These definitions are "governed" by ZFC theory at their conception, then there are other rules (like FTC, Mean value theorem for real numbers; other rules for elements of vector space)
I'm confused about why you call FTC, MVT, etc. "rules". These are theorems (yes, they can be proved about the set $\mathbb{R}$ from the axioms of ZFC!).
Then I know that FTC is not a theorem of RCF. I am surprised because why a rule that applied to a structure doesn't hold anymore when we study the general theory (RCF) with the same structure being a model ?
The problem is that RCF is a first-order theory. One restriction of a first-order theory is that quantifiers only range over objects in the domain (in this case real numbers), not over functions and subsets. So it's not that RCF and ZFC somehow disagree about whether FTC is true. It's that FTC is not even expressible in the language of RCF. You have to be able to write something down before you can ask if it's true!
