Well, the point is that when you have some objects you can define operations between then at your will. These operations can be meaningful or not and that's the point on the axioms of a vector space, they are meaningful and useful.
You can define vectors in geometry as equivalence classes of oriented line segments. Then you define the sum using the paralellogram rule and define the multiplication by scalar as stretching or compressing the segment. You have defined operations, and those properties are obeyed just because the operations you've defined behave linearly. What does that mean?
Linear Algebra is devoted to study not "vectors and matrices" as many people think at first when they begin the course. Linear Algebra is devoted to study linear structures. There are many things linear in mathematics. For instance, take two functions $f, g: \mathbb{R} \to \mathbb{R}$ and define their sum to be the function $f+g$ such that $(f+g)(x)=f(x)+g(x)$ and define their multiplication by a scalar $\lambda$ to be the function $\lambda f$ such that $(\lambda f)(x) = \lambda f(x)$.
If you go now and check the axioms of a vector space, they will all hold. But wait, functions in such abstraction cannot be seen as oriented line segments! And that's right, however if you introduce these operations they behave linearly. Linearity is a property of greatest importance, we can see it everywhere in mathematics, so a systematic study of it is indeed needed. Linear Algebra provides this study very well.
So you come and ask: "ok, but why then we call a set with operations like that a vector space?" and the answer is simple, because usually we use the "geometrical vectors" to jump into linear algebra, so that the objects with linearity that we "borrow" the property to generalize as vectors. It's just a matter of terminology. There are authors like Kostrikin that prefer to call a vector space as a linear space, just to make clear that those things are just things behaving linearly.
The geometrical vectors that you talk about, as seen on physics and on basic analytic geometry are just one case of something that behave linear. They are just a part of a larger class of objects with the property that when we want to combine them we are allow to create "linear combinations". Your doubt is perfectly well, try to understand this thing this way, it'll help you a lot in making things disjoint - the geometrical appeal, and the algebraic generalization of a property.
By the way, good luck with your studies of linear algebra. As you go further and further into the concepts and results you'll be able to understand that just this single property ("linearity") grants lots and lots of things that once proved in the category of linear spaces will be available as results in each of it's specific cases (like geometrical vectors, or functions with pointwise operations and so on).
EDIT: also you ask how can vectors be used in geometry. Well, the basic definition is an equivalence class of oriented line segments, however to really work with it we prefer to understand a vector as a "machine that transports points", in other words, a vector is something that when we plug it on a point we get the point on the other side of the oriented line, where lies the arrow.
With this idea we can generalize things to $n$ dimensions. We simply say that $n$-space, which we call $\mathbb{R}^n$ is the set of ordered $n$-tuples (generalizing coordinates on the plane and space) and then we say that a vector is also an $n$-tuple of numbers, each one representing the amount that we should transport a point parallel to one of the $n$ totally independent directions, when we plug the vector at it.
In this way, representing a line in $n$-space is pretty easy. What is a line? A line is the result of transporting one point in the same direction forever. So we pick a point in $n$-space $p \in \mathbb{R}^n$ and we pick a vector $v \in \mathbb{R}^n$ and we transport $p$ along all possible multiples of that vector. In other words, the line is the set:
$$L(p, v)=\left\{q \in \mathbb{R}^n \mid q = p + tv, t \in \mathbb{R}\right\}$$
There is one assumption there: we sum points and vectors at will. Why? Well, for each point we can associate a vector (the position vector, from the origin to the point) and then we can sum this vector with the new one as we sum vectors normally. This will transport the point as I've said.
I think this will help you a little more. Good luck.