Your question is very wide and philosophical! I give it a try.
There are several equivalent définition of the Tangent space at a point $x\in
M$ where $M$ is a differential manifold. Recall that the notion of differentiable function is given chart-wise and do not use the tangent space.
First definition see for instance "semi-riemannian geometry" of Barett O'neil :
Let $M$ be a differential manifold and $x\in M$.
Let $\mathcal C^1(M,\mathbb R)$ the set of continuously differntiable function from $M$ to $\mathbb R$.
The Tangent space $T_xM$ of $M$ at $x$ is
the set of $\mathbb R$-linear function $u : \mathcal C^1(M,\mathbb R)\rightarrow \mathbb R$ such that
$$ \forall f,g\in \mathbb R, \quad u(fg)=f(x)u(g)+u(f)g(x) $$
Second definition :
Let $M$ be a differential manifold and $x\in M$. Let
$$E_x:=\{\gamma\in\mathcal C^1(]-1,1[,M) ~|~ \gamma(0)=x \}$$ the set of continuousy differentiable curve in $M$. Define the equivalence relation $ \gamma_1\sim \gamma_2 $
if their exists a chart $\phi : \mathcal U \rightarrow \mathcal V\subset \mathbb R^n$ around $x$ such that $(\phi \circ\gamma_1-\phi\circ \gamma_2) : ]-1,1[\rightarrow \mathbb R^n$ has derivative zero at $0$.
To come back to your question, the straight line between $x$ and $y$ is not well defined in your situation. You would need a riemannian metric or some additionnal geometric structure on you manifold. Another way to put it : $x$ does not see $y$ so their is no most direct way of going from the one to the other. However the second definition allows you to obtain the tangent space in a "path-wise way".
Now, from your argumentation (and the example you chose), I guess you are particularly acquainted to sub-manifold of $\mathbb R^n$. You can prove that every compact differentiable manifold is a submanifold of $\mathbb R^N$ for $N$ big enough, so you can be tempted to consider sub-manifold instead of the abstract notion of manifold. However, the fact that a manifold can be embedded into $\mathbb R^n$ is almost never useful to answer interesting questions. You might as well ask wether we consider groups instead of subgroup of permutation groups since the former can allways be realized as the latter. There are many such example in mathematics (vector spaces versus $\mathbb R^X$, rings versus quotient rings of polynomial rings with coefficients in $\mathbb Z$,...).
Mathematicians are linguist that construct new languages to encapsulate their intuition of the problems they solve.
There are plenty of operation that are not well suited for submanifolds :
- Quotient manifolds (they may be embedded in the same $\mathbb R^n$ as the manifold you first considered. You can embed $\mathbb R$ into $\mathbb R$ but the circle $\mathbb S^1 = \mathbb R/\mathbb Z$ can only be embedded into $\mathbb R^2$).
- You might want to consider several riemannian metric on a given manifold which make uneasy a common representation (think about Ricci flow and Poincaré conjecture). For instance, the circle you represented is the "round circle" but ellipse are also "differential circles" and from the manifold view point, they cannot be distinguished. The picture has lot of additionnal properties you may not want and might not be relevant to your problem
- Fiber bundles. You may not know that (yet) but these are very very important objects. Many geometrical structures are section of fiber bundles, they are omnipresent in High Energy physics or General Relativity. Classification of manifolds make systematic use of fiber bundles (Chern's proof of Gauss-Bonnet-Chern is a basic example). Many operation you can do on Vector spaces can be done on Vector fiber bundle over a given manifold (direct sum, product, tensor product, quotients,...).
- One broad class of example of manifolds is given by Lie groups and their quotients. The theory makes uses of quotients, covering, group sum,...