If $A=\bar S$ for an open set $S$ and any two points in $A$ can be joined by a unique minimizing geodesic with interior completely lying in $S$ (i.e. $S$ is strongly convex), then $S$ is homeomorphic to the open unit ball by this argument. This does not directly imply that $A$ is homeomorphic to a closed unit ball (think of the sphere with a point removed - this is topologically an open disk, but its closure in the sphere is not a closed disk). However, one could think of finding $S'\supset A $ also strongly convex and using a homeomorphism of $S'$ to a larger ball, but apparently this does not always work, see here.
If you do not assume that $A$ is the closure of an open, strongly convex set, then there are many counterexamples, as has been pointed out in the comments already. The most apparent being $A=M$ itself or $A$ being a small closed segment of a geodesic.