The way I think about category theory is a general setting to formalize the term "Mathematical object" as meaning "some object in some category". (this to some extent includes "categories of categories"...) You are free to choose what your objects (if you prefer keeping them around) and morphisms are. As long as they satisfy the axioms, category theory provides you methods to study those objects. Whether a certain category is worth studying is a matter of personal preference, or perhaps applicability of results about that category to problems to want so solve.
You seem primarily concerned with concrete categories where objects are sets with structure and morphisms are functions preserving that structure. In topological spaces we study continuous maps. Continuous maps are very general: The majority of continuous maps $\mathbb{R}\to\mathbb{R}$ are completely crazy, wildly fluctuating nonsense that we will never find applications of. Nevertheless they have nice properties, which you surely know many of. When we find a function we deem interesting for some reason AND it it continuous, we can use what we know about continuous functions to study the function we want to study.
The problem with defining "nice well-behaved" is difficult. Just about any definition of nice morphisms includes a lot of pathological cases. That is why we have so many spaces of functions to fit the needs in each application: Just talking about functions $\mathbb {R}\to\mathbb{R}$ you can consider continuous, bounded, of bounded variation, Lipschitz-continuous, uniformly continuous, measurable, integrable in some way... You can consider equivalence classes of functions almost everywhere w.r.t. some measure and pretend they're functions, define Sobolev spaces and the like... On the other hand distributions and measures can be thought of as generalized concepts of function. There is no one way to define "a function that is worth studying".
A different question is certainly how to get from "structure" to "morphism preserving that structure". The most common types of structure on a set X are selected elements (e.g. identity element), unary and binary maps (inversion, addition, multiplication) and subsets of the power set (open sets, measurable sets). In the algebraic setting it is easier to justify that definition. In more general settings it requires some though and the answer might not be unique and different definitions be suited for solving different problems. It would be unreasonable to expect that only those definitions of arbitrary mathematical objects make sense, that are obvious enough to be derived from a simple rule.
For the discussion of the algebraic setting see answers in the related topics:
Especially
What does Structure-Preserving mean?
and maybe
What is a homomorphism and what does "structure preserving" mean?