The Riemann-Stieltjes integral in general is not as easy to work with as the Riemann integral, which is just a special case for a continuous and increasing integrator.
If $g$ is nonincreasing or nondecreasing, then it is easy to show using Riemann-Stieltjes sums that
$$|F(x) - F(c)| = \left|\int_c^x f \, dg \right| \leqslant \sup_{z \in [a,b]}|f(z)||g(x) - g(c)|,$$
and by continuity of $g$, we have $F(x) \to F(c)$ as $x \to c$.
If $g$ is of bounded variation, then
$$|F(x) - F(c)| = \left|\int_c^x f \, dg \right| \leqslant \sup_{z \in [a,b]}|f(z)|V_c^x(g)$$
This result is proved here using another result given here. In this answer it is shown that the total variation function $V_c^x(g)$ is continuous if $g$ is continuous. Again it follows that $F(x) \to F(c)$ as $x \to c$ since $V_c^c(g) = 0.$
The most general case, where it is not given that $g$ is of bounded variation, is probably beyond the scope of what is expected. In fact, if $g$ is of unbounded variation, then it is not guaranteed that the Riemann-Stieljes integral exists for every continuous function $f$.