The arithmetic functions are only defined on $\mathbb N^+,$ so you can’t use the calculus limit definition to get a standard derivative.
So in your case, you just apply the definition:
$$f’(n)=(n^2-9\bmod 16)\log n$$
The relation to calculus definitions is if $f$ is arithmetical, then there is a related function, $F,$ (called a Dirichlet series):
$$F(t)=\sum_{n=1}^\infty f(n)n^t$$
Usually defined for $t$ with a negative upper bound on the real part, but you can also think of $F$ as an abstract series, the way you can think of power series abstractly. (Dirichlet series are usually defined in terms of $s,$ where $t=-s,$ but I chose this version because it otherwise changes the sign of $F’.$)
Then:
$$F’(t)=\sum_{n=1}^\infty \log(n)\cdot f(n)n^t=\sum_{n=1}^\infty f’(n)n^t$$
where $F’$ is the usual calculus/complex analysis derivative, and $f’$ is Apostol’s definition for arithmetical functions.
The relation between $F$ and $f$ is crucial in studying arithmetical functions. Given $f,g$ arithmetical and corresponding Dirichlet series $F,G,$ the Dirichlet series for $f+g$ is $F+G,$ and the Dirichlet series of $h=f*g,$ where:
$$(f*g)(n)=\sum_{d\mid n}f(d)g(n/d)$$ has corresponding Dirichlet series, $H(t)=F(t)G(t).$
On thing you get from this correspondence is the usual identities:$$(\alpha f)’=\alpha\cdot (f’)\\(f+g)’=f’+g’\\(f*g)’=f’*g +f*g’.$$
where $\alpha$ is a constant.