This might not seem like an answer to your question at first sight, but I'd say, read about Hopf algebras and then your pain might be mended. We already have an excellent answer about books on Hopf algebras on this site here. For a quick intro, you could also use these notes by Kytölä (in the introductory section on group representations, he defines duals etc. "non-functorially", but when he gets to Hopf algebras the constructions are clarified). Let me add the very recent book "From Rings and Modules to Hopf Algebras - One Flew Over the Algebraist's Nest" which provides an intro to characteristic zero representations of finite groups from a categorical/Hopf point of view.
Of course, I should argue for why this answers the question, i.e. why Hopf algebras might cure your pain as a category lover.
The reason is a general point that is important to keep in mind. For any random $k$-algebra $A$, you will have a lot less structure on the category $A\textrm{-}\mathsf{Mod}$ than you have on $k[G]\textrm{-}\mathsf{Mod}$: you won't have duals, internal Homs, tensors. If you want to have all these structures in a nice way that is compatible with the forgetful functor $A\textrm{-}\mathsf{Mod} \to k\textrm{-}\mathsf{Mod}$, then you need a Hopf algebra structure on $A$.
The Hopf algebra operations on $k[G]$ may derived from the functoriality of the free $k$ vector space functor applied to inversion $G \to G, g \mapsto g^{-1}$ and the diagonal $G \to G\times G, g \mapsto (g,g)$ (the latter can be derived from the universal property of the product). Then all operations on the category of modules (duals, tensors, internal Homs) may be derived from the Hopf algebra structure. Or arguing differently, the free vector space functor is strictly monoidal with respect to the cartesian product on $\mathsf{Set}$ and the tensor product on $k\textrm{-}\mathsf{Mod}$, thus it sends Hopf monoids to Hopf monoids.
As evidence for why it is "the correct view" to consider $k[G]$ as a Hopf algebra or (equivalently) to consider $k[G]\textrm{-}\mathsf{Mod}$ as a rigid monoidal $k\textrm{-}\mathsf{Mod}$-enriched category, one could point out that a group $G$ is not determined by $k[G]\textrm{-}\mathsf{Mod}$ as an abstract category (even with the $k\textrm{-}\mathsf{Mod}$-enrichment), but if you're willing to put in more structure, you get -as an easy instance of a Tannaka duality result- that $G$ is determined by $k[G]\textrm{-}\mathsf{Mod}$, see here. See also this answer for some categorical "fluff" about the proof of Maschke's theorem.
