Nakahara's book covers a lot of ground and is an overall very nicely written book. In terms of a mathematician's treatment, I recommend looking over Frankel or Nakahara, picking out topics that you're interested in, and look up resource recommendation for each topic. To get you started, some things (in my opinion) any self-respecting physicist should have a grasp of are:
For general mathematical maturity, as well as topological aspects of physical systems (e.g. field theories), you need to get acquainted with topology:
- Point-set topology - There's a number of Math SE threads on this, for example this one
- Algebraic Topology - For these, Hatcher's book is a standard and well-motivated treatment. Also see this this thread
- homology
- cohomology - usually De Rham cohomology is what a physicist should be comfortable with
- homotopy
The setting of most physical theories (e.g. GR & QFTs in general backgrounds) is some smooth manifold, you need to get acquinted with basic differential geometry.
- Differentiable manifolds - definition of real and complex manifolds, vectors & forms & tensors, integration of forms
- (pseudo) Riemannian geometry - metrics, connections, curvature.
For these see this Math SE thread. I personally enjoy the two books by Lee.
Classical and quantum gauge theories are described by bundles associated to principal bundles of the symmetry group, we need
- Fibre bundles
- Connections over fibre bundles
For these, see this thread.
Again I think this is the basics of what every theoretical physicist should familiarize themselves with, and to have a decent grasp of basic theories in modern physics. Again Nakahara is the place to start, and a good reference to come back to for summaries.