How can I monitor the usage of NVLink connections?

Question

If I'm running a Keras model on some Nvidia GPUs which are connected via NVLink, how can I monitor the usage of the NVLink connections?

I want to be sure the NVLink connections are being used and see at what rate.

Answer 1

nvidia-smi nvlink

This command shows various information about nvlink including usage. If nvlink connections are utilized, usage should go up during training.

nvidia-smi nvlink -h

Shows available performance counters on present cards.

"NVLink Usage Counters" section in this tutorial shows how to see if data is being transferred across nvlink.

https://www.exxactcorp.com/blog/HPC/exploring-nvidia-nvlink-nvidia-smi-commands