Power of NVIDIA GPU out of limits (Nvidia A100)

Question

I am executing a CUDA kernel in my A100 GPU and I've realized that the power consumption at some points is higher than nvidia-smi given range:

The picture has been taken from nvtop.

Is it something that I should be worry about?

Answer 1

The only worrisome aspect is the temperature, which seems to be at an unimpressive 52°C. This doesn't make sense if the power draw is truly above the max.

So, take your pick. Either:

• The power draw figure is false
• The temperature reported is false
• nvtop is not working correctly with your GPU.

I would suggest verifying the temperature using other applications. If they also report the same readings, then you don't need to worry. Check both CPU and GPU and motherboard.

Useful references:

• How To Check CPU Temperature in Ubuntu Linux
• 4 Useful Tools to Monitor CPU and GPU Temperature in Ubuntu

Answer 2

The power draw of a GPU is uneven - it has spikes and lows. The specified power draw of a card is ment to be read as "rolling average over one second" during which time it can fluctuate over and belo that value - this is one of the reasons why PSU specs are recommended to be way over the sum of specified component power draw in a GPU-heavy rig.

nvidia-smi and friends report not the rolling average, but the momentary power draw, which can of course exceed the specified value. If you randomly sample your GPU power draw over a statistically relevant sample, you are very likely to see a value very close to specs.