I use nvidia-smi to see the status of each GPU on a computing node but find one of them is E. Thread. Is there any easy way to switch it back to default mode?
------------------------------------------------------+
| NVIDIA-SMI 346.46 Driver Version: 346.46 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 680 Off | 0000:03:00.0 N/A | N/A |
| 30% 30C P8 N/A / N/A | 10MiB / 4095MiB | N/A E. Thread |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 680 Off | 0000:04:00.0 N/A | N/A |
| 30% 29C P8 N/A / N/A | 10MiB / 4095MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
Asked
Active
Viewed 2.3k times
3
James LT
- 733
- 2
- 12
- 23
2 Answers
10
nvidia-smi -h
will give you command line help, and there is also a man page:
man nvidia-smi
the following command should reset that device to compute mode of 0 (default)
nvidia-smi -i 0 -c 0
should reset that device (0) to compute mode of 0 (default)
You need to have root privilege to modify the device this way, so either be a root user or run the command with sudo
Robert Crovella
- 143,785
- 11
- 213
- 257
-
I am looking at the nvidia-smi manual. Nowhere does it mention which numbers in compute mode (0, 1, 2, 3) refer to which mode! – user13107 Apr 27 '18 at 06:25
-
@user13107 `nvidia-smi -h` (the very first thing I said in my answer). I did that for the 387.26 driver and it says: `-c, --compute-mode= Set MODE for compute applications: 0/DEFAULT, 1/EXCLUSIVE_PROCESS, 2/PROHIBITED` 3 is not not listed and 1 has been changed to EXCLUSIVE_PROCESS, because EXCLUSIVE_THREAD was deprecated and then removed some time ago. – Robert Crovella Apr 27 '18 at 13:46
-
and, for example, instead of specifying `-c 0` for DEFAULT, you can also specify `-c DEFAULT` – Robert Crovella Apr 27 '18 at 13:57
8
In order to know the current Compute Mode, use the following command
nvidia-smi --query | grep 'Compute Mode'
In order to set the Compute Mode, use the following command
sudo nvidia-smi -c $i
Where $i could be 0, 1, 2 or 3. Meaning of these numbers are given below -
0 Default
1 Exclusive_Thread
2 Prohibited
3 Exclusive_Process
Answer to your question
sudo nvidia-smi -c 0
This will set all GPUs in default mode. Use -i 0 flag if you want to apply it only to GPU 0.
user13107
- 3,239
- 4
- 34
- 54
-
Brief comment that Exclusive_Thread is deprecated with newer drivers ``` sudo nvidia-smi -c 1 Warning: Exclusive_Thread was deprecated! Setting Exclusive_Process instead. Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:3B:00.0. Warning: Exclusive_Thread was deprecated! Setting Exclusive_Process instead. Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:AF:00.0. All done. ``` – Kevin Lee Oct 14 '20 at 23:16