PC with 32 screen matrix freaks

Question

I've just finished building a new control room at work. It has 32 monitors and the plan was to have a single computer powering it. The old room had a few computers with odd screens keyboards/mice everywhere and decided it was time to simplify things and have a single PC - with it being a single operator most of the time.

There's not an awful lot of demanding stuff running on the machine, some scada packages, IP camera viewing software, office etc.

The issue that I'm having isn't down to performance. At least I don't think, the computer is of a fairly high spec. It's a HP Z840 with 2 Intel Xeon E5-2670's 4 nvidia nvs810s, 256GB of ram and a 500GB SSD. The operating system is Windows 10 Enterprise 64bit. The screens are all HP Z24n.

My slots are used as follows.

1. PCIe3x4 - None
2. PCIe3x16 - NVS 810 1
3. PCIe3x8 - None
4. PCIe3x16 - NVS 810 2
5. PCIe3x8 - NVS 810 3
6. PCIe3x16 - NVS 810 4
7. PCIe2x4 - None

I've realised after looking at the manual 1 that I should have GPU 3 in slot 3. However the behavior of the machine is strange, I connected all 32 at first and most came on with the windows background and task bar. About 10 had no background but had the taskbar. The mouse moved at a snails pace and I was unable to position the screens in nvidia control panel as it would crash/freeze. I unplugged the cables from GPUs 1 and 2 and managed to get 16 screens on from cards 3 and 4. When I got to screen 21, 5th screen on GPU 2 the machine when crazy again. The mouse started to lag again, and some screens were showing as duplicates of each other.

I've had a look in task manager, I've not seen the CPU or RAM go any higher than 4% when it locks up its just nvidia control panel that is not responding.

I'm thinking it must be some sort of bandwidth problem but not sure how to prove this or fix it.

Should I be able to get 32 1920x1200 screens of this hardware?

Is this behavior normal? I will try moving NVS 810 3 to slot 3 and see what difference that makes, any other ideas would be appreciated.

The screens are arranged in a 8 by 4 matrix.

update from 30/07/16

There had been questions about whether I had reached the max horizontal limit for windows so I wanted to give this a test and prove it.

So I uninstalled the video card driver and removed 1 card so I only had cards in slots 2, 4 and 6. I connected up 16 screens in a 8 by 2 matrix to the cards in slots 2 and 6 and it worked OK. The PC was still struggling when using Windows display settings and nvidia control panel. After applying the video settings it took at least a minute to settle and allow me to accept the config. I stretched a window across the whole screen matrix.

I then tried to put a 17th screen on and all hell broke loose again. So as you can see below I added the 17th screen in the middle of the two rows. And applied the settings. The PC took ages to settle and allow me to accept again.

So at this point it the newly added screen is duplicated off the bottom left and windows display settings is showing some freaky 6|17 instead of what the nvidia control panel is showing.

I had a go at building the matrix up 4 x 4 and adding more in. Again I made it to 16 screens with no great shakes still a little struggle waiting for it to settle and apply the config but nothing major.

I connected them to the card as follows

NVS 810 1 - top 2 rows of 4 NVS 810 2 - bottom 2 rows of 4 (dont worry about the white screen,it was just an explore window)

I moved the right side top four and connect 2 of them.

They worked 'OK' however they had black wall papers not like the others. Also when you did like a left click drag to select things it wouldn't clear off. So I could draw blue boxes all over so I knew at this point something was up. For the heck of it I connected the next 2 and it threw all the toys out the pram again. It merged/duplicated the top 2 middle screens.

8/1/16

Ordering 6 x AMD Firepro W600 hopefully will have them for the end of the week and will feed back!

8/4/16

Installed 3 x AMD Firepro W600 and hit the same wall at 16 screens, however it was less flaky to setup compared to the nvidia settings, the AMD display settings never crashed and allowed Windows display settings to control the screen layout.

Answer 1

I've been there and done that.
Even with the same hardware :-)

Never got it to work beyond 2 cards or 16 screens (4 screens per card and 4 cards also doesn't work properly) in Windows.
Worked fine with the free Nvidia drivers in Linux, but not with Nvidias own proprietary driver. But that wasn't a solution as we needed to run Windows only software on these.

We concluded that the Nvidia drivers are really crappy and badly tested (if at all) for these configurations even though, in theory, it should be possible.

We ended up using 2 computers. On for the top 2 rows, one for the bottom rows.

Another thing to consider: The cards can handle camera streams from that many monitors, but Windows really doesn't like streaming more than 20 or so simultaneously. Get's really choppy even though the hardware didn't get stressed. Seems a limitation of the Windows video codecs or the Windows desktop manager.
Splitting over 2 computers also allowed us to prevent that happening.

Answer 2

I bet you're running into limitations on the memory bridge and other bottlenecks not typically monitored under windows or UNIX since it's things like the CPU and GPU that normally cap out... but since you're pushing the PCIe bus to its maximum you're seeing it.

This is along the lines of what Tony and others who have tried this say: "things get choppy even though the hardware isn't stressed". But it is stressed, just not in a way that is monitored ie; taskmanager and GPU tools.

North/south bridges and the inter-CPU communication pathways all have limitations and you're hitting them with that much bit-slinging.

For this reason I think switching between AMD/nVidia or Windows/Linux is going to make no difference.

My suggestion: Break this up into 3 or 4 machines and then run something like Mulitplicity so it's all seamlessly controllable from one keyboard/mouse.