Step 1: remove the DLL, preferably together with the install dir. In my case, PhotoViewer existed on both paths:
c:\Program Files\Windows Photo Viewer
c:\Program Files (x86)\Windows Photo Viewer\
These directories and their contents were owned by TrustedInstaller, so I could not delete them using administrator privileges, because... well, because that's how MS thinks ownership and priviliges should work.
So I had to set the ownership of these dirs to Administrators and then give full permissions to Administrators (recursively). (There are pretty good guides on how to do this on the Internet, so I won't bore you with that.)
After that, I only managed to remove the exe and dll files from here, de en-US etc. localization dirs remained, with all the *.mui files in them. Not elegant, but the general goal of removing PhotoViewer.dll was still achieved.
Step 2: clear associated keys from the registry.
regedit.exe > Edit > Find > PhotoViewer.dll
I had a single hit. Again I had to assume ownership and give full access to Administrators. After that, I managed to remove the key containing the path to the dll.
Now my original problem is fixed, the deprecated and buggy Windows Photo Viewer won't stand in the way of using Microsoft Photo Viewer by default.