I'm overworked, so don't have time to give you the answer; just some pointers - and will excuse it by claiming not to give you a fish, but to teach you to fish...
In XP, I would have used Tweak Xp, the PowerToy. Having said that, I wouldn't have deleted it, just redirected it to my D: drive (which is Data for me), or D:\music, D:\photos, etc. The one that annoys me most is automatically installing to C:\Program Files, since I have only Windows on C: and my programs are on P: - for ease of reinstalling Windows.
Anyway - short answer - Google.
Medium answer - regedit. Hit
(Windows key)+R and enter regedit (and Enter), to edit the registry. Search for keys whose value is C:\My Music, etc... At this point, I can't say what happens if you delete them. Maybe Windows will recreate them, but you probably have your own personal choice of directories for photos/music/etc., so why not just change the values?