Short but boring answer: That depends
Longer, but still boring: ...on the ones making the drivers.
Simply put, your system will automatically install ANY driver that the provider considers "Yup, this is good enough to release". So the question then is, are you sure the vendors standards are at least as high as yours? Over the years I've heard a lot of grief about bugged drivers. This is not necessarily a sign of incompetence or malintent from the vendor, but bugs happen. Sometimes due to a poor QA process, sometimes due to the devs not being provided their daily caffeine fix, sometimes because of silly details that are easy to overlook. This combined with the fact that now and again you hear public outcry of some popular device that broke or became unstable after an auto-update (Didn't Apple have this recently?)
All in all, most of the time, for most people, it'll be fine. However, the difference between "most" and "all" never seems as big as when you're not in the Most-Club.
I'm a bit more cautious than most people, so I avoid auto-updated drivers - If it works, don't fix it. While I keep most of my software up to date with apt-get running now and then via cron, When it comes to drivers, I only update them if I am actually experiencing an issue with the ones I'm using.