Yes, there is an easy way to avoid this. The X11 packages are pulled by the recommended packages configuration, more exactly by the recommended doxygen-latex package — this is in Debian 7 (Wheezy), see note below for Debian 8. Debian automatically installs any recommended packages and, because of the latex and ghostscript automatic dependencies, a lot of X11 packages are then installed.
In your specific case, if you don't need Doxygen to generate latex documents, you can avoid installing the recommended packages by calling apt-get with the --no-install-recommends option. This results in the following installation:
# apt-get install --no-install-recommends doxygen
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
doxygen-doc doxygen-gui graphviz
Recommended packages:
doxygen-latex
The following NEW packages will be installed:
doxygen
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2578 kB of archives.
After this operation, 7348 kB of additional disk space will be used.
You can also configure Debian to avoid all recommended packages for any installation. Check this other SU question to see how. But, unless you want a specific purpose machine with the minimal set of packages, I wouldn't recommend it.
On Debian 8 (Jessie), the doxygen packagers have moved this one dependency (doxygen-latex) from the Recommends: field to the Suggests: field. As suggested packages are not installed by default, a regular install of doxygen on Debian 8 isn't so heavy.