the whole desktop environment seems like way overkill for what I need.
It's not. It's probably more efficient even for SSH than any alternatives (of which there aren't many), but it's pretty much the only option for browsers – practically all of them are X11 programs, so you need to run them on an X11 display server, which means running Xorg.
Keep in mind this is Linux – the actual desktop environment does not need to be full 3D-accelerated KDE or GNOME; you can manually assemble a basic environment that consists of a window manager such as Openbox (or twm or Fvwm if you want it to look more retro) and some Xterm windows.
Although these days, Linux uses kernel mode-setting for most (even old) graphics controllers, which means initializing Xorg takes less than a second as the GPU is already configured, unlike in the old days where Xorg itself had to do it from scratch. That is, it is quite possible to stay in the console most of the time and startx whenever needed, exiting Xorg when no longer needed.
However, I would not recommend that; the Linux kernel's built-in console is slow and inefficient (among other problems) and probably the worst choice there is for daily use. You'll have a much better experience SSHing from xfce4-terminal or Xterm or something such.
But in any case, the browser – not the rest of the GUI! – will be the most inefficient part. Modern browsers and webapps are large; you could try Ladybird as it's getting close to usable, but trying to load a massive JavaScript-driven GUI in modern Firefox or SeaMonkey will be probably 80% of your battery usage. Trying to do it in something else than Xorg won't really change much.
(I mean, you could probably go all the way with the "terminal" setup and use your laptop as a VNC terminal to connect to a browser running on another machine...)