I don't know what more I need to configure for the .htaccess to work.
ErrorDocument 404 /404.html
I have it in my document root /var/www/html where the 404.html is, but it doesn't work. 
I tried editing apache2.conf,
<Directory /var/www/>
  Options Indexes FollowSymLinks
  AllowOverride All
  Require all granted
</Directory>
a suggestion I got from here, but when I entered a non-existent page I got an Internal Server Error 500, so I had to change AllowOverride All back to None.
Everything I look at tells me the same things, but nothing's working for me. This is on a new dedicated server, so it might be something I need to configure in apache2. Any help would be appreciated.