I noticed that chrome caches the base64 string but the string is quite long and bloats my HTML (I have a lot of images on a page). 
If that is the case, consider placing a 'real' src attribute pointing to always the same placeholder. You do need an extra HTTP request, but:
- it will be almost certainly pipelined and take little time.
- it will trigger the image caching mechanism, which base64 does not do, so that the image will actually be only decoded once. Not a great issue given today's CPUs and GPUs, but anyway.
- it will also be cached as a resource, and with the correct headers, it will stay a long time, giving zero load time in all subsequent page hits from the same client.
If the number of images on a page is significant, you might easily be better off with a "real" image.
I'd go as far as to venture that it will be more compatible with browsers, spiders and what not -- base64 encoding is widely supported, but plain images are even more so.
Even compared with the smallest images you can get in base64, 26 bytes become this 
src="data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs="
while you can go from
src="/img/p.png"
all the way to
src="p.png"
which looks quite unbloaty - if such a word even exists.
Test
I have ran a very basic test
<html>
<body>
<?php
    switch($_GET['t']) {
        case 'base64':
            $src = 'data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs=';
            break;
        case 'gif':
            $src = 'p.gif';
            break;
    }
    print str_repeat("<img src=\"{$src}\"/>", $_GET['n']);
?>
</body>
</html>
and I got:
images   mode      DOMContentLoaded   Load      Result
200      base64    202ms              310ms     base64 is best
200      gif       348ms              437ms
1000     base64    559ms              622ms     base64 is best
1000     gif       513ms              632ms
2000     base64    986ms             1033ms     gif is best
2000     gif       811ms              947ms
So, at least on my machine, it would seem I'm giving you a bad advice, since you see no advantages in page load time until you have almost two thousand images.
However:
- this heavily depends on server and network setup, and even more on actual DOM layout.
- I only ran one test for each set, which is bad statistics, using Firebug, which is bad methodology - if you want to have solid data, run several dozen page loads in either mode using some Web performance monitoring tool and a clone of your real page.
- (what about using PNG instead of gif?)