Unfortunately there doesn't currently seem to be a way to tell Chrome not to download resources until you explicitly request them (I even checked in their Extensions Web Store to see if someone wrote an extension that could do this).
The Opera Developer's guide for HTML5, Everything you need to know about HTML5 video and audio states:
What's up with all that downloading?
Opera, Chrome and Safari will
automatically download the whole video
file even if it hasn't started to
play yet. Firefox 3.6 only loads
enough to render a frame and determine
duration, unless the autobuffer
attribute is present. Note that the
spec changed from autobuffer to
preload, which hasn't been implemented
anywhere yet. Opera plans to change to
the Firefox behavior of only loading
enough to render a frame and determine
duration by default, unless the
preload attribute says otherwise.
It was a known issue that Chrome ignores the preload attribute, which means it's always preloading the video, even if it's not supposed to. The good news, though, is that this bug has been fixed in the latest developer version of Chrome:
Comment 72 by project member vrk@chromium.org, Apr 5 (6 days ago)
The patches to implement preload have now landed! The preload attribute will be finally recognized in Chrome 12. (Dev channel should get this update in a few weeks.)
Once this change takes place (depends on what version of Chrome you're using), you should see the audio files not preloading unless the RSS feed, webpage, etc, explicitly states that it should preload.
My understanding of HTML5 leads me to believe that if the web designer has it set to preload, there's nothing that you can do to tell it not to. My guess, though, is that we'll soon see an extension or option in browsers that will allow you to choose whether a source preloads or not (I would think kind of like how Chrome's Click to Play for plugins works).