This is nothing to worry about. Homebrew is designed to play well with other things that install in to /usr/local/, which is the standard place for user-built software to go. Plenty of other things install themselves there. The brew list --unbrewed comamnd just lists all the files under /usr/local which weren't put there by Homebrew. It is not diagnostic of an actual problem, it's more for identifying stuff you might want to put under a Homebrew formula, or to help formula maintainers check for stray files while doing installation work. Any normal system will probably have a nonempty brew list --unbrewed.
Do a brew doctor instead. That checks for files in certain places under /usr/local which could actually be problematic. If brew doctor complains about something, you probably need to address that. Otherwise, don't worry about them, and don't try to clean them up - you probably put those files there for a reason.
Here's an example of what my test 10.8.3 system reports. Plenty of --unbrewed files, but doctor is happy. Homebrew is functioning just fine on it.
$ brew list --unbrewed | wc
1721 1727 82663
$ brew doctor
Your system is ready to brew.
So:
1) No, don't worry about this or fix it unless brew doctor tells you to. Removing these files will probably break something you installed.
2) Those paths are relative to brew --prefix, which is probably /usr/local.
3) Hypothetically, to "clean it up", for each file, you'd just determine what installed it and uninstall it, reinstalling it using a Homebrew formula if it's something you actually need, or come to a conclusion that it was a mistake or no longer needed, and delete it. There's no standard or easy way to do this. And that is exactly the problem that Homebrew and other package managers exist to solve.
Some of that might be worthwhile if you want to use Homebrew more extensively to control your installations. Each time you're about to install some non-Homebrew software that sticks itself in /usr/local/, instead of following the normal installation instructions, use brew create to make a formula for it. Then homebrew will track the files it installs.