You will see here my config for DiffMerge or KDiff3.
Based on that, I would recommend for p4merge:
git config --global merge.tool merge
git config --global mergetool.merge.cmd "merge.sh \"$PWD/$LOCAL\" \"$PWD/$BASE\" \"$PWD/$REMOTE\" \"$PWD/$MERGED\""
and merge.sh being a wrapper (copied in a directory referenced by your PATH environment variable), able to take into account the case where no BASE exists.
(when a file is created in two different branches being then merged, there would be no common ancestor for that file)
#!/bin/sh
# Passing the following parameters to mergetool:
#  local base remote merge_result
alocal=$1
base=$2
remote=$3
result=$4
if [ -f $base ]
then
    p4merge.exe -dl "$base" "$alocal" "$remote" "$result" 
else
    p4merge.exe -dl "$result" "$alocal" "$remote" "$result" 
fi
You may note:
- the use of PWDin the config of the merge
- the use of "merge" as name of the merge.tool name (since the actual tool is called in themerge.shscript, where you can switch between any number of merge tool you want)
- the use of double quotes around $base,$alocal,$remote,$resultwithin the script
- the conditional path for calling the tool, based on the existence of a "base" file.
- the need to always have 3 files to merge as parameters (even when 'base' does not exist...)
Just tested it (it turns out, you can download and install only p4merge -- section Client/Visual Merge Tool --, even if you do not have any other P4 product installed).
With the settings describe above, MSysGit1.6.3, DOS session or Git bash session:
It just worksTM.
Update msysgit 1.7.x
Benjol mentions in the comments:
p4merge is now supported natively by msysgit.
This means you can just do:
git config --global merge.tool p4merge
# and I recommend 
git config --global mergetool.keepBackup false