core.whitespace instructs git to flag certain whitespace problems:
- trailing-spacewarns about whitespace at the end of a line or at the end of a file
- space-before-tabwarns when there is a space before a tab used for indentation
apply.whitespace is used when applying a patch. It checks for whitespace errors (the ones listed above, in core.whitespace) and applies the patch after attempting to fix them (i.e., remove them).
These options go in ~/.gitconfig -- that is, a .gitconfig file at the root of your user's home directory (typically /home/user/.gitconfig on Linux, /Users/user/.gitconfig on Mac OS X, and I don't know where on Windows but I suppose somewhere in C:\Documents and Settings\user).