It could be a plugin or the syntax highlighting; try a :syntax off when this happens and see whether Vim instantly gets faster.
With plugins, a "general slowness" usually comes from autocommands; a :autocmd lists them all. Investigate by killing some of them via :autocmd! [group] {event}. Proceed from more frequent events (i.e. CursorMoved[I]) to less frequent ones (e.g. BufWinEnter).
If you can somewhat reliably reproduce the slowness, a binary search might help: Move away half of the files in ~/.vim/plugin/, then the other, repeat in the set that was slow.
If you really need to look under the hood, get a Vim version that has the :profile command enabled. (Not the vanilla BIG Windows version, but the one that ships with Cygwin has it; also, self-compiling is quite easy under most distros.)