I can monitor I/O with tools such as iotop. It allows me to identify which processes do how much I/O.
Now I'd like to know which files those processes write all that data to. How would I do that?
E.g. something like "ok I know Tomcat is doing a lot of I/O. Which files is it mostly reading/writing from/to"?
I know two ways to get that kind of information.
Manually using lsof.
The good old lsof can show you what files are being accessed by a process or thread, along with a few other pieces of information. In iotop -o, eyeball and take note of the TID (Thread ID) value of the process or thread you need to inspect. Then close iotop and run lsof -p [pid/tid]. If you need to sort the output, pipe it to sort. For example, lsof -p [pid or tid] | sort -n -k 7,7 -r will sort the output from lsof by the seventh column (SIZE/OFF) in reverse order (biggest to smallest).
Using fatrace.
This new addition to Linux is similar to inotify, except it doesn't target specific files/directories. It shows you aggregate disk I/O based on files being accessed. Depending on your distribution, you might or might not have access to this nifty little program in precompiled executable form. The oldest distribution that provides fatrace in its official repositories that I know of is Ubuntu 12.04. Debian 7, which I'm using, doesn't have it.
csysdig is probably your best bet. It has a nice ncurses interface. https://github.com/draios/sysdig/wiki/Csysdig-Overview
