Here is one way to do it. Generates a LOT of files, but I can't think of another way to do it.
Assuming you're not around to see when the server gets busy, you have to use sysstat to capture your historical resource data.
First way: use sysstat and collect stats every 5 minutes (will take up about 8mb a day). Your cron entry would look something like
5 * * * * /usr/lib/sa/sa1
or
5 * * * * /usr/lib64/sa/sa1
sysstat will keep up to a month of data. But you can always copy the files and archive them as well.
You would use the "sar" command to look back through historical data to see what time your server got busy.
Now that you have historical data on your server, you also need to capture process output so you can see what is running at the time. You can use "ps" or "top" for this
So every 5 minutes you can also capture the output of top and ps auxww so you can see all processes that are running. I would create two subdirs "toparchive" and "psarchive" and then put these as my cron jobs.
5 * * * * top -b -n 1 > /root/toparchive/top.`date +"%Y%m%d_%H%M%S"`
or
5 * * * * ps auxww > /root/psarchive/ps.`date +"%Y%m%d_%H%M%S"`
(that will be a lot of files, alternatively you can script it so you have additional subdirs with a datestamp, like /root/toparchive/20140701/)
But basically watch your sysstat and you can find out when your server gets busy, then check the appropriate top or ps output you've captured to see what took up your space
If the process is hogging memory, you can additionally use "pmap" and "pgrep" to look at current processes and actual memory usage as well.