I think my program is bottlenecked by file I/O, and I need a better solution

Question

My program interacts with an API, performs calculations, and builds a Gui using information, some of which is stored in local files (to preserve information between logins). When using cProfile to profile my code, I get the following output:

C:\Users\cheek\Documents\Code\LoL-Performance-Tracker>python -m cProfile -s tottime LoL-Performance-Tracker.py
LoL-Performance-Tracker.py:271: SyntaxWarning: name 'apiKey' is used prior to global declaration
  global apiKey
Entered buildMatchHistory
         262880 function calls (261634 primitive calls) in 11.867 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      161    8.684    0.054    8.684    0.054 decoder.py:370(raw_decode)
        1    1.100    1.100    1.100    1.100 {question}
        1    0.694    0.694    1.794    1.794 {built-in method exec_}
        1    0.506    0.506    9.848    9.848 LoL-Performance-Tracker.py:88(buildMatchHistory)
      168    0.361    0.002    0.361    0.002 {method 'read' of 'file' objects}
       84    0.073    0.001    0.149    0.002 ConfigParser.py:464(_read)
        1    0.040    0.040   11.867   11.867 LoL-Performance-Tracker.py:7(<module>)
       80    0.035    0.000    9.323    0.117 MatchHistoryBuilder.py:37(buildMatch)
      161    0.026    0.000    9.074    0.056 __init__.py:258(load)
      251    0.025    0.000    0.025    0.000 {open}
        1    0.023    0.023    0.023    0.023 {built-in method show}
    23338    0.019    0.000    0.019    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
    23176    0.017    0.000    0.017    0.000 collections.py:59(__setitem__)
        2    0.016    0.008    0.016    0.008 {built-in method setWidget}
       84    0.010    0.000    0.010    0.000 {built-in method setStyleSheet}
    11844    0.008    0.000    0.008    0.000 {method 'readline' of 'file' objects}
        1    0.006    0.006   11.704   11.704 LoL-Performance-Tracker.py:326(main)

...

The buildMatchHistory method is what I thought would be the problem, because it builds gui objects and is generally pretty cumbersome, but it doesn't seem to be.

I'm using a json encoder/decoder to perform file I/O with a good amount of information. I wouldn't think that it would take multiple seconds to perform these operations, but it looks like it is. Am I correct in my understanding of this output? If not, where should I look instead?

If I'm right, what's a better solution for pulling and storing information between logins?

You are interpreting the results correctly.How many files are you storing? if the target information is spread across many files, you may see some speed increase by parallelizing the reads using multiprocessing. — skrrgwasme, Apr 06 '16 at 00:15
The profiler output would be far more helpful together with the code. — Klaus D., Apr 06 '16 at 00:47
Some people do [*this*](http://stackoverflow.com/a/4299378/23771) to see what is taking time. — Mike Dunlavey, Apr 07 '16 at 17:54

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

Try Using PerfMon

cProfile Looks really cool, but have you ever used Performance Monitor? If you're running windows it will come with an application called "Perfmon". If you pull up this application you can monitor almost any performance issue through the use of performance counter (PF's are not important). This could really help you find your bottleneck. I'll walk you through how to do it for disk usage. Here's how it works...

Open up the application and left click on "Performance Monitor" under the monitoring tools folder. Right click on the graph and click "Add Counters...". You will be presented with a window.

If you navigate down to PhysicalDisk you can add a counter for Disk Read and Disk Write percentage. Once you add these counters you can monitor your disk usage on on the graph.

Note: You can apply this tactic to any performance issue you have!

Addressing Your Problem

If your disk usage is low, then I/O probably isn't a problem.

Your cProfile seems to think that the "decoder" contributes a lot to "tottime". Try checking your processor to see if it is running at 100%. If it is, maybe you could simplify the decoder, or offload these calculations to a GPU.

If your disk is connected via USB, you can monitor USB performance as well.

I see "login" in your post. Maybe there is something making a call to an online resource. Try monitoring the network.

Maybe you don't have enough RAM, check it out.

You can use this method to procedurally eliminate possible bottle-necks one by one if all else fails. I wish you luck in finding your bottle-neck and look forward to see other's ideas.

Thanks, this is helpful. I checked my Read/Write %s, but both don't get over 10%, and are mostly very small. I don't think this indicates a hangup from reading and writing. Given this, what would be a better way to store information so that I don't have to do this 161 times in my code? — Tyler Cheek, Apr 06 '16 at 23:28

I think my program is bottlenecked by file I/O, and I need a better solution

1 Answers1

Try Using PerfMon

Addressing Your Problem