Oddity calculating runtime with timeit in Python?

Question

I want to calculate the runtime of two different algorithms in the same program. When I wrote a program calculating the runtime of each individually, I obtained very different results, so to test this new program, I had python calculate the runtime for the same algorithm twice. When I did this (in the program found below), I found that the runtimes of the same algorithm were in fact different! What am I missing and how do I fix this so I can compare algorithms?

import timeit

def calc1(x):
    return x*x+x+1

def calc2(x):
    return x*x+x+1

def main():
    x = int(input("Input a number to be tested: "))
    start1 = timeit.default_timer()
    result1 = calc1(x)
    end1 = timeit.default_timer()

    start2 = timeit.default_timer()
    result2 = calc2(x)
    end2 = timeit.default_timer()

    print("Result of calculation 1 was {0}; time to compute was {1} seconds.".format(result1,end1-start1))
    print("Result of calculation 2 was {0}; time to compute was {1} seconds.".format(result2,end2-start2))
main()

How different were they? There's a few ways of running `timeit` — roganjosh, Oct 17 '17 at 17:56
I've run it locally (your indentation was off) and I see it. What OS are you on? — roganjosh, Oct 17 '17 at 18:01
Maybe this is not so concerning in and of itself, but when testing non-identical algorithms, some ran faster than others where they seemingly should not have. — int-elligænce, Oct 17 '17 at 18:02
Windows 10. And apologies for the indentation, I am new to this. — int-elligænce, Oct 17 '17 at 18:02
Actually, no, this is quite a difference so I would like to look into it :) Also, I have got rid of `eval` and cast the input to `int` instead. Please don't use `eval`: https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html — roganjosh, Oct 17 '17 at 18:03
The things you're trying to time are way too short to get reliable measurements. — user2357112, Oct 17 '17 at 18:06
@user2357112 however, I can also pin some of it down on Windows power management. `something = 5**100000` before the timed loops makes sense since it pulls the runtime closer together. It's an annoying feature of Windows that it throttles CPU through the power management system. Then I will try clarify testing method with multiple iterations. — roganjosh, Oct 17 '17 at 18:08

roganjosh · Accepted Answer · 2017-10-17T19:13:37.663

I think you're being bitten by Windows power management in one part, and the second part, your testing method is flawed.

In the first case, I also got bitten by the fact that Windows, by default, will throttle CPU throughput in order to save power. Currently, there is a dramatic difference in your calculated runtimes; this can be dramatically reduced just by having a nonsense calculation like something = 5**1000000 immediately after x = int(input("Input a number to be tested: ")) to ramp up the CPU resources. I hate Windows 10 so I don't know how to change this off the top of my head; I need to check how you shift Power Options" to "High Performance" and remove CPU throttling (Edit soon), which will reduce this gap considerably.

The second issue is that you only run one test cycle. You cannot get stable results from this. Instead, you need multiple iterations. For example, with a million iterations you would see some similarity between the numbers:

import timeit

exec1 = timeit.timeit(stmt='def calc1():\n return x*x+x+1', number=1000000)
exec2 = timeit.timeit(stmt='def calc2():\n return x*x+x+1', number=1000000)

print("Execution of first loop: {}".format(exec1))
print("Execution of second loop: {}".format(exec2))

Depending on your IDE (if it's Canopy/Spyder) then there could be cleaner ways of running timeit such as using your existing definitions of calc1 and calc2.

Oddity calculating runtime with timeit in Python?

1 Answers1