python numerical error: big number add small number

Question

This a exercise from Udacity's Deep learning course.

Can anyone explain why the final answer is not 1.0?

v1 = 1e9
v2 = 1e-6
for i in range(int(1e6)):
    v1 = v1 + v2
print 'answer is', v1 - 1e9
# answer is 0.953674316406

kostya · Answer 1 · 2016-03-18T01:22:29.087

0

Because 1e-6 cannot be represented exactly as floating point value:

print("{:.75f}".format(1e-6))
'0.000000999999999999999954748111825886258685613938723690807819366455078125000'

If you use a number that can be represented exactly such as v2 = 1.0/(2**20) and change iteration count to 2**20 you will get 0. However as @user2357112 pointed out even this property holds only if all of the intermediate results can be represented exactly using floating point value.

Check Python tutorial for more details: https://docs.python.org/3/tutorial/floatingpoint.html

edited Mar 18 '16 at 01:22

answered Mar 17 '16 at 02:20

kostya

9,221
1
29
36

1

"If you use a number that can be represented exactly for `v2` (e.g. `v2 = 1.0/(2**20)`) the result of running the program will be 0." - no, the result is still about 0.95, as it should be, since we don't subtract out the million multiples of v2. Also, exact representability of the value used for v2 isn't sufficient; we also need exactness of the intermediate sums to be guaranteed an exactly correct final result. For example, using `v2 = 1000.0 / 2**30` still produces rounding error in the intermediate values, despite the initial value being unrounded. – user2357112 Mar 17 '16 at 17:38
1

Indeed, you would need a number that is exactly expressible with 1+3 mantissa bits. 1000 = binary(1111101000) needs 1+6 mantissa bits. – Lutz Lehmann Mar 17 '16 at 20:17
@user2357112, thanks for the correction. Indeed I didn't account for the loss of precision and the fact that number of iterations must be changed. I will update the answer to be more precise. – kostya Mar 18 '16 at 01:12

score 0 · Answer 2 · answered Mar 17 '16 at 17:23

Let's check what v1 sees of v2 in the floating point addition:

>>> v3 = (v1+v2)-v1
>>> print "%.25f %.25f" % (v3,1e6*v3)
0.0000009536743164062500000 0.9536743164062500000000000

What happens is that all but the leading 1 get shifted out of the binary mantissa of 1e-6 while equalizing the exponent with 1e9. This means that the final value is 10**6 * 2**(-20) = (1.024)**(-2), which gives exactly the observed value

>>> print "%.17f" % (1.024)**(-2)
0.95367431640625000

python numerical error: big number add small number

2 Answers2