0

Problem:

Due to precision errors I get a number, say, 0.004999999999. Mathematically speaking 0.00499(9) equals 0.005 precisely, but not in computers.

Such level of precision is still fine, but when I display the number, the user expects to see 0.01, i.e. rounded to two decimal points, but obviously any sane rounding algorithm would return 0.00 instead since 0.004999999999 is definitely closer to 0.00 than to 0.01.

But the user understanably expects to see 0.01

Solution(?):

It can be done with "multistage rounding" like 0.004999999999.round(10).round(2) given we internally calculate everything to a precision of 10 decimal points.

It seems to be a very common problem, but I surprisingly couldn't find any conventional solution to it.

diginoise
  • 7,352
  • 2
  • 31
  • 39
Daniel Vartanov
  • 3,243
  • 1
  • 19
  • 26
  • No, the double rounding won't work, because even with 10 decimal digits you can't get closer to 0.005. See [Is floating point math broken?](https://stackoverflow.com/q/588004/5987). Oops, bad example, that specific number rounds up instead of down. But there are certainly numbers that won't. – Mark Ransom Jun 28 '18 at 15:49
  • @MarkRansom, yes, I know about floating point math problems. This question is not related to floating point arithmetics as the same can happen even with decimal numbers. Also `0.004999999999.round(10).round(2)` returns `0.01` as expected, it works but just does not seem right – Daniel Vartanov Jun 28 '18 at 15:51
  • So you're saying you were given a *decimal* number of `0.004999999999` as input? Why then would you expect it to round to `0.01`? – Mark Ransom Jun 28 '18 at 15:52
  • Becasue it should have been `0.005` but even decimal number hit a limit at some point, `1.to_decimal / 3 * 3` => `0.999999999999999999` – Daniel Vartanov Jun 28 '18 at 15:55
  • I think the reason you don't see more relating to this problem is that you're the only one who assumes trailing decimal 9's must be infinite. – Mark Ransom Jun 28 '18 at 16:03

1 Answers1

0

There's nothing wrong with the double rounding approach, just be aware that binary numbers can't exactly represent decimals so sometimes the results aren't what you expect. Your example works in Python but another very similar one does not:

>>> round(round(0.004999999999, 10), 2)
0.01
>>> round(round(0.044999999999, 10), 2)
0.04

In the first case the first rounding produces a number just over 0.005 while the second produces a number just under 0.045; the exact numbers are 0.005000000000000000104083408558608425664715468883514404296875 and 0.04499999999999999833466546306226518936455249786376953125. Nothing closer is possible using binary numbers. If your language has a decimal number system, you could use that to get consistent results by avoiding the conversion to binary.

An alternate approach is to multiply the number by a very small amount so that something close to the boundary gets pushed over it.

>>> round(0.004999999999 * 1.000000001, 2)
0.01
>>> round(0.044999999999 * 1.000000001, 2)
0.05

Again, this approach won't work for every number, but it prioritizes the case you care about - you're more likely to get numbers rounded up when they should be rounded down.

Mark Ransom
  • 299,747
  • 42
  • 398
  • 622