0

I am doing some midpoint math using this formula

A1-MOD(A1+(Constants!$A$1/2),Constants!$A$1)

(Where values in column A are decimals and the value in Constants!$A$1 is an exact integer)

Then matching on an auxiliary worksheet with values entered exactly with precision to the tenth place.

VLOOKUP(B1,Data!$A:$B,2,FALSE)

But VLOOKUP is treating my values inconsistently. Its result (for the sake of this post the values are identical in both columns of the table_array parameter):

Sample lookup data

According to this question about Excel's numeric precision, it supports up to 15 digits, so I cranked up decimal precision just to see what it was computing:

Appears to be perfectly accurate

Wrapping my values in the "Band" column with ROUND(..., 2) made it work as expected, but why?

P.S. Note I am using Excel 2013.

2 Answers2

2

A point to realize and remember is that EXCEL is not hiding anything. If anything is hidden, it is the USER doing the hiding.

One can always format so that 15 decimal places are shown (so that any value is fully shown, even though, if there are places to the decimal's left, it means some (otherwise unnecessay) 0's will occur at the end). So instead of a nice "12.5" you could always display "12.500000000000000" and KNOW nothing is hidden.

However, that "begs the question" a little AND no one wants to do it and then "hand examine" each part of a data set.

One thing you can do is to use "Precision As Displayed" to force the values to be ONLY what you format the display to. So if Excel computed a value of "0.100000000000001" and you set the formatting for two decimal places, the end result in the cell will be "0.10" no matter what. Nothing can misuse it because it is precisely that.

However, further use of the value means further imprecision can occur that can affect formula output because, while the OUTPUT will be only, say, two decimals if formatted for such, the INTERNAL calculation results will NOT. So:

=2187.53 / 12.96 * 25.25

does not see the first operation result in 168.79, then get divided by 25.25, but rather, it results (internally to the formula) in 168.790895061728 which is then multiplied by the 25.25. The former results in 4261.97 while the latter results in 4261.95. So, same inputs being calculated, but different results. That means that your results can show a difference that depends upon how you set up the calculations... not a happy idea!

Further, that's something happening at two decimal places (six digits of precision in this example), not out at the very last of 15 places of precision!

Another approach then would be to do the calculation filling the column, then in the VLOOKUP() turn it to text formatted to the decimal places you want (two places, say) so you now have a string in the lookup. But wrap that with VALUE() to convert it back to a number for the lookup. (Lookups care hugely about number vs. string.)

Why does that work? It's the same thing? Nay. The column currently has values computed but some containing the imprecision. The lookup cannot match them as they actually ARE different. But the imprecision only occurs after one or more calculations. Making it text and cutting it off way, way before the imprecision occurs wipes out the imprecision. Then converting back to a number gives you a fully precise, fresh number with no idea of its actually imprecise history.

And if you use a Named Range to act so upon it, with relative references so it's useful here (OR, nowadays, use LET() to do the cleaning in-formula, so-to-speak, so it's completely easy), it can even look fine in the lookup, not cause it to be really hard for someone else to understand.

(And soon, you can make your very own LAMBDA() for the idea.)

Naturally, all the usual ways of using the roundings, the floors, the ceilings, the mods, truncations, etc., can all be used but you would seem to desire to avoid all of them, not just ROUND(). Can't say I blame you! (In a way, this is just a fancy truncation and I believe either the latter method above or TRUNCATE() would be completely the same, but a better way of being sure of that is to use this, not TRUNCATE()...!

For understanding how to address it, the essence is to remember that the imprecision NEVER EVER arises from a simple value itself. It ALWAYS arises only from arithmentical operations on that number. (And not even always then... divide a number by 3, then multiply by 5 and it might be completely right, but divide it by 5, then multiply by 3 and it might not. Or it might be good either way, or wrong either way. Lordy.) So get the value, convert to a string that is shortened, then back to a value and the imprecision is completely gone.

Jeorje
  • 21
1

This video has a very good explanation about binary storage and floating point numbers. https://www.youtube.com/watch?v=PZRI1IfStY0&t=1s

After that, please just accept that floating point inaccuracies will occur and round numbers if you need them to be precise.

teylyn
  • 23,615