3

My question is not about the fine points of floating-point calculations, but about the differences in how it's evaluated. Given this:

float a = 1f / 12;
int b = (int)(1f / a);

At runtime, b will be 12. But if I evaluate its expression in something like QuickWatch or LINQPad, it's 11. In fact, I'm unable to recreate the result of 11 at runtime using float, double, or decimal; truncation or Math.Floor.

What's behind the difference between dynamic evaluation of the expression and the evaluation at runtime?

EDIT:

Since I'm seeing comments about people getting a result different from mine, here is what I'm seeing.

The result at runtime:

Runtime result

The result when evaluated by VS QuickWatch:

Eval result

The result in LINQPad:

LINQPad result

My VS and framework versions:

VS and framework versions

redman
  • 1,510
  • 17
  • 33
  • QuickWatch being the QuickWatch of Visual Studio? Because my VS2019 QuickWatch calcs correctly 12, if I paste `(int)(1f / a)` – xanatos Dec 29 '20 at 10:02
  • LINQPad version 6.11.11 (X64) says the result is 12 – ahybertz Dec 29 '20 at 10:12
  • 1
    @xanatos That is indeed interesting. I just tried it myself and reproducibly get 11 (with two separate entries in Quickwatch). How did you make your entries? – Saber Dec 29 '20 at 10:19
  • Which version of .NET Core / Framework are you running? 32 or 64-bit? – mjwills Dec 29 '20 at 10:20
  • 1
    The short answer is that float results are not consistent across runtimes and bitness. – mjwills Dec 29 '20 at 10:26
  • 1
    Does this answer your question? [Is floating-point math consistent in C#? Can it be?](https://stackoverflow.com/questions/6683059/is-floating-point-math-consistent-in-c-can-it-be) – phuzi Dec 29 '20 at 10:38
  • Probably a bug in the quickwatch of Visual Studio... Your error is reproducible, but in the same quickwatch `(int)(float)(1f / a)` is correct. I'm not sure how the quickwatch is executed and if it really does C# code or it emulates it in some way – xanatos Dec 29 '20 at 11:33
  • related and possible duplicates: [Floating point inconsistency between expression and assigned object](https://stackoverflow.com/q/13290758/995714), [Why does this double arithmetic give two different answers on a single machine?](https://stackoverflow.com/q/48784911/995714), [C# - Inconsistent math operation result on 32-bit and 64-bit](https://stackoverflow.com/q/2461319/995714), [Strange compiler behavior with float literals vs float variables](https://stackoverflow.com/q/3088372/995714) – phuclv Dec 29 '20 at 15:27

1 Answers1

1

I think it's a matter of floating point calculations

Program with 32Bit ( Any CPU -> prefer 32Bit checked )

enter image description here

Program with 64Bit

enter image description here

Visual Studio is wrapped in 32-bit, so I think that's because QuickWatch runs in 32-bit.

This is like calculating (int)(1f/12) in QuickWatch

enter image description here

So I think it would be better to check what was calculated

enter image description here

takim
  • 81
  • 7