0
Single singleVal = 0.9f;
int u = (int)(singleVal * 10.0f); # u = 8

I know this is probably a dumb error but I cant figure out why result of the above expression is 8. I thought it would be 9 for obvious reasons. Is their something special going on in c# that I am failing to notice?

Robo Maklen
  • 152
  • 1
  • 11
  • 2
    Welcome to Floating Point Numbers. Have fun tearing your hair out: https://www.youtube.com/watch?v=PZRI1IfStY0 – Christopher May 18 '18 at 21:34
  • [This answer](https://stackoverflow.com/a/618596/9453080) will help you understand a little better. With `Single` (`float`) you'll only get an approximation (because the value is stored as a binary point type). You may want to change to `Decimal` (`decimal`) (stored as decimal point type) – Ivan García Topete May 18 '18 at 21:57

1 Answers1

2

That has to do with how float numbers are calculated and stored in binary. If you did if ((3/10 - 2/10) == 1/10) you would actually get false because the value the computer is getting is 3.0000000001.

That happens because 1/10 can't be written in as a decimal binary. It's something like 0.0011 where the 0011 bit repeats forever.

It's similar to how 1/3 can't be rewitten as a decimal in our number system. We normally do 0.333333334. This causes a lot of weird and annoying problems.

If you want more accurate numbers, use decimal. The decimal numbers are actually stored in base ten, not base 2, so that sort of stuff doesn't happen. Note that these are slower, however.

AustinWBryan
  • 3,249
  • 3
  • 24
  • 42