-1

I have this formula to get the FinalQuantity = Quantity / Price. Quantity is declared as double and Price is float.

If I enter the values manually in the Code. FinalQuantity = 9000000 / 102.6411 it will have an output of 87684.173299000118.

But if I assign the values to the variables Quantity and Price. It will output 87684.174988370942. I have tried converting Quantity to Integer and Price to double but still getting this same result.

Could someone help me how to fix it.

Dai
  • 141,631
  • 28
  • 261
  • 374
Kris Toledo
  • 41
  • 1
  • 5
  • 1
    Post. Your. Code. – Dour High Arch Nov 08 '17 at 16:53
  • The constant 102.6411 is treated as a double, but when assigned to your variable its a float. Either way, should use decimal for currency purposes. – Valuator Nov 08 '17 at 16:56
  • 1
    Use decimal please. – Gustavo F Nov 08 '17 at 16:56
  • What do you mean by fix it - it is working as expected - what answer do you want? If you try FinalQuantity = 9000000F / 102.6411F you could even get the answer 87684.17 – PaulF Nov 08 '17 at 17:00
  • To actually recreate your values in code you need to do `FinalQuantity = 9000000D / 102.6411F` to make the numbers a double and a float. if you don't you are doing a int divided by a double instead of a double divided by a float. – Scott Chamberlain Nov 08 '17 at 17:01

1 Answers1

2

The thing is that floating point values are calculated by a value and an exponenent. This allows you have store huge number representations in small amounts of memory. This also means that you don't always get exactly the number you're looking for, just very very close. This is why when you compare floating point values, you compare them within a certain tolerance.

In other words... as others said... Always use Decimal

NicoRiff
  • 4,803
  • 3
  • 25
  • 54