0

Running into some weird outputs using floor with params for decimal places.

100.1111.floor(1)
=> 100.1
100.1111.floor(2)
=> 100.11
100.1111.floor(3)
=> 100.111
100.1111.floor(4)
=> 100.111 # why is this even the correct answer
100.1111.floor(5)
=> 100.1111

Why would the floor(4) only give 3 decimal places back and why would floor(5) give the correct response?

I attempted to see if this was an error so I tried a different float.

100.111111.floor(1)
=> 100.1
100.111111.floor(2)
=> 100.11
100.111111.floor(3)
=> 100.111
100.111111.floor(4)
=> 100.1111
100.111111.floor(5)
=> 100.11111
100.111111.floor(6)
=> 100.111111

But it seems that this isnt the case because floor works correctly for other float numbers. I tried with 100.111 and other floats and they all seem to work. Only 100.1111 seems to be giving the issue. Is this a problem with the implementation of Float#floor?

Ruby version is ruby 3.0.3p157 if that helps.

Jeff
  • 650
  • 5
  • 16
  • Do some searching, and you’ll see that the concept of float precision is pretty basic stuff (as in fundamental, not as in simple) and his been covered ad nauseam. As such, I think this question should be closed. Having said that, here’s a good place to start a little reading: https://www.rubyguides.com/2016/07/numbers-in-ruby/ –  Jan 16 '22 at 10:16
  • 2
    The _exact_ decimal value for `100.1111` as a 64-bit binary float is `100.111099999999993315213941968977451324462890625`. Maybe that gives some insights. – Stefan Jan 16 '22 at 11:53
  • @Stefan perfect explanation thank you – Jeff Jan 17 '22 at 03:16
  • @MichaelB that was not helpful even after reading through it and scouring through the answers about floating point inaccuracies on StackOverflow. It mentioned nothing of declaring a value as being inaccurate (ie `a = 0.1`); only mentioned how the math from two floating points are inaccurate. – Jeff Jan 17 '22 at 03:18
  • Read it again. Its pretty clearly spelled out that the floating point numbers themselves are imprecise. Also, I think you may have missed the fact that you ARE performing calculations in your example. You can't complete the rounding operation without doing some math. –  Jan 17 '22 at 23:22

1 Answers1

4

Float point numbers can not represent every number/precision, I suggest you to try the most famous example 0.1 + 0.2 that fails to represent 0.3 in most languages.

If you really care about precision in ruby use BigDecimal

require 'bigdecimal/util'

puts BigDecimal('100.1111').floor(1).to_digits
puts BigDecimal('100.1111').floor(2).to_digits
puts BigDecimal('100.1111').floor(3).to_digits
puts BigDecimal('100.1111').floor(4).to_digits
puts BigDecimal('100.1111').floor(5).to_digits
puts BigDecimal('100.1111').floor(6).to_digits

https://ruby-doc.org/stdlib-3.0.3/libdoc/bigdecimal/rdoc/BigDecimal.html

javiyu
  • 1,336
  • 6
  • 9