0

For some reason if I do 1.0 - Double.leastNonzeroMagnitude it returns 1.0.

First question, why?

Second, how can I get the smallest Double that is bigger than zero and actually works when I do some math?

Ps.: The reason I'm looking for this is to avoid some NaNs I'm getting when multiplying the sigmoid derivative with a cross entropy loss derivative.

A) sigmoid: 1 / (1 + exp(-z))
B) sigmoid derivative: sigmoid * (1 - sigmoid) // s * (1 - s)
C) cross entropy loss derivative: -y / s + (1 - y) / (1 - s)

B * C = s - y

Multiplying B and C should give me an actual number, but I'm calculating B, then calculating C, then multiplying the two, which is giving me NaN in some cases (because of division by zero on B).

Ps.2: I do know that having epsilon = 1e-16 and simply add that to my functions works, but I'm looking for something "more correct", if you know what I mean.

EDIT:

Explaining why this question is not a duplicate of the one mentioned, on this one I'm asking for the solution for an "epsilon" number, not why floating point math is broken. I don't care that much about the low level details, I just want a solution that is reliable in the future.

Rodrigo Ruiz
  • 4,248
  • 6
  • 43
  • 75
  • 2
    Possible duplicate of [Is floating point math broken?](https://stackoverflow.com/questions/588004/is-floating-point-math-broken) – Oliver Charlesworth Dec 02 '17 at 21:46
  • To answer your second question - you already have the smallest `Double` that is bigger than zero :) – Oliver Charlesworth Dec 02 '17 at 21:46
  • `1.0.nextDown` may be what you are looking for, it is the greatest representable Double value strictly less than `1.0` – Martin R Dec 02 '17 at 21:50
  • @MartinR why does `nextDown` seems to work but `nextUp` doesn't? `> 1.0 - 0.0.nextUp` `> Double = 1` `> 1.0 - (1.0.nextDown)` `> Double = 0.00000000000000011102230246251565`. The reason I'm asking this is because maybe `nextDown` is not so reliable, since `nextUp` isn't. – Rodrigo Ruiz Dec 02 '17 at 22:43
  • You probably mean `1.0 - 1.0.nextUp` ? – Martin R Dec 02 '17 at 22:44
  • No, I really meant `1.0 - 0.0.nextUp`. I'd expect it to be a little less than `1.0` – Rodrigo Ruiz Dec 02 '17 at 22:45
  • 1
    `0.0.nextUp` is the same as `Double.leastNonzeroMagnitude` and approx. 4.94e-324. `1.0 - 4.94e-324` is indistinguishable from `1.0` within the precision of a 64-bit float. – Martin R Dec 02 '17 at 22:50
  • @OliverCharlesworth I edited my question to explain why this is not a duplicate. Also, about your other comment, that solution I mentioned is not reliable, in the future Swift might change that value (or depending on the operating system), that's what I meant with "more correct" solution. – Rodrigo Ruiz Dec 02 '17 at 22:51
  • What I meant that you always have `x.nextDown < x < x.nextUp`, so in your case `let eps = 1.0 - 1.0.nextDown` or `let eps = 1.0.nextUp - 1.0` instead of `let eps = Double.leastNonzeroMagnitude` *might* be what you need. – Martin R Dec 02 '17 at 22:54
  • There is no fixed epsilon value in floating-point arithmetic - it's always relative. – Oliver Charlesworth Dec 02 '17 at 23:03

0 Answers0