Questions tagged [floating-point]
7 questions
3
votes
1 answer
Excel rounding errors
In a new worksheet (I'm using Excel 2013), starting in the top left, I type:
A
B
C
=18.6 + 67.2
=A1-85.8 = 0
=(A1-85.8) * 1000000000000
Expectation
A
B
C
85.8
TRUE
0
Actual Result
A
B
C
85.8
FALSE
0.014210855
I'm…
Bob Sammers
- 226
2
votes
1 answer
Excel floating point and boolean unexplained behavior
Thanks for checking out my question.
I was putzing around in Excel today for work, and encountered some anomalous behavior involving Excel's floating-point arithmetic and boolean tests. Here is a link to the Excel file test case I made.
In order…
Patrick
- 121
1
vote
0 answers
PostgreSQL sum function resurns different result when run multiple of times with psql
I have this query :
SELECT SUM(median) FROM my_table WHERE id = 100
-- median is of type float4
When I run from psql, it returns slightly different result for multiple of runs (For example 1000.08, 1001.02, ...)
The number of entries contributing…
Philippe
- 220
0
votes
4 answers
Excel modulus function MOD(0.9, 0.3) returns unexpected result
MOD function in Excel should return the Modulo Remainder in Excel
I believe the function =MOD(0.9, 0.3) should give 0 since basically 0.3*3 is 0.9
But it gives 5.55112E-17
Any hints to what's wrong?
0
votes
0 answers
How-to determine the number of (physical) FPUs under both Windows and Linux?
How-to determine the number of (physical) FPUs under both Windows and Linux?
The output of cat /proc/cpuinfo may contain (for example, stripped):
processor : 0
...
fpu : yes
processor : 1
...
fpu : yes
Here we see fpu : yes. Does it…
pmor
- 464
0
votes
0 answers
How to paste floating point number from text file?
I paste this text:
0.354
1.225
to Excel.
After pasting Excel interprets 0.354 as 0.354, but 1.225 as 1225.
How to make Excel interpret 1.225 as 1.225?
UPD0. I've found that Excel has somewhere two decimal digits as a default setting. If I type…
pmor
- 464
0
votes
2 answers
Excel not parsing very small floating point numbers from text to value
From a calculation I get results in text form that I want to paste into Excel.
Some of the numbers are absolute small, down to the minimum IEEE 754 double value larger zero, 2^-1074, e.g. about 10^-324.
When I paste this into Excel, Excel only…
Jens
- 121