Stack Overflow
Stack Overflow

Questions tagged [numeric-conversion]

24 questions
255
votes
6 answers

Why is 0 < -0x80000000?

I have below a simple program: #include #define INT32_MIN (-0x80000000) int main(void) { long long bal = 0; if(bal < INT32_MIN ) { printf("Failed!!!"); } else { printf("Success!!!"); …
Jayesh Bhoi
  • 24,694
  • 15
  • 58
  • 73
81
votes
7 answers

Is it more efficient to perform a range check by casting to uint instead of checking for negative values?

I stumbled upon this piece of code in .NET's List source code: // Following trick can reduce the range check by one if ((uint) index >= (uint)_size) { ThrowHelper.ThrowArgumentOutOfRangeException(); } Apparently this is more efficient (?) than if…
enzi
  • 4,057
  • 3
  • 35
  • 53
30
votes
3 answers

How does testing if a string is 'greater' than another work in Bash?

In Bash I can write the following test [[ "f" > "a" ]] which results in returning 0, i.e. true. How does bash actually perform this string comparison? From my understanding > does an integer comparison. Does it try to compare the ASCII value of the…
helpermethod
  • 59,493
  • 71
  • 188
  • 276
21
votes
2 answers

Best practice in C++ for casting between number types

What is the best practice for casting between the different number types? Types float, double, int are the ones I use the most in C++. An example of the options where f is a float and n is a double or an int: float f = static_cast(n); float f…
keith
  • 5,122
  • 3
  • 21
  • 50
9
votes
3 answers

XSLT Compare Numbers as Strings

Background I was recently surprised to notice that XSL was able to intelligently handle numbers; i.e. knowing to treat numbers in text as numeric when performing comparisons (i.e. it understood that 7 < 10 rather than thinking '10' < '7'). In my…
JohnLBevan
  • 22,735
  • 13
  • 96
  • 178
6
votes
4 answers

SQL Server CONVERT(NUMERIC(18,0), '') fails but CONVERT(INT, '') succeeds?

PRINT CONVERT(NUMERIC(18,0), '') produces Error converting data type varchar to numeric. However, PRINT CONVERT(INT, '') produces 0 without error... Question: Is there some SQL Server flag for this or will I need to do case statements for…
Zachary Scott
  • 20,968
  • 35
  • 123
  • 205
4
votes
1 answer

convert (long mantissa) and (sbyte exponent) to decimal

upd placed my version in the description at the end I need to convert mantissa and exponent to decimal. This is how I coded that: // long field.Decimal.Mantissa // sbyte field.Decimal.Exponent decimal MDEntryPx = field.Decimal.Mantissa *…
Oleg Vazhnev
  • 23,239
  • 54
  • 171
  • 305
4
votes
4 answers

convert Arabic numerical to English

i am looking for a way to convert the Arabic numerical string "٠١٢٣٤٥٦٧٨٩" to an English numerical string "0123456789" Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click dim Anum as…
hamitay
  • 53
  • 2
  • 7
4
votes
2 answers

C++ how to set a fixed decimal precision for a float

I have an API call which returns a double. The double's decimal length can variate from many decimal places to a few (it all comes down to the state of an actuator). This double represents the current position on the range radius of an actuator. I…
Ælex
  • 14,432
  • 20
  • 88
  • 129
2
votes
3 answers

Is there a purpose to using boost::numeric_cast(long) (i.e., converting from long to double)?

I thought that this conversion cannot fail. So boost::numeric_cast(long) should produce the same result as just a regular cast. Is this correct? If so, why is the boost::numeric_cast slower than a regular cast? Is there some sort of check it…
jhourback
  • 4,381
  • 4
  • 26
  • 31
2
votes
2 answers

Can one assign 4 little-endian-ordered bytes of an unsigned integer to a Java primitive using just bitwise operators?

I want to read 4 bytes which are a little-endian encoding of an unsigned 32-bit integer, and assign the value to a Java int (Yes, in reality I will use a 'long', but in this case I 'know' that the unsigned value is never so big that it will…
David Bullock
  • 6,112
  • 3
  • 33
  • 43
2
votes
0 answers

swift 2.0 NSDecimalNumber possible discrepency converting to long

I can't make heads or tails of this. I am using NSDecimalNumber to truncate the fractional portion from a string. This works in most cases, but not apparently in the case of infinite decimals (or just too many). Here is an…
absmiths
  • 1,144
  • 1
  • 12
  • 21
2
votes
0 answers

.NET Double storage in DataTable?

.NET looks like it's formatting a double inside a DataRow in the wrong format, so i can't load it in a second moment. If I store "0.000000001", .NET stores it as "1E-09.0", when it should be "1E-09". So Convert.ToDouble("1E-09.0") returns a…
sawk
  • 87
  • 7
2
votes
1 answer

how to take apart numbers into integer & fractional part in scheme?

I want a function number->second-pair that accepts a number and returns a pair of integer representing its integer part & fractional part multipled with 1000000. i.e.: (number->second-pair 1) ; returns (1 . 0) ; 1 sec -> (1 sec + 0…
Javran
  • 3,394
  • 2
  • 23
  • 40
1
vote
1 answer

How do I convert the value of a TextView to integer

I am designing a basic BMI calculator and I have the calculator working, but I need to convert the calculated answer from a TextView which is a double to an integer to enable me right statements with <> operators based on the calculated answer. How…
Oyubu Obukohwo
  • 45
  • 1
  • 8
1
2