4

Why is & 0xff applied to a byte variable in the reference implementation of the time-based OTP (TOTP, RFC 6238)? In my opinion, this does not change the value since the byte data type has a length of 8 bits.

byte[] hash = hmac_sha(crypto, k, msg);
int offset = hash[hash.length - 1] & 0xf;

int binary =
  ((hash[offset] & 0x7f) << 24) |
  ((hash[offset + 1] & 0xff) << 16) |
  ((hash[offset + 2] & 0xff) << 8) |
  (hash[offset + 3] & 0xff);

int otp = binary % DIGITS_POWER[codeDigits];

Besides, what is the reason to apply the XAND operator to the first element with 0x7f? It can still result in a number with 10 digits, which is larger than the largest entry in DIGITS_POWER, which is 100 000 000.

(https://www.rfc-editor.org/rfc/rfc6238, page 13)

Community
  • 1
  • 1
nrainer
  • 2,542
  • 2
  • 23
  • 35
  • See https://stackoverflow.com/questions/11380062/what-does-value-0xff-do-in-java, but you've actually asked two questions here: one about the utility of `& 0xFF` and another about why this code is clearing the most-significant bit with `& 0x7F`. – Radiodef Jan 26 '18 at 19:03

3 Answers3

2
byte b = -5;
System.out.println(b);
System.out.println(b & 0xFF);

This produces the following output:

-5
251

When & is executed, both operands first get promoted to 32 bits. -5 is represented as 11111111111111111111111111111011, while 0xFF is just '24 zeroes and 8 ones on the end', so (-5) & 0xFF gives 11111011 (leading zeroes omitted).

Roman Puchkovskiy
  • 11,415
  • 5
  • 36
  • 72
1

The literal 0xff is, as per the JLS, of type int.
Also as per the JLS, when an operation is performed between a byte and and int, the byte is safely widened to an int and the result is an int.

What this means is that the expression:

hash[offset + 1] & 0xff

Is more or less the same as:

(int)(hash[offset + 1]) & 0xff

In fact, an int is needed as the result type for the following bit-shift operation to make sense.

(hash[offset + 1] & 0xff) << 16

If the bit-shift was done on a byte, the bits would just rotate back to their original position (16 being an exact multiple of 8).

The whole code is constructing an int from a byte[].

The odd mask 0x7f used on the high byte is used instead of 0xff to mask off the left-most (or most significant) bit, which is the sign bit, to ensure the final result is not negative.

Bohemian
  • 412,405
  • 93
  • 575
  • 722
  • So what would happen without the `& 0xff` here? Would the byte value be widened to `int` with a sign extension? – Fred Larson Jan 26 '18 at 19:08
  • I think you were editing the answer to answer my question while I was typing my question. 8v) – Fred Larson Jan 26 '18 at 19:09
  • 2
    @Fred if you didn't perform the bitwise AND, and instead just cast each `byte` directly to an `int`, you would get negative numbers being bit-shifted. While this doesn't necessarily cause a problem except for 1 bit of the hash being lost 1/2 of the time, the requirement could be that all implementations should not have this side effect. – Bohemian Jan 26 '18 at 19:10
1

The operation does change the value. byte & 0xff converts the byte, which is signed, to an int representing an unsigned value corresponding to the byte's bit pattern.

Jonathan Rosenne
  • 2,159
  • 17
  • 27