Problem with Caesar cipher in x86 assembly

Question

I have the following instructions as part of a Caesar cipher program.

overFlow:
    sub bl, 1Ah
    ret

underFlow:
    add bl, 1Ah
    ret

correctFlow:
    cmp bl, 7Ah
    jg overFlow
    cmp bl, 61h
    jl underFlow
    ret

enc_byte:
    add bl, encOffset
    call correctFlow
    ret

An ASCII lowercase letter is put into BL and after enc_byte is called, it shifts the letter by encOffset letters and corrects for an overflow.

But for some reason the compare in correctFlow doesn't work correctly. When BL=8Dh in correctFlow, the jg overFlow instruction does not jump, and instead jl underFlow jumps after the second cmp. Why is this happening? 8Dh is clearly greater than 7Ah, so why doesn't it jump as expected?

I know the returns are weird. The overFlow and underFlow labels are the ones that return the call to correctFlow. This is intentional and as far as I know, doesn't have anything to do with the issue.

score 1 · Accepted Answer · answered Mar 12 '22 at 17:34

This happens because jg and jl treat the outcome of cmp as if the two operands were signed numbers.

7Ah and 8Dh represent signed numbers +122 and -115, respectively. Obviously, the latter is the smallest.

What you need is unsigned comparison. Use instructions ja and jb instead.

Problem with Caesar cipher in x86 assembly

1 Answers1