The problem is, plain and simple, sign extension when incorrectly treating unsigned values as signed ones.
Let's examine the bit patterns for 5 and -5 in both 8-bit and 16-bit two's complement numbers:
      8-bit          16-bit
    =========  ===================
+5  0000 0101  0000 0000 0000 0101
-5  1111 1011  1111 1111 1111 1011
When converting a number from 8-bit to 16-bit, the top bit is extended to the left. In other words, a zero-bit at the left of an 8-bit number will extend to the top half of the 16-bit number.
Similarly, a one-bit in that top bit will extend to the left.
This is the way C widens it's signed numbers (for two's complement anyway, the ones' complement and sign-magnitude encodings are a different matter but few implementations use them nowadays).
So, if you are converting signed char to signed int, or unsigned char to unsigned int, there is no problem. C will give you the correct value.
The problem exists when you switch to or from signed types to the other.
, and the problem is that the underlying data may be treated differently from what you may expect.
See, for example, the following code, with 8-bit char and 32-bit int types:
#include <stdio.h>
int main (void) {
    printf ("unsigned char  50 -> unsigned int %11u\n", (unsigned char)50);
    printf ("unsigned char -50 -> unsigned int %11u\n", (unsigned char)-50);
    printf ("unsigned char  50 ->   signed int %11d\n", (unsigned char)50);
    printf ("unsigned char -50 ->   signed int %11d\n", (unsigned char)-50);
    printf ("  signed char  50 -> unsigned int %11u\n", (  signed char)50);
    printf ("  signed char -50 -> unsigned int %11u\n", (  signed char)-50);
    printf ("  signed char  50 ->   signed int %11d\n", (  signed char)50);
    printf ("  signed char -50 ->   signed int %11d\n", (  signed char)-50);
    return 0;
}
The output of this shows the various transformations, with my annotations:
unsigned char  50 -> unsigned int          50
unsigned char -50 -> unsigned int         206 # -50 unsigned is 256-50
unsigned char  50 ->   signed int          50
unsigned char -50 ->   signed int         206 # same as above
  signed char  50 -> unsigned int          50
  signed char -50 -> unsigned int  4294967246 # sign extend, treat as unsigned
  signed char  50 ->   signed int          50                      (2^32 - 50)
  signed char -50 ->   signed int         -50
The first unusual case there is the second line. It actually takes the signed char -50 bit value, treats that as an unsigned char, and widens that to an unsigned int, preserving correctly its unsigned value of 206.
The second case does the same thing since a signed int is more than capable of holding the full range of unsigned char values (in this implementation).
The third unusual case widens -50 to a signed int and then treats the underlying bit pattern as an unsigned int, giving you the large positive value.
Note that there are no issues when the "signedness" of the value does not change.
The C standard doesn't mandate what signedness the char type has by default, it could be signed or unsigned. So, if you want truly portable code, it shouldn't contain any "naked" char types.
If you want to work with signed values, use signed values. That includes explicitly using signed char instead of char. Likewise, if you want to use unsigned value, use unsigned everywhere (including explicitly with unsigned char). Don't promote from signed to unsigned or vice versa unless you absolutely know what will happen.