one could cast any pointer type to any pointer type
Not quite. void * is defined to convert any object pointer1 to void * and back again with an equivalent value. It does not need any cast.
In less common architectures, the size and range of some other pointers may be smaller than void *. Casting between other pointers type may lose necessary information.
void * provides a universal object pointer type.
void *p = any_object_pointer; // No casts required
any_object_pointer = p; // No casts required
char * could substitute for void *, except conversion to and from other object pointers requires casts.
OP's char b = 5; int*a = (int*)&b; risks undefined behavior as the alignment needs of int * may exceed char *.
1 Function pointers may be wider than void*. void * and other pointers are object pointers. C lacks a truly universal pointer type.