HashSet uses a HashMap internally.
HashMap stores its elements in a hash table using each Object's hashCode() method.
For Character class, the hashcode is it's corresponding int value.
public static int hashCode(char value) {
    return (int)value;
}
For int and double datatypes, these are auto-boxed into the Integer and Double classes. When you make a HashSet of ints, it uses Integer's hashCode() method, which just returns the int. So if you add ints, they get stored in a sorted order. 
But for double, Double's hashCode() method is much more complicated, because of the way doubles are represented in memory.
But, over time, when the elements exceed the bucket size, you'll see it won't maintain order.