With structs, I cannot directly change its fields as the whole struct is of the value type.
You cannot change fields in situations when C# makes a copy of the struct for you. In other situations, you can modify fields of a struct. Here is a small demonstration of the difference:
struct Foo {
    public int A {get;set;}
    public void SetA(int a) {
        A = a;
    }
}
class Bar {
    Foo f;
    public Foo F {
        get{return f;}
        set {f = value;}
    }
    public void SetFooA(int x) {
        f.SetA(x);
    }
}
public static void Main() {
    Bar b = new Bar();
    b.F.SetA(123);  // b.F makes a copy, so changing A fails
    Console.WriteLine("{0}", b.F.A);
    b.SetFooA(456); // Inside Bar, f.SetA is called directly, so the change works fine
    Console.WriteLine("{0}", b.F.A);
    b.F = new Foo { A = 112233 }; // This also works, because the whole F is assigned
    Console.WriteLine("{0}", b.F.A);
}
If I have a public field int X, I can modify it properly.
The same rule applies to user-defined structs, as long as you modify the whole of it. You can't modify part of an int, because it is not a composite. Modification of a struct works fine when you assign the whole struct at once. In other words, assigning
b.F = new Foo { A = 112233 };
in my example replaces assigning
B = 100;
in your example.
Demo.