Actually, I should've asked: how can I do this and remain CLS Compliant? Because the only way I can think of doing this is as follows, but using either __makeref, FieldInfo.SetValueDirect or just System.TypedReference in general invalidates CLS Compliance.
// code illustrating the issue:
TestFields fields = new TestFields { MaxValue = 1234 };  // test struct with one field
FieldInfo info = fields.GetType().GetField("MaxValue");  // get the FieldInfo
// actual magic, no boxing, not CLS compliant:
TypedReference reference = __makeref(fields);
info.SetValueDirect(reference, 4096);
The compliant counterpart of SetValueDirect is SetValue, but it takes an object as the target, hence my struct will be boxed, making me setting a value on a copy, not the original variable. 
A generic counterpart for SetValue doesn't exist as far as I know. Is there any other way of setting the field of a (reference to a) struct through reflection?