Your properties have protected setters, and DataContractSerializer will only serialize properties that are fully public, unless marked with data contract attributes.
Once you apply the necessary [DataContract] and [DataMember] attributes, you will discover a second problem, namely that DataContractSerializer cannot serialize values of type System.Type.  (Demo fiddle #1 here).  To serialize your Type RefEntityType property, you will need to use some sort of surrogate for Type that can be serialized, for instance a string.
Thus the following version of RefDomain can be serialized via the data contract serializer:
public class RefDomain
{
    private string name;
    private RefDomain parent;
    private Type entityType;
    [DataMember]
    public string Name
    {
        get => this.name;
        protected set => this.name = value;
    }
    [DataMember]
    public RefDomain Parent
    {
          get => this.parent;
          protected set => this.parent = value;
    }
    public Type RefEntityType
    {
          get => this.entityType;
          protected set => this.entityType = value;
    }
    [DataMember(Name = "RefEntityType")]
    string RefEntityTypeString 
    {
        get
        {
            return RefEntityType?.FullName;
        }
        set
        {
            // Note that simply deserializing a type supplied over the wire will make your application vulnerable to type injection attacks,
            // in which an attacker tricks you into constructing a type that effects an attack when constructed, deserialized or disposed.  
            // This is a known vulnerability with Json.NET's TypeNameHandling.None.  See for details
            // https://www.blackhat.com/docs/us-17/thursday/us-17-Munoz-Friday-The-13th-JSON-Attacks-wp.pdf
            // https://stackoverflow.com/questions/39565954/typenamehandling-caution-in-newtonsoft-json/
            // https://stackoverflow.com/questions/49038055/external-json-vulnerable-because-of-json-net-typenamehandling-auto
            var type = value == null ? null : Type.GetType(value);
            // Check the type against a whitelist here?
            RefEntityType = type;
        }
    }
    
    // Your class was had no way to set the protected properties, so I added a parameterized constructor.
    public RefDomain() { }
    public RefDomain(string name, RefDomain parent, Type entityType) => (this.name, this.parent, this.entityType) = (name, parent, entityType);
}
Notes:
- Deserializing a type supplied over the wire will make your application vulnerable to attack gadget deserialization attacks, in which an attacker tricks your app into constructing a type that effects an attack when constructed, deserialized or disposed.  This is a known vulnerability with Json.NET's - TypeNameHandling.  See for details:
 - If your app ever constructs an instance of your - RefEntityType, you app may become vulnerable to such attacks.  To prevent this, you could check the deserialized type against a whitelist of allowed types.
 
- In your question, you ask why the partially public properties of - RefDomainare serialized when it is serialized as a member of some other root object- RefEntity.  There is not enough information in your question to explain why this is happening.  Perhaps the endpoint where the- RefEntityis serialized is using a has subclassed- RefDomain, or uses a serialization surrogate, or has set- DataContractSerializerSettings.SerializeReadOnlyTypes = true.
 - But regardless of why partially public properties - RefDomainproperties are serialized when inside- RefEntity, if you want those properties serialized in all contexts, it is correct and sufficient to mark- RefDomainwith- [DataContract]and the properties with- [DataMember]
 
Demo fiddle #2 here.