How to mask sensitive values in JSON for logging purposes

Question

I have several similar JSON structures that I want to write into a SQL table for logging purposes. However, some of the fields in the JSON contain sensitive information, which I want to partially mask so the full value is not visible in the log.

Here is an example of one of the JSON structures:

{
  "Vault": 1,
  "Transaction": {
    "gateway": {
      "Login": "Nick",
      "Password": "Password"
    },
    "credit_card": {
      "number": "4111111111111"
    }
  }
}

In this case I'm trying to change the 4111 credit card number so that it appears like 4xxx1111 in the JSON. I am using Newtonsoft and have deserialized the JSON into a JObject, but I am stuck on how to mask the value. I think the clue is something with JToken, but haven't figured it out yet. I'd like to make the solution as generic as possible so that it will work with any JSON structure that I might need to log out.

Any help would be appreciated.

Answer 1

Here is the approach I think I would take:

1. Make a helper method that can take a string value and obscure it in the manner you require for your log. Maybe something like this, for example:

   public static string Obscure(string s)
   {
       if (string.IsNullOrEmpty(s)) return s;
       int len = s.Length;
       int leftLen = len > 4 ? 1 : 0;
       int rightLen = len > 6 ? Math.Min((len - 6) / 2, 4) : 0;
       return s.Substring(0, leftLen) +
              new string('*', len - leftLen - rightLen) +
              s.Substring(len - rightLen);
   }
   

2. Make another helper method that can accept a JToken and a list of JSONPath expressions. In this method, match each path against the contents of the token using SelectTokens. For each match found, use the first helper method to replace the sensitive value with an obscured version.

   public static void ObscureMatchingValues(JToken token, IEnumerable<string> jsonPaths)
   {
       foreach (string path in jsonPaths)
       {
           foreach (JToken match in token.SelectTokens(path))
           {
               match.Replace(new JValue(Obscure(match.ToString())));
           }
       }
   }
   

3. Finally, compile a list of JSONPath expressions for the values that you want to obscure across all the JSON bodies you expect to get. From your example JSON above, I think you would want to obscure Password wherever it occurs and number if it occurs inside credit_card. Expressed as JSONPath, these would be $..Password and $..credit_card.number, respectively. (Keep in mind that JSONPath expressions are case sensitive in Json.Net.) Take this list and put it into a configuration setting somewhere so you can change it easily when you need to.

4. Now, whenever you want to log out some JSON, just do this:

   JToken token = JToken.Parse(json);
   string[] jsonPaths = YourConfigSettings.GetJsonPathsToObscure();
   ObscureMatchingValues(token, jsonPaths);
   YourLogger.Log(token.ToString(Formatting.None));
   

Demo fiddle: https://dotnetfiddle.net/dGPyJF

Answer 2

You can use Json Converter to convert a specific named property to do masking. Here is an example :

public class KeysJsonConverter : JsonConverter
{
private readonly Type[] _types;
private readonly string[] _pinValues= new[] { "number","Password" };

public KeysJsonConverter(params Type[] types)
{
    _types = types;
}

public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
    JToken t = JToken.FromObject(value);

        if (t.Type != JTokenType.Object)
        {
            t.WriteTo(writer);
        }
        else
        {
            JObject o = (JObject)t;
            IList<JProperty> propertyNames = o.Properties().Where(p => _pinValues.Contains(p.Name)).ToList();

            foreach (var property in propertyNames)
            {
                string propertyValue = (string)property.Value;
                property.Value = propertyValue?.Length > 2 ? propertyValue.Substring(0, 2).PadRight(propertyValue.Length, '*') : "Invalid Value";
            }
            o.WriteTo(writer);
        }
}

public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
    throw new NotImplementedException();
}

public override bool CanRead
{
    get { return false; }
}

public override bool CanConvert(Type objectType)
{
    return _types.Any(t => t == objectType);
}  

}

ANd then called the Json as : JsonConvert.SerializeObject(Result, new KeysJsonConverter(typeof(Method)))

Answer 3

You could achieve this using reflection but first, create an attribute and mark properties that you would like to hide:

[AttributeUsage(AttributeTargets.Property | AttributeTargets.Field)]
public class SensitiveDataAttribute: Attribute{}

public class User
{
    public string Username { get; set; }

    [SensitiveData]
    public string Password { get; set; }
}

public static class Obfuscator
{
    private const string Masked = "***";

    public static T MaskSensitiveData <T> (T value) 
    {
        return Recursion(value, typeof(T));
    }

     # region Recursive reflection

    private static object Recursion(object inputObj, Type type) 
    {
        try {
            if (inputObj != null) 
            {
                if (type.IsArray) 
                {
                    //Input object is an array
                    //Iterate array elements
                    IterateArrayElements(ref inputObj);
                } 
                else 
                {
                    //Input object is not an array
                    //Iterate properties
                    IteratePropertiesAndFields(ref inputObj);
                }

                return inputObj;
            }
        } 
        catch 
        {
            //Die quietly :'(
        }

    return null;
}

private static void IterateArrayElements(ref object inputObj) 
{
    var elementType = inputObj ? .GetType().GetElementType();
    var elements = (IEnumerable)inputObj;

    foreach(var element in elements) 
    {
        Recursion(element, elementType);
    }
}

private static void IteratePropertiesAndFields(ref object inputObj) 
{
    var type = inputObj ? .GetType();

    if (type == null)
        return;

    if (type.IsArray) 
    {
        //is an array
        IterateArrayElements(ref inputObj);
    } 
    else 
    {
        foreach(var property in type.GetProperties().Where(x => x.PropertyType.IsPublic)) 
        {
            if (Attribute.IsDefined(property, typeof(SensitiveDataAttribute))) 
            {
                if (property.PropertyType == typeof(string) || type == typeof(string)) 
                {
                    //we can mask only string
                    property.SetValue(inputObj, Masked);
                } 
                else 
                {
                    //all properties that are not string set to null
                    property.SetValue(inputObj, null);
                }
            } 
            else if (property.PropertyType.IsArray) 
            {
                //Property is an array
                Recursion(property.GetValue(inputObj), property.PropertyType);
            }
        }
        foreach(var property in type.GetRuntimeFields().Where(x => x.FieldType.IsPublic)) 
        {
            if (Attribute.IsDefined(property, typeof(SensitiveDataAttribute))) 
            {
                if (property.FieldType == typeof(string) || type == typeof(string)) 
                {
                    //we can mask only string
                    property.SetValue(inputObj, Masked);
                } 
                else 
                {
                    //all Fields that are not string set to null
                    property.SetValue(inputObj, null);
                }
            } 
            else if (property.FieldType.IsArray) 
            {
                //Field is an array
                Recursion(property.GetValue(inputObj), property.FieldType);
            }
        }
    }
}
 # endregion
}

And then call it like

var user = new User
{
   Username = "Joe",
   Password = "12345"
}
var myobj = Obfuscator.MaskSensitiveData<User>(user);