This problem is not specific to nullable enums. Even if you use a non-nullable enum, it won't work. That's because while specific enum types are value types, System.Enum is a reference type as explained here. So, you need to add a struct constraint in addition to the Enum constraint.
The following should work:
public static T? ToEnum<T>(this string enumName) 
    where T : struct, Enum
    => Enum.TryParse<T>(enumName, out T val) ? val : default(T?);
Also, note that you can't use null in the second branch of the ternary operator because the compiler can't infer the type. Use default(T?) instead.
Example of usage:
enum MyEnum { a, b, c }
static void Main(string[] args)
{
    var e1 = "b".ToEnum<MyEnum>();
    var e2 = "d".ToEnum<MyEnum>();
    Console.WriteLine("e1 = " + (e1?.ToString() ?? "NULL"));  // e1 = b
    Console.WriteLine("e2 = " + (e2?.ToString() ?? "NULL"));  // e2 = NULL
}