Monitor Enter ArgumentNullException

Question

I'm trying to understand why I get ArgumentNull exception on Monitor.Enter method. The calling code only uses a single instance of NamedLocker. I'm using a lock to protect from accessing the shared resource. Is the semaphore being disposed causing the null reference exception?

System.ArgumentNullException: Value cannot be null.
at System.Threading.Monitor.Enter(Object obj)
at System.Threading.SemaphoreSlim.<WaitUntilCountOrTimeoutAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

The exception seems to be coming when doing a WaitAsync after getting the Semaphore.
Following is my code:

public class NamedLocker
{
    private static readonly Dictionary<string, SemaphoreSlim> _lockDict =
        new Dictionary<string, SemaphoreSlim>();
    private readonly Config config;
    private static readonly object semLock = new object();
    public NamedLocker(Config config)
    {
        this.config = config;
    }

    public async Task RunWithLock(string name, Func<Task> body)
    {
        try
        {
            _ = await GetOrCreateSemaphore(name)
               .WaitAsync(TimeSpan.FromSeconds(5))
               .ConfigureAwait(false);
            await body().ConfigureAwait(false);
        }
        finally
        {
            _ = Task.Run(async () =>
            {
                await Task.Delay(this.config.LockDelayMilliseconds)
                    .ConfigureAwait(false);
                RemoveLock(name);
            });
        }
    }

    public void RemoveLock(string key)
    {
        SemaphoreSlim outSemaphoreSlim;
        lock (semLock)
        {
            if (_lockDict.TryGetValue(key, out outSemaphoreSlim))
            {
                _lockDict.Remove(key);
                outSemaphoreSlim.Release();
                outSemaphoreSlim.Dispose();
            }
        }
    }

    private SemaphoreSlim GetOrCreateSemaphore(string key)
    {
        SemaphoreSlim outSemaphoreSlim;
        lock (semLock)
        {
            if (!_lockDict.TryGetValue(key, out outSemaphoreSlim))
            {
                outSemaphoreSlim = new SemaphoreSlim(1, 1);
                _lockDict[key] = outSemaphoreSlim;
            }
            return outSemaphoreSlim;
        }
    }
}

This is the test I used to validate the exception:

[TestMethod]
public async Task PPP()
{
    var namedLocker = new NamedLocker(new Config
    {
        LockDelayMilliseconds = 100
    });

    Func<Task> funcToRunInsideLock = async () => await Task.Delay(300);
    var tasks = Enumerable.Range(0, 100).Select(
        i => namedLocker.RunWithLock((i % 2).ToString(), funcToRunInsideLock));
    await Task.WhenAll(tasks).ConfigureAwait(false);
}

Edit: Removing the Dispose() in finally block (inside RemoveLock) makes it work without the null exception. I don't understand why this would be case.