Edit 20220715: After some feedback on our approach from Dominic Baier we removed our Scoped UserSubProvider service in favour of using AuthenticationStateProvider instead. This has simplified our approach. I have edited the following answer to reflect this change.
This approach combines advice from Microsoft on how to pass tokens to a Blazor Server app (here), with server side storage of tokens in a Singleton service for all users (inspired by Dominick Baier’s Blazor Server sample project on GitHub here).
Instead of capturing the tokens in the _Host.cshtml file and storing them in a Scoped service (like Microsoft do in their example), we use the OnTokenValidated event in a similar way to Dominick Baier’s sample, storing the tokens in a Singleton service that holds tokens for all Users, we call this service ServerSideTokenStore.
When we use our HttpClient to call an API and it needs an access_token (or refresh_token), then it retrieves the User’s sub from an injected AuthenticationStateProvider, uses it to call ServerSideTokenStore.GetTokensAsync(), which returns a UserTokenProvider (similar to Microsoft’s TokenProvider) containing the tokens. If the HttpClient needs to refresh the tokens then it populates a UserTokenProvider and saves it by calling ServerSideTokenStore.SetTokensAsync().
Another issue we had was if a separate instance of the web browser is open while the app restarts (and therefore loses the data held in ServerSideTokenStore) the user would still be authenticated using the cookie, but we’ve lost the access_token and refresh_token. This could happen in production if the application is restarted, but happens a lot more frequently in a dev environment. We work around this by handling OnValidatePrincipal and calling RejectPrincipal() if we cannot get a suitable access_token. This forces a round trip to IdentityServer which provides a new access_token and refresh_token. This approach came from this stack overflow thread.
(For clarity/focus, some of the code that follows excludes some standard error handling, logging, etc.)
Getting the User sub claim from AuthenticationStateProvider
Our HttpClient gets the user's sub claim from an injected AuthenticationStateProvider. It uses the userSub string when calling ServerSideTokenStore.GetTokensAsync() and ServerSideTokenStore.SetTokensAsync().
    var state = await AuthenticationStateProvider.GetAuthenticationStateAsync();
    string userSub = state.User.FindFirst("sub")?.Value;
UserTokenProvider
    public class UserTokenProvider
    {
        public string AccessToken { get; set; }
        public string RefreshToken { get; set; }
        public DateTimeOffset Expiration { get; set; }
    }
ServerSideTokenStore
    public class ServerSideTokenStore
    {
        private readonly ConcurrentDictionary<string, UserTokenProvider> UserTokenProviders = new();
    
        public Task ClearTokensAsync(string userSub)
        {
            UserTokenProviders.TryRemove(userSub, out _);
            return Task.CompletedTask;
        }
    
        public Task<UserTokenProvider> GetTokensAsync(string userSub)
        {
            UserTokenProviders.TryGetValue(userSub, out var value);
            return Task.FromResult(value);
        }
    
        public Task StoreTokensAsync(string userSub, UserTokenProvider userTokenProvider)
        {
            UserTokenProviders[userSub] = userTokenProvider;
            Return Task.CompletedTask;
        }
    }
Startup.cs ConfigureServices (or equivalent location if using .NET 6 / whatever)
    public void ConfigureServices(IServiceCollection services)
    {
        // …
        services.AddAuthentication(…)
        .AddCookie(“Cookies”, options =>
        {
            // …
            options.Events.OnValidatePrincipal = async context =>
            {
                if (context.Principal.Identity.IsAuthenticated)
                {
                    // get user sub 
                    var userSub = context.Principal.FindFirst(“sub”).Value;
                    // get user's tokens from server side token store
                    var tokenStore =
                        context.HttpContext.RequestServices.GetRequiredService<IServerSideTokenStore>();
                    var tokens = await tokenStore.GetTokenAsync(userSub);
                    if (tokens?.AccessToken == null 
                        || tokens?.Expiration == null 
                        || tokens?.RefreshToken == null)
                    {
                        // if we lack either an access or refresh token,
                        // then reject the Principal (forcing a round trip to the id server)
                        context.RejectPrincipal();
                        return;
                    }
                    // if the access token has expired, attempt to refresh it
                    if (tokens.Expiration < DateTimeOffset.UtcNow) 
                    {
                        // we have a custom API client that takes care of refreshing our tokens 
                        // and storing them in ServerSideTokenStore, we call that here
                        // …
                        // check the tokens have been updated
                        var newTokens = await tokenStore.GetTokenAsync(userSubProvider.UserSub);
                        if (newTokens?.AccessToken == null 
                            || newTokens?.Expiration == null 
                            || newTokens.Expiration < DateTimeOffset.UtcNow)
                        {
                            // if we lack an access token or it was not successfully renewed, 
                            // then reject the Principal (forcing a round trip to the id server)
                            context.RejectPrincipal();
                            return;
                        }
                    }
                }
            }
        }
        .AddOpenIdConnect(“oidc”, options =>
        {
            // …
            options.Events.OnTokenValidated = async n =>
            {
                var svc = n.HttpContext.RequestServices.GetRequiredService<IServerSideTokenStore>();
                var culture = new CultureInfo(“EN”) ;
                var exp = DateTimeOffset
                          .UtcNow
                          .AddSeconds(double.Parse(n.TokenEndpointResponse !.ExpiresIn, culture));
                var userTokenProvider = new UserTokenProvider() 
                {
                    AcessToken = n.TokenEndpointResponse.AccessToken,
                    Expiration = exp,
                    RefreshToken = n.TokenEndpointResponse.RefreshToken
                }
                await svc.StoreTokensAsync(n.Principal.FindFirst(“sub”).Value, userTokenProvider);
            };
            // …
        });
        // …
    }