I had the same issue today.
Remove your logging configuration from Startup.cs and go to your Program.cs file and add something like:
var host = new WebHostBuilder()
    .UseKestrel()
    .UseContentRoot(Directory.GetCurrentDirectory())
    .UseIISIntegration()
    .UseStartup<Startup>()
    .ConfigureLogging((hostingContext, logging) =>
    {
        logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
        logging.AddConsole();
        logging.AddDebug();
    })
    .Build();
This used the 'builder' because the variable 'logging' is an IloggingBuilder  (whereas your code is still using ILoggerFactory)
UPDATE:  The other method I just tried is to stay inside Startup.cs but move the logging stuff from  the 'Configure' method to 'ConfigureServices' like this:
public void ConfigureServices(IServiceCollection services)
{
    services.AddLogging(loggingBuilder =>
    {
        loggingBuilder.AddConfiguration(Configuration.GetSection("Logging"));
        loggingBuilder.AddConsole();
        loggingBuilder.AddDebug();
    });
}
Perhaps keeps the Program.cs less polluted...