I found a solution with some help. See github.com/NLog/NLog/issues/172 thanks to Julian
It is possible to write a custom WrapperLayoutRenderer that switches the culture before the exception is logged.
namespace NLog.LayoutRenderers.Wrappers
{
    [LayoutRenderer("InvariantCulture")]
    [ThreadAgnostic]
    public sealed class InvariantCultureLayoutRendererWrapper : WrapperLayoutRendererBase
    {
        protected override string Transform(string text)
        {
            return text;
        }
        protected override string RenderInner(LogEventInfo logEvent)
        {
            var currentCulture = Thread.CurrentThread.CurrentUICulture;
            try
            {
                Thread.CurrentThread.CurrentUICulture = CultureInfo.InvariantCulture;
                return base.RenderInner(logEvent);
            }
            finally
            {
                Thread.CurrentThread.CurrentUICulture = currentCulture;
            }
        }
    }
}
It must be registered before any logger is created
ConfigurationItemFactory.Default.LayoutRenderers.RegisterDefinition("InvariantCulture", typeof(InvariantCultureLayoutRendererWrapper));
And can be used like that in the config
layout="${longdate}|${level:uppercase=true}|${logger}|${message}|${InvariantCulture:${exception:format=tostring}}"