This is known as the curiously recurring template pattern.
As far as I know, RollingFileAppender.Builder doesn't have any subclasses, so this isn't technically needed for RollingFileAppender.Builder to work. It could have been declared as a non-generic type:
public static class RollingFileAppender.Builder extends 
    AbstractOutputStreamAppender.Builder<RollingFileAppender.Builder>
and its builder methods, which all return a B (e.g. withAdvertise), can just as well return RollingFileAppender.Builder.
However, consider the case where someone is trying to write a subclass of this non-generic class:
public class FooBuilder extends RollingFileAppender.Builder {
    public FooBuilder withFoo(String foo) {
        // ...
        return this;
    }
}
// Let's try using MyBuilder:
new FooBuilder()
    .withAdvertise(false) // let's call one of the methods in RollingFileAppender.Builder first...
    .withFoo("Foo") // and now we try to call my subclass's method
    .build()
The call withFoo would not compile. This is because in our hypothetical, non-generic RollingFileAppender.Builder, withAdvertise returns RollingFileAppender.Builder, not MyBuilder.
Making RollingFileAppender.Builder generic, and making its builder methods return the generic type parameter B solves this problem. Because then we could write:
// in this pattern,
// you are always expected to put the class that you are declaring as B
public class MyBuilder extends RollingFileAppender.Builder<MyBuilder>
Whatever methods declared in RollingFileAppender.Builder that returns a B will now return a MyBuilder.
new MyBuilder()
    .withAdvertise(false) // this now returns a MyBuilder
    .withFoo("Foo") // and this now works
    .build()
It would also be helpful to see the inheritance hierarchy of RollingFileAppender.Builder:
java.lang.Object
    org.apache.logging.log4j.core.filter.AbstractFilterable.Builder<B>
        org.apache.logging.log4j.core.appender.AbstractAppender.Builder<B>
            org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.Builder<B>
                org.apache.logging.log4j.core.appender.RollingFileAppender.Builder<B>
As you can see, this class inherits a lot of builder methods. If none of the classes followed this curiously recurring template pattern, you would not be able to call methods declared in RollingFileAppender.Builder after using one of the methods inherited from its superclasses.
new RollingFileAppender.Builder()
    .setImmediateFlush(true) // this is declared in AbstractOutputStreamAppender.Builder
    // so the above would have returned an AbstractOutputStreamAppender.Builder
    .withAdvertise(true); // now you wouldn't be able to do this
To summarise, this B is simply a way for all the builder methods to return whatever Builder subclass that you are using, rather than the class that declared them. Although it is not needed for RollingFileAppender.Builder since it has no subclasses, the authors still declared it like that, presumably to follow the pattern.
As for why static classes can have constructors, static classes just means that you don't need an instance of the outer class (RollingFileAppender) in order to instantiate the nested class. It does not prevent declaring constructors.