The basic mechanism is treated in this question. The problem that is addressed here is that both in map and flatMap you need to reference the outer Generator from the newly created inner Generator.
If you would write this here instead of self:
def map[S](f: T => S): Generator[S] = new Generator[S] {
  def generate = f(this.generate) // wrong!
}
you would not identify the outer but the inner instance. this always refers to the "closest" class instance seen from the spot where you use it.
Similarly, if you added a method def self = this to Generator, you then call again the wrong self in the inner scope (the new generator has its own self method now).
The alias in comparison is not a method, it doesn't exist "twice" in the code example so to speak, so there is no shadowing happening when you write new Generator. This is just a more cleaned up way for
def map[S](f: T => S): Generator[S] = {
  val outer = this  // capture outer instance, since `this` will change meaning
  new Generator[S] {
    def generate = f(outer.generate)  // refer to the outer instance
  }
}