I've been able to change things which are descendants of the overlay like .mat-menu-content but not to overlay itself. 
Depending on what you are trying to style you may be able to do this way:
app.component.scss
::ng-deep .mat-menu-panel {
  background-color: unset;
}
::ng-deep .overlay-style-one .mat-menu-content {
  background-color: aquamarine;
}
::ng-deep .overlay-style-two .mat-menu-content {
  background-color: coral;
}
app.component.html
<ng-container>
  <button mat-button [matMenuTriggerFor]="menu1">Menu 1</button>
  <mat-menu #menu1="matMenu" class="overlay-style-one">
    <button mat-menu-item>Item 1</button>
    <button mat-menu-item>Item 2</button>
  </mat-menu>
</ng-container>
<ng-container>
  <button mat-button [matMenuTriggerFor]="menu2">Menu 2</button>
  <mat-menu #menu2="matMenu" class="overlay-style-two">
    <button mat-menu-item>Item 3</button>
    <button mat-menu-item>Item 4</button>
  </mat-menu>
</ng-container>
If you want to change the CSS of .cdk-overlay-pane itself, I can't see how I would do this. There is a single div with the cdk-overlay-container that's reused for all overlays and it's a direct child of the body element. There is no parent selector in CSS which if it existed could have helped here.
There is an OverlayConfig in the API but it seems to be for changing Overlays globally.