Here's a recursive solution that works for dungeons of arbirary size and relatively few monsters.
If there is one monster (x, y) in the dungeon (n, w, s, e), there are two cases. Case 1 is trivial: The monster is outside the dungeon. Then the maximum rectangle is the dungeon. (Dungeons are always rectangular, right?).
In Case 2, the maximum rectangle is one of the rectangles north, west, south or east of the monster:
NNNNNNNNNN    ....EEEEEE    ..........    WWW.......
NNNNNNNNNN    ....EEEEEE    ..........    WWW.......
NNNNNNNNNN    ....EEEEEE    ..........    WWW.......
NNNNNNNNNN    ....EEEEEE    ..........    WWW.......
NNNNNNNNNN    ....EEEEEE    ..........    WWW.......
...@......    ...@EEEEEE    ...@......    WWW@......
..........    ....EEEEEE    SSSSSSSSSS    WWW.......
..........    ....EEEEEE    SSSSSSSSSS    WWW.......
..........    ....EEEEEE    SSSSSSSSSS    WWW.......
Now apply this reasoning recursively for your list of monster locations and keep track of the maximum so far. Here's some pseudo code:
max_area = 0
max_rect = Null
sub find_max_rect(Dungeon, Monsters)
    if area(Dunegon) <= max_area: 
        return                      # save time for small dungeons
    if Monsters is empty:
        if area(Dungeon) > max:
            max_rect = Dungeon
            max_area = area(Dungeon)
    else
        M = Monsters.pop()          # Remove head from list
        if M in Dungeon:
            find_max_rect(Subrect(Dungeon, M, North), Monsters)
            find_max_rect(Subrect(Dungeon, M, East), Monsters)
            find_max_rect(Subrect(Dungeon, M, South), Monsters)
            find_max_rect(Subrect(Dungeon, M, West), Monsters)
        else:
            find_max_rect(Dungeon, Monsters)
Note: I've actually made a glaring mistake in the sketches above: @ represents, of course, the player and not a monster.