Found this answer on the net. I'll quote the registry key and values that relate to the actual alt-tab window (the post also has the key and values for similar windows - desktop switcher (?) and window-snap-window-previewer thingy (i think)):
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\MultitaskingView\AltTabViewhost]
"Grid_leftmargin"=dword:0000005A
"Grid_rightmargin"=dword:0000005A
"Grid_topmargin"=dword:00000028
"Grid_bottommargin"=dword:00000028
"Grid_adjacentspacing"=dword:00000009
"Grid_rowspacing"=dword:0000000A
"Thumbnail_min_width"=dword:0000004B
"Thumbnail_min_height_percent"=dword:0000000F
"Thumbnail_max_height_percent"=dword:00000028
"Thumbnail_max_aspectratio"=dword:00000096
"Thumbnail_icon_size"=dword:0000001E
"Thumbnail_icon_plate"=dword:00000018
"Thumbnail_label_leftmargin"=dword:00000005
"Thumbnail_label_rightmargin"=dword:00000005
"Thumbnail_label_topmargin"=dword:00000000
"Thumbnail_label_bottommargin"=dword:00000000
"Thumbnail_hover_leftmargin"=dword:0000005
"Thumbnail_hover_rightmargin"=dword:00000005
"Thumbnail_hover_topmargin"=dword:00000005
"Thumbnail_hover_bottommargin"=dword:00000005
"Thumbnail_icon_leftmargin"=dword:00000000
"Thumbnail_icon_rightmargin"=dword:00000005
"Thumbnail_icon_topmargin"=dword:00000005
"Thumbnail_icon_bottommargin"=dword:00000000
"Thumbnail_hover_color"=dword:00000000
"Thumbnail_hover_percent"=dword:00000036
"Thumbnail_focus_border_width"=dword:00000002
"Thumbnail_focus_border_color"=dword:00FFFFFF
"Thumbnail_focus_percent"=dword:00000064
"Thumbnail_mousedown_shrinkpercent"=dword:0000005F
"Thumbnail_closebutton__width"=dword:0000001E
"Thumbnail_closebutton__height"=dword:0000001E
"ScrollButtonContainer_leftmargin"=dword:00000000
"ScrollButtonContainer_rightmargin"=dword:00000000
"ScrollButtonContainer_topmargin"=dword:00000000
"ScrollButtonContainer_bottommargin"=dword:00000000
"ScrollButtonContainer_buttonspacing"=dword:0000000F
"ScrollButtonContainer_button__width"=dword:00000069
"ScrollButtonContainer_button__height"=dword:00000028
"Desktop_background_margin_leftmargin"=dword:00000000
"Desktop_background_margin_rightmargin"=dword:00000000
"Desktop_background_margin_topmargin"=dword:00000014
"Desktop_background_margin_bottommargin"=dword:0000000
"Desktop_background_color"=dword:001D1D1D
"Desktop_layout_margin_leftmargin"=dword:0000000A
"Desktop_layout_margin_rightmargin"=dword:0000000A
"Desktop_layout_margin_topmargin"=dword:00000000
"Desktop_layout_margin_bottommargin"=dword:0000000F
"Desktop_item_spacing"=dword:0000000A
"Desktop_click_scale"=dword:0000005F
"Desktop_rest_overlay_alpha"=dword:0000008A
"New_button_glyph_size"=dword:0000000C
"New_button_plate"=dword:0000000C
"New_button_plate_hover"=dword:00121212
"New_button_glyph_color"=dword:00FFFFFF
"New_button_focus_margins_leftmargin"=dword:00000010
"New_button_focus_margins_rightmargin"=dword:00000010
"New_button_focus_margins_topmargin"=dword:00000010
"New_button_focus_margins_bottommargin"=dword:00000010
"Desktop_closebutton__width"=dword:0000001E
"Desktop_closebutton__height"=dword:0000001E
"DimmingLayer_percent"=dword:00000023
"Wallpaper"=dword:00000001
"DimmingLayer_leftmargin"=dword:00000000
"DimmingLayer_rightmargin"=dword:00000000
"DimmingLayer_topmargin"=dword:00000000
"DimmingLayer_bottommargin"=dword:00000000
"Grid_Desktop_margin"=dword:000000F
"Desktop_height"=dword:00000078
Of these the most interesting are Thumbnail_min_width, Thumbnail_min_height_percent and Thumbnail_max_height_percent - reduce these values to drastically decrease miniature size. The Grid_* values can also be reduced to pack the alt-tab grid tighter. I am currently using:
"Thumbnail_min_width"=dword:00000032
"Thumbnail_min_height_percent"=dword:00000001
"Thumbnail_max_height_percent"=dword:00000020
I would like to take this opportunity to suggest that Thumbnail_icon_size should be increased to hex 0x20 (decimal 32), as this is the standard icon size (in absence of a real 30x30 icon, Windows scales the 32x32 down 30x30, which slightly blurs more sharp-edged icons).
The post linked above claims that you do not need to reboot or restart Explorer for changes to these values to apply, and i can confirm that this is so.