Let rectangles center is coordinate origin (and center of rotation).
Large rect dimensions are 2W, 2H, small rect 2w, 2h.
First vertex (A) of small rect has initial coordinates (w,h), they become (k*w,k*h) after fitting, second one (B) : (-k*w,k*h), k is unknown yet coefficient (we need only two vertices for calculations))

For rotation angle f in range 0..Pi/2 (and Pi..3*Pi/2) side of large rect which can contain A, has starting point (F)
px = W*cos(f)-H*sin(f)
py = W*sin(f)+H*cos(f)
and direction vector
dx = sin(f)
dy = -cos(f)
and this side has parametric equation
x = px + t * dx
y = py + t * dy
We can write equation that side contains A
k*w = px + t * dx
k*h = py + t * dy
and solve it for unknowns k, t
k1 = (px * dy - py * dx) / (w * dy - h * dx)
Similarly side that might contain B
px = W*cos(f)-H*sin(f) //same point
py = W*sin(f)+H*cos(f)
and direction vector
d2x = -cos(f)
d2y = -sin(f)
-k*w = px + t * d2x
k*h = py + t * d2y
k2 = (px * d2y - py * d2x) / (-w * d2y - h * d2x)
We need to choose smaller valid value from k1 and k2 (perhaps some results are just invalid - negative etc)
Similar calculations for angle f in range Pi/2..Pi (and 3*Pi/2..2*Pi) with corresponding sides.
Implemented in Delphi to visually control correctness. Formulas depend on rotation angle quadrant: quad = 0,1,2,3 for rotation angle f in range 0..Pi/2, Pi/2..Pi and so on. (Really we can diminish ranges to 0..Pi due to the symmetry)
procedure TForm1.RectInRotRect(hw, hh, hWW, hHH: Integer; f: Double);
var
px, py, dx, dy, k, kmin: Double;
pts: array [0 .. 3] of TPoint;
quad: Integer;
begin
kmin := 999999;
quad := Floor(2 * f / Pi);
if Odd(quad) then
begin
px := hWW * cos(f) + hHH * sin(f);
py := hWW * sin(f) - hHH * cos(f);
dx := -cos(f);
dy := -sin(f);
if (hw * dy - hh * dx) <> 0 then
begin
k := (px * dy - py * dx) / (hw * dy - hh * dx);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end;
if (-hw * dx + hh * dy) <> 0 then
begin
k := (px * dx + py * dy) / (-hw * dx + hh * dy);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end
end
else
begin
px := hWW * cos(f) - hHH * sin(f);
py := hWW * sin(f) + hHH * cos(f);
dx := sin(f);
dy := -cos(f);
if (hw * dy - hh * dx) <> 0 then
begin
k := (px * dy - py * dx) / (hw * dy - hh * dx);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end;
if (-hw * dx + hh * dy) <> 0 then
begin
k := (px * dx + py * dy) / (-hw * dx + hh * dy);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end;
end;
kmin is used to scale inner rectangle
Result of work:
