The rotation is not taken into account when measuring the view and calculating the scale ratio. A possible solution is to do it yourself :
public class RotatedImageView extends ImageView {
    ...
    constructors
    ...
    private double mRotatedWidth;
    private double mRotatedHeight;
    private boolean update() {
        Drawable d = getDrawable();
        if (d == null) {
            return false;
        }
        int drawableWidth = d.getIntrinsicWidth();
        int drawableHeight = d.getIntrinsicHeight();
        if (drawableWidth <= 0 || drawableHeight <= 0) {
            return false;
        }
        double rotationRad = getRotation() / 180 * Math.PI;
        // calculate intrinsic rotated size
        // see diagram
        mRotatedWidth = (Math.abs(Math.sin(rotationRad)) * drawableHeight
                + Math.abs(Math.cos(rotationRad)) * drawableWidth);
        mRotatedHeight = (Math.abs(Math.cos(rotationRad)) * drawableHeight
                + Math.abs(Math.sin(rotationRad)) * drawableWidth);
        return true;
    }
    protected void onMeasure(int widthMeasureSpec, int heightMeasureSpec) {
        if (update()) {
            double ratio = mRotatedWidth / mRotatedHeight;
            int wMax = Math.min(getDefaultSize(Integer.MAX_VALUE, widthMeasureSpec), getMaxWidth());
            int hMax = Math.min(getDefaultSize(Integer.MAX_VALUE, heightMeasureSpec), getMaxHeight());
            int w = (int) Math.min(wMax, hMax * ratio);
            int h = (int) Math.min(hMax, wMax / ratio);
            setMeasuredDimension(w, h);
        } else {
            super.onMeasure(widthMeasureSpec, heightMeasureSpec);
        }
    }
    private final float[] values = new float[9];
    protected void onDraw(Canvas canvas) {
        if (update()) {
            int availableWidth = getMeasuredWidth();
            int availableHeight = getMeasuredHeight();
            float scale = (float) Math.min(availableWidth / mRotatedWidth, availableHeight / mRotatedHeight);
            getImageMatrix().getValues(values);
            setScaleX(scale / values[Matrix.MSCALE_X]);
            setScaleY(scale / values[Matrix.MSCALE_Y]);
        }
        super.onDraw(canvas);
    }
    @Override
    public void setRotation(float rotation) {
        super.setRotation(rotation);
        requestLayout();
    }
}
adjustViewBounds must be true :
<com.mypackage.RotatedImageView
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_margin="20dp"
    android:adjustViewBounds="true"
    android:rotation="90"
    android:maxWidth="100dp"
    android:maxHeight="100dp"
    android:scaleType="fitCenter"
    android:src="@drawable/test" />
A nice explanation of the calculation, courtesy of Cheticamp :




UPDATE: Now trying to adjust the bounds. There is no difference between wrap_content and match_parent (both grow as much as possible, based on the image aspect). You should instead use maxWidth and / or maxHeight, or put it in a LinearLayout with a 0 size and a weight.     
It is also not animatable, adjusting bounds while animating requires a layout pass for each frame, which is very inefficient. See the other answer for a version usable with View.animate()