You can do it inside your DialogFragment, without changing
getDialog().getWindow()
            .getAttributes().windowAnimations
You should animate "decor view" in onStart and onClick.
This is the code snipped :
Create Dialog first
@Override
    public Dialog onCreateDialog(Bundle savedInstanceState) {
        return new AlertDialog.Builder(getActivity())
                .setTitle("Hello from animated dialog :)")
                .setNegativeButton("Cancel",
                        new DialogInterface.OnClickListener() {
                            public void onClick(DialogInterface dialog, int whichButton) {
                                //we have to add button here and then override it's click in onStart
                            }
                        }
                )
                .setCancelable(false)
                .create();
    }
Then override onStart method
@Override
    public void onStart() {
        super.onStart();
        AlertDialog dialog = (AlertDialog)getDialog();
        final View decorView = getDialog()
                .getWindow()
                .getDecorView();
        ObjectAnimator scaleDown = ObjectAnimator.ofPropertyValuesHolder(decorView,
                PropertyValuesHolder.ofFloat("scaleX", 0.0f, 1.0f),
                PropertyValuesHolder.ofFloat("scaleY", 0.0f, 1.0f),
                PropertyValuesHolder.ofFloat("alpha", 0.0f, 1.0f));
        scaleDown.setDuration(2000);
        scaleDown.start();
        Button positiveButton = dialog.getButton(Dialog.BUTTON_NEGATIVE);
        positiveButton.setOnClickListener(new View.OnClickListener()
        {
            @Override
            public void onClick(View v)
            {
                final View decorView = getDialog()
                        .getWindow()
                        .getDecorView();
                ObjectAnimator scaleDown = ObjectAnimator.ofPropertyValuesHolder(decorView,
                        PropertyValuesHolder.ofFloat("scaleX", 1.0f, 0.0f),
                        PropertyValuesHolder.ofFloat("scaleY", 1.0f, 0.0f),
                        PropertyValuesHolder.ofFloat("alpha", 1.0f, 0.0f));
                scaleDown.addListener(new Animator.AnimatorListener() {
                    @Override
                    public void onAnimationEnd(Animator animation) {
                        dismiss();
                    }
                    @Override
                    public void onAnimationStart(Animator animation) {
                    }
                    @Override
                    public void onAnimationCancel(Animator animation) {
                    }
                    @Override
                    public void onAnimationRepeat(Animator animation) {
                    }
                });
                scaleDown.setDuration(2000);
                scaleDown.start();
            }
        });
    }
Here is the result animation

And if you remove scale properties from my code you will get only alpha animation. Exactly as you wanted.
Remove this:
PropertyValuesHolder.ofFloat("scaleX", 1.0f, 0.0f),
PropertyValuesHolder.ofFloat("scaleY", 1.0f, 0.0f),