The best way of avoid this problem is to use DialogFragment.
Create a new class extended to DialogFragment. Override onCreateDialog and return your old Dialog or an AlertDialog.
Them you can show it with DialogFragment.show(fragmentManager, tag).
Here an example with the activity like listener:
public class MyDialogFragment extends DialogFragment {
    public interface YesNoListener {
        void onYes();
        void onNo();
    }
    @Override
    public void onAttach(Activity activity) {
        super.onAttach(activity);
        if (!(activity instanceof YesNoListener)) {
            throw new ClassCastException(activity.toString() + " must implement YesNoListener");
        }
    }
    @Override
    public Dialog onCreateDialog(Bundle savedInstanceState) {
        return new AlertDialog.Builder(getActivity())
                .setTitle(R.string.dialog_my_title)
                .setMessage(R.string.dialog_my_message)
                .setPositiveButton(android.R.string.yes, new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {
                        ((YesNoListener) getActivity()).onYes();
                    }
                })
                .setNegativeButton(android.R.string.no, new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {
                        ((YesNoListener) getActivity()).onNo();
                    }
                })
                .create();
    }
}
And in the Activity you call:
new MyDialogFragment().show(getSupportFragmentManager(), "tag"); // or getFragmentManager() in API 11+
This kind of questions already asked,and there are also solution for it, these three questions are matched with your problem (and they answered):
Android Best way of avoid Dialogs to dismiss after a device rotation
Android DialogFragment vs Dialog
How can I show a DialogFragment using compatibility package?