You can track the state of showing/hiding the dialog in the tag attribute of the fragment view.
Initially set it as true (equivalent to shown) in onCreateView()
@Nullable
@org.jetbrains.annotations.Nullable
@Override
public View onCreateView(@NonNull LayoutInflater inflater, @Nullable @org.jetbrains.annotations.Nullable ViewGroup container, @Nullable @org.jetbrains.annotations.Nullable Bundle savedInstanceState) {
    View view = inflater.inflate(R.layout.bottom_sheet, container, false);
    view.setTag(true);
    return view
}
And whenever you show/hide it set it to true/false:
final CustomCalendarDialogFragment newFragment = new CustomCalendarDialogFragment("CHOOSE_WEEK");
button.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View view) {
        if (newFragment.isAdded()){
            newFragment.getDialog().show();
        } else {
            newFragment.show(getFragmentManager(), "CUSTOM_CALENDAR");
            newFragment.requireView().setTag(true);
        }
    }
});
And set newFragment.requireView().setTag(false); when you call getDialog().hide();
And when the app goes to the background, check that tag on onResume() to see if you want to keep the dialog shown or hide it:
@Override
protected void onResume() {
    super.onResume();
    Object tag = newFragment.requireView().getTag();
    if (tag instanceof Boolean){
        if ((!(boolean)tag))
            newFragment.getDialog().hide();
    }
}