I found pretty easy way how to handle this:
1) Make global variable of type ActionMode:
ActionMode actionMode = null;
2a) Assign ActionMode in onCreateActionMode() method:
@Override
public boolean onCreateActionMode(ActionMode mode, Menu menu) {
    getActivity().getMenuInflater().inflate(R.menu.action_mode, menu);
    actionMode = mode;
    return true;
}
2b) Also put this in onDestroyActionMode() method:
@Override
public void onDestroyActionMode(ActionMode mode) {
    actionMode = null;
}
3) Override onDestroy() method (you can also use onPause() if you want to close CAB every time fragment is paused, but this may be annoying to users):
@Override
public void onDestroy() {
    super.onDestroy();
    //Destroy action mode
    if(actionMode != null)
        actionMode.finish();
}
That's it, now every time you'll replace Fragment, it will cancel ActionMode.