You should use a FragmentPagerAdapter which is adequate to put Fragments inside ViewPagers.
A complete tutorial from android developer here
EDIT: 
First of all, you need to override the getItemPosition of the adapter, and return the POSITION_NONE constant in order to be able to replace pager's fragments.
@Override
public int getItemPosition(Object object)
{
    return POSITION_NONE;
}
Then create a listner for handling fragment switching:
public interface MyListener
{
   public void onFragmentChange();
}
Your adapter should implements the listner and add some code inside onFragmentChange() method:
public class TabsPagerAdapter extends FragmentPagerAdapter {
    private FragmentManager fm;
    public TabsPagerAdapter(FragmentManager fm) {
        super(fm);
        this.fm = fm;
    }
    @Override
    public Fragment getItem(int index) {
        switch (index) {
        case 0:
            fragment1 = TopRatedFragment.newInstance(this); 
            return fragment1;
        case 1:
            // ...
            return fragment2;
        case 2:
            // ...
            return fragment3;
        case 3:
            // ...
            return fragment4;
        }
        return null;
    }
        @Override
        public int getCount() {
            // get item count - equal to number of tabs
            return 4;
        }
        public void onFragmentChange(){
            fm.beginTransaction().remove(fragment1).commit();
                        fragment1 = NewFragment.newInstance();
                        notifyDataSetChanged(); // notify the viewpager adapter that content has changed
        }
    }
Pass an instance of the listner in the fragment you want to remove/replace later :
public class TopRatedFragment extends Fragment {
    public static TopRatedFragment newInstance(MyListner listner) {
        this.myListner = listner;
        return new TopRatedFragment (); 
    }
    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
            Bundle savedInstanceState) {
        View rootView = inflater.inflate(R.layout.fragment_top_rated, container, false);
        return rootView;
    }
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        // some logical here ... and when you want to change fragment do this
        myListner.onFragmentChange();
    }
}
Here is some more explanation of this.
You should also read about FragmentStatePagerAdapter : here
Hope it helps !