If you do not wish to clear polylines and only the markers need to be removed follow the steps below. 
First create a new Marker Array like below
List<Marker> AllMarkers = new ArrayList<Marker>();
Then when you add the marker on the google maps also add them to the Marker Array (its AllMarkers in this example)
for(int i=0;i<places.length();i++){
                LatLng location = new LatLng(Lat,Long);
                MarkerOptions markerOptions = new MarkerOptions();
                markerOptions.position(location);
                markerOptions.title("Your title");
                 Marker mLocationMarker = Map.addMarker(markerOptions); // add the marker to Map
                    AllMarkers.add(mLocationMarker); // add the marker to array
                }
then finally call the below method to remove all markers at once 
 private void removeAllMarkers() {
        for (Marker mLocationMarker: AllMarkers) {
            mLocationMarker.remove();
        }
        AllMarkers.clear();
    }
call from anywhere to remove all markers 
removeAllMarkers();
I found this solution when i was looking for a way to remove only the map markers without clearing the polylines. Hope this will help you too.