As there is currently no way to provide a context to listeners, need to keep track of all attached listeners and remove them as shown by David in the answer here: https://stackoverflow.com/a/33782474/1820644
For managing value listeners inside the RecyclerView I have came up with a solution that works well. I am using FirebaseUI's RecyclerView, but the solution should work on every RecyclerView.
Step 1: Firstly create a global Map<String,ValueEventListener> in your activity:
HashMap<String, ValueEventListener> mRecyclerViewFirebaseListeners = new HashMap<>();
Step 2: Bind ValueEventListener in populateViewHolder (in firebaseUI) or onBindViewHodler and add it to our map mRecyclerViewFirebaseListeners. Also, attach a Tag to view holder with the value of our key
protected void populateViewHolder(final ViewHolder viewHolder,
                                  Boolean model, int position) {
    Log.v(TAG, "Populating viewHolder for position: " + position);
    // Get the key and add listener
    String key = this.getRef(position).getKey();
    ValueEventListener listener = mRootRef.child("events").child(key)
            .addValueEventListener(new ValueEventListener() {
                public void onDataChange(DataSnapshot dataSnapshot) {
                    Log.v(TAG, "Event Data Change");
                    // ...
                }
                @Override
                public void onCancelled(DatabaseError databaseError) {
                    Log.v(TAG, databaseError.getMessage());
                }
            });
    Log.v(TAG, "Listener set for event key: " + key + " (position: " +
            position + ")");
    // Add to our map and set tag on view holder
    mRecyclerViewFirebaseListeners.put(key, listener);
    viewHolder.itemView.setTag(R.id.TAG_RCV_EVENT_KEY, key);
    Log.d(TAG, "Added tag for position:" + position);
    // do other work here
}
Step 3: Remove a listener for the view holder when a view is recycled.
@Override
public void onViewRecycled(ViewHolder holder) {
    super.onViewRecycled(holder);
    // Get key from the view holder
    String key = (String) holder.itemView.getTag(R.id.TAG_RCV_EVENT_KEY);
    Log.v(TAG, "Recycling view set for key: " + key);
    // Remove listener for the retrieved key and also remove it from our map
    mRootRef.child("events").child(key)
            .removeEventListener(mRecyclerViewFirebaseListeners.get(key));
    mRecyclerViewFirebaseListeners.remove(key);
    Log.v(TAG, "On View Recycled, Removed Event listener for key: " + key);
    // ...
}
Step 4: Remove remaining listeners in onDestroy()
@Override
public void onDestroyView() {
    super.onDestroyView();
    // Iterate over map and remove listeners for the key
    for (Map.Entry<String, ValueEventListener> entry : mRecyclerViewFirebaseListeners.entrySet()) {
        String key = entry.getKey();
        ValueEventListener value = entry.getValue();
        mRootRef.child("events").child(key).removeEventListener(value);
        Log.v(TAG, "Removed Event listener for key: " + key);
    }
}
This will remove all your listeners from recycler view. 
If you have more value listeners other than recyclerview, use a new map for those others all combine it in this map if it matches the key pattern for the same path.
Alternatively you can put a full path instead of just a key as the key of a Map. Like in example we have used value of 'key' under the child events as the key of a Map, but you can put events/<key> as the key of a Map and remove in one go in onDestroy with mRootRef.child("<your_combined_key_from_map>").removeListener(listener).
Hope this helps someone.