How to fix incorrect position I get when dispatching an update to ListUpdateCallback using DiffUtil

Question

I have an Android application where the user can modify multiple String items using EditText at the same time, so I need to figure out what have changed in order to notify the server about the changes (create new item, update existing item or delete the latter), that is why I am using DiffUtil with ListUpdateCallback to do that. The problem that if the old list has size of 2 items; and I removed the item at index 0, then added 3 items to the end of the list, I get a callback to onInserted with incorrect position paramter which leads to IndexOutOfBoundsException, (and it is the behavior of removing any item in the old list except for the last one) please take a look at this GIF that shows the problem.

I have tried the following code with other changes made to the new list like removing at index 1 and then adding 3 items to end of the list and it works fine!

The array I am working with is of type Answer which is a class:

public class Answer {
    private String id;
    private String questionId;
    private String text;
    private Integer count;
}

Please note that 2 different objects could be identified by the id, if two items the same; then the content could be identified by the text.

DiffUtil.Callback

public class AnswersDiffCallback extends DiffUtil.Callback {

    List<Answer> newAnswers;
    List<Answer> oldAnswers;

    public AnswersDiffCallback(List<Answer> newAnswers, List<Answer> oldAnswers) {
        this.newAnswers = newAnswers;
        this.oldAnswers = oldAnswers;
    }

    @Override
    public int getOldListSize() {
        return oldAnswers == null ? 0 : oldAnswers.size();
    }

    @Override
    public int getNewListSize() {
        return newAnswers == null ? 0 : newAnswers.size();
    }

    @Override
    public boolean areItemsTheSame(int oldItemPosition, int newItemPosition) {
        Answer oldAnswer = oldAnswers.get(oldItemPosition);
        Answer newAnswer = newAnswers.get(newItemPosition);
        return Objects.equals(oldAnswer.getId(), newAnswer.getId());
    }

    @Override
    public boolean areContentsTheSame(int oldItemPosition, int newItemPosition) {
        Answer oldAnswer = oldAnswers.get(oldItemPosition);
        Answer newAnswer = newAnswers.get(newItemPosition);
        return Objects.equals(oldAnswer.getText(), newAnswer.getText());
    }
}

in ListUpdateCallback I am trying to log the callbacks I get in order to test if it is working before I talk to the server.

Log.d(TAG, "answers: oldAnswers = " + Utils.serializeObject(oldAnswers));
Log.d(TAG, "answers: newAnswers = " + Utils.serializeObject(newAnswers));
Log.d(TAG, "-----------------------------------------------------------------------------");
DiffUtil.DiffResult diffResult = DiffUtil.calculateDiff(new AnswersDiffCallback(newAnswers, oldAnswers), true);
diffResult.dispatchUpdatesTo(new ListUpdateCallback() {
    @Override
    public void onInserted(int position, int count) {
        try {
            Log.d(TAG, String.format("onInserted: (position, count) = (%d, %d)", position, count));
            for (int i = position; i < position + count; i++) {
                Log.d(TAG, "onInserted: newAnswer.text = " + newAnswers.get(i).getText());
            }
        } catch (Exception ex) {
            Log.e(TAG, "onInserted: Exception", ex);
        }
        Log.d(TAG, "-----------------------------------------------------------------------------");
    }

    @Override
    public void onRemoved(int position, int count) {
        try {
            Log.d(TAG, "onRemoved: (position, count) = (" + position + ", " + count + ")");
            for (int i = position; i < position + count; i++) {
                Log.d(TAG, String.format("onRemoved: (oldAnswer.id, oldAnswer.text) = (%s, %s)", oldAnswers.get(i).getId(), oldAnswers.get(i).getText()));
            }
        } catch (Exception ex) {
            Log.e(TAG, "onRemoved: Exception", ex);
        }
        Log.d(TAG, "-----------------------------------------------------------------------------");
    }

    @Override
    public void onMoved(int fromPosition, int toPosition) {
        try {
            Log.d(TAG, "onMoved: (fromPosition, toPosition) = (" + fromPosition + ", " + toPosition + ")");
            Log.d(TAG, String.format("onMoved: (oldAnswer.id, oldAnswer.text) = (%s, %s)", oldAnswers.get(fromPosition).getId(), oldAnswers.get(fromPosition).getText()));
            Log.d(TAG, String.format("onMoved: (newAnswer.id, newAnswer.text) = (%s, %s)", newAnswers.get(toPosition).getId(), newAnswers.get(toPosition).getText()));
        } catch (Exception ex) {
            Log.e(TAG, "onMoved: Exception", ex);
        }
        Log.d(TAG, "-----------------------------------------------------------------------------");
    }

    @Override
    public void onChanged(int position, int count, @Nullable Object payload) {
        try {
            Log.d(TAG, "onChanged: (position, count) = (" + position + ", " + count + ")");
            for (int i = position; i < position + count; i++) {
                Log.d(TAG, String.format("onChanged: (oldAnswer.id, oldAnswer.text) = (%s, %s)", oldAnswers.get(i).getId(), oldAnswers.get(i).getText()));
                Log.d(TAG, String.format("onChanged: (newAnswer.id, newAnswer.text) = (%s, %s)", newAnswers.get(i).getId(), newAnswers.get(i).getText()));
            }
        } catch (Exception ex) {
            Log.e(TAG, "onChanged: Exception", ex);
        }
        Log.d(TAG, "-----------------------------------------------------------------------------");
    }
});

and here is the logcat with the exception I have got:

2019-06-19 16:32:00.461 24515-24515/com.example.myApp D/AdminQuestionsFragment: answers: oldAnswers = [{"count":0,"id":"5d09a1236969e249cca42e96","questionId":"5d09a1236969e249cca42e95","text":"old answer 0"},{"count":0,"id":"5d09a1236969e249cca42e97","questionId":"5d09a1236969e249cca42e95","text":"old answer 1"}]
2019-06-19 16:32:00.501 24515-24515/com.example.myApp D/AdminQuestionsFragment: answers: newAnswers = [{"count":0,"id":"5d09a1236969e249cca42e97","questionId":"5d09a1236969e249cca42e95","text":"old answer 1"},{"text":"new answer 0"},{"text":"new answer 1"},{"text":"new answer 2"}]
2019-06-19 16:32:00.501 24515-24515/com.example.myApp D/AdminQuestionsFragment: -----------------------------------------------------------------------------
2019-06-19 16:35:19.550 24515-24515/com.example.myApp D/AdminQuestionsFragment: onInserted: (position, count) = (2, 3)
2019-06-19 16:35:19.550 24515-24515/com.example.myApp D/AdminQuestionsFragment: onInserted: newAnswer.text = new answer 1
2019-06-19 16:35:19.550 24515-24515/com.example.myApp D/AdminQuestionsFragment: onInserted: newAnswer.text = new answer 2
2019-06-19 16:35:19.599 24515-24515/com.example.myApp E/AdminQuestionsFragment: onInserted: Exception
    java.lang.IndexOutOfBoundsException: Index: 4, Size: 4
        at java.util.ArrayList.get(ArrayList.java:437)
        at com.example.myApp.views.AdminQuestionsFragment$5.onInserted(AdminQuestionsFragment.java:333)
        at androidx.recyclerview.widget.BatchingListUpdateCallback.dispatchLastEvent(BatchingListUpdateCallback.java:61)
        at androidx.recyclerview.widget.BatchingListUpdateCallback.onRemoved(BatchingListUpdateCallback.java:96)
        at androidx.recyclerview.widget.DiffUtil$DiffResult.dispatchRemovals(DiffUtil.java:921)
        at androidx.recyclerview.widget.DiffUtil$DiffResult.dispatchUpdatesTo(DiffUtil.java:836)
        at com.example.myApp.views.AdminQuestionsFragment.lambda$onActivityResult$8$AdminQuestionsFragment(AdminQuestionsFragment.java:320)
        at com.example.myApp.views.-$$Lambda$AdminQuestionsFragment$KZmQo8gdnjCYX1JsaACEVkjSd1s.onChanged(Unknown Source:8)
        at androidx.lifecycle.LiveData.considerNotify(LiveData.java:113)
        at androidx.lifecycle.LiveData.dispatchingValue(LiveData.java:126)
        at androidx.lifecycle.LiveData$ObserverWrapper.activeStateChanged(LiveData.java:424)
        at androidx.lifecycle.LiveData$LifecycleBoundObserver.onStateChanged(LiveData.java:376)
        at androidx.lifecycle.LifecycleRegistry$ObserverWithState.dispatchEvent(LifecycleRegistry.java:361)
        at androidx.lifecycle.LifecycleRegistry.addObserver(LifecycleRegistry.java:188)
        at androidx.lifecycle.LiveData.observe(LiveData.java:185)
        at com.example.myApp.views.AdminQuestionsFragment.onActivityResult(AdminQuestionsFragment.java:387)
        at androidx.fragment.app.FragmentActivity.onActivityResult(FragmentActivity.java:170)
        at android.app.Activity.dispatchActivityResult(Activity.java:7454)
        at android.app.ActivityThread.deliverResults(ActivityThread.java:4353)
        at android.app.ActivityThread.handleSendResult(ActivityThread.java:4402)
        at android.app.servertransaction.ActivityResultItem.execute(ActivityResultItem.java:49)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:108)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:68)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1808)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:193)
        at android.app.ActivityThread.main(ActivityThread.java:6669)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
2019-06-19 16:35:19.599 24515-24515/com.example.myApp D/AdminQuestionsFragment: -----------------------------------------------------------------------------
2019-06-19 16:35:19.600 24515-24515/com.example.myApp D/AdminQuestionsFragment: onRemoved: (position, count) = (0, 1)
2019-06-19 16:35:19.603 24515-24515/com.example.myApp D/AdminQuestionsFragment: onRemoved: (oldAnswer.id, oldAnswer.text) = (5d09a1236969e249cca42e96, old answer 0)
2019-06-19 16:35:19.603 24515-24515/com.example.myApp D/AdminQuestionsFragment: -----------------------------------------------------------------------------

the problem is with this line:

2019-06-19 16:35:19.550 24515-24515/com.example.myApp D/AdminQuestionsFragment: onInserted: (position, count) = (2, 3)

why the position is not 1?

Update 1 here is the logcat if I have old list of 2 items and I removed the item at index 1 then added 3 items to the end of the list:

2019-06-19 18:25:24.368 28118-28118/com.example.myApp D/AdminQuestionsFragment: answers: oldAnswers = [{"count":0,"id":"5d09a1236969e249cca42e96","questionId":"5d09a1236969e249cca42e95","text":"old answer 0"},{"count":0,"id":"5d09a1236969e249cca42e97","questionId":"5d09a1236969e249cca42e95","text":"old answer 1"}]
2019-06-19 18:25:24.370 28118-28118/com.example.myApp D/AdminQuestionsFragment: answers: newAnswers = [{"count":0,"id":"5d09a1236969e249cca42e96","questionId":"5d09a1236969e249cca42e95","text":"old answer 0"},{"text":"new answer 0"},{"text":"new answer 1"},{"text":"new answer 2"}]
2019-06-19 18:25:24.370 28118-28118/com.example.myApp D/AdminQuestionsFragment: -----------------------------------------------------------------------------
2019-06-19 18:25:24.370 28118-28118/com.example.myApp D/AdminQuestionsFragment: onRemoved: (position, count) = (1, 1)
2019-06-19 18:25:24.371 28118-28118/com.example.myApp D/AdminQuestionsFragment: onRemoved: (oldAnswer.id, oldAnswer.text) = (5d09a1236969e249cca42e97, old answer 1)
2019-06-19 18:25:24.371 28118-28118/com.example.myApp D/AdminQuestionsFragment: -----------------------------------------------------------------------------
2019-06-19 18:25:24.372 28118-28118/com.example.myApp D/AdminQuestionsFragment: onInserted: (position, count) = (1, 3)
2019-06-19 18:25:24.372 28118-28118/com.example.myApp D/AdminQuestionsFragment: onInserted: newAnswer.text = new answer 0
2019-06-19 18:25:24.372 28118-28118/com.example.myApp D/AdminQuestionsFragment: onInserted: newAnswer.text = new answer 1
2019-06-19 18:25:24.372 28118-28118/com.example.myApp D/AdminQuestionsFragment: onInserted: newAnswer.text = new answer 2
2019-06-19 18:25:24.372 28118-28118/com.example.myApp D/AdminQuestionsFragment: -----------------------------------------------------------------------------

Answer 1

It seems to be a known issue for ListUpdateCallback. It is not possible within onInserted for the inserted oldList item to identify its related position in the newList.

https://issuetracker.google.com/issues/115701827

My solution for this is: Insert a null-dummy list item in the oldList, and after diffResult.dispatchUpdatesTo is done, replace the null-dummies with the newList items at same positions.

Example, I have left some debugging code and logs to identify the problem:

    public static final class ListUpdate<T extends BaseIdentifier> implements ListUpdateCallback {

    private final List<T> oldList;
    private final List<T> newList;

    public ListUpdate(@NonNull List<T> oldList, @NonNull List<T> newList) {
        //logger.trace("ListUpdate" + System.lineSeparator()  + "old={}" + System.lineSeparator() + "new={}", BaseIdentifier.toString(oldList), BaseIdentifier.toString(newList));

        this.oldList = oldList;
        this.newList = newList;
    }

    public int inserts = 0;

    public boolean hasInserts() {
        return inserts > 0;
    }

    public void finishInserts() {
        if (inserts <= 0) {
            return;
        }

        //logger.trace("finishInserts inserts={}", inserts);

        ListIterator<T> oldListIterator = oldList.listIterator();
        ListIterator<T> newListIterator = newList.listIterator();

        while (inserts > 0 && oldListIterator.hasNext() && newListIterator.hasNext()) {
            T oldItem = oldListIterator.next();
            T newItem = newListIterator.next();

            if (oldItem == null) {
                //Replaces the last element returned by next()
                oldListIterator.set((T) newItem.copy());
                inserts--;
            }
        }

        if (inserts > 0 || oldList.contains(null)) {
            //There must be something wrong
            logger.error("finishInserts inserts={} remaining", inserts);
        }
    }

    /** {@inheritDoc} */
    @Override
    public void onInserted(int position, int count) {
        //logger.trace("onInserted position={} count={}", position, count);

        for (int i = 0; i < count; i++) {
            /*
            T item = newList.get(position + i);
            oldList.add(position + i, (T) item.copy());
             */
            //We don't know the related position of the newList, so we add null
            oldList.add(position + i, null);
            inserts++;
        }
    }

    /** {@inheritDoc} */
    @Override
    public void onRemoved(int position, int count) {
        //logger.trace("onRemoved position={} count={}", position, count);

        for (int i = 0; i < count; i++) {
            oldList.remove(position);
        }
    }

    /** {@inheritDoc} */
    @Override
    public void onMoved(int fromPosition, int toPosition) {
        //logger.trace("onMoved fromPosition={} toPosition={}", fromPosition, toPosition);

        T item = oldList.remove(fromPosition);
        oldList.add(toPosition, item);
    }

    /** {@inheritDoc} */
    @Override
    public void onChanged(int position, int count, Object payload) {
        logger.trace("onChanged position={} count={}", position, count);

        for (int i = 0; i < count; i++) {
            T item = newList.get(position + i);
            //noinspection unchecked
            oldList.set(position + i, (T) item.copy());
        }
    }
}

Call like

        DiffUtil.DiffResult diffResult = DiffUtil.calculateDiff(new BaseIdentifier.BaseIdentifierDiffUtilCallback(oldProcessEvents, processEvents));
        BaseIdentifier.ListUpdate<ProcessEvent> updater = new BaseIdentifier.ListUpdate<>(oldProcessEvents, processEvents);
        diffResult.dispatchUpdatesTo(updater);
        updater.finishInserts();

Answer 2

for (int i = position; i < position + count; i++) {
    Log.d(TAG, "onInserted: newAnswer.text = " + newAnswers.get(i).getText());
}

This loop will definitely throw IndexOutOfBoundError. count is number of items that have been added. and position is the position of the new item. When you do position + count that would be definitely more than the size of your list at some point. And hence the error.

The problem that if the old list has size of 2 items; and I removed the item at index 0, then added 3 items to the end of the list

Do you call dispatchUpdates after removing and before adding the new items?

I think first it's calculating items that are inserted and then item that is removed and hence the position is 2 in onInserted.

You can wrap your ListUpdateCallback in BatchingListUpdateCallback if you want batched updates.