How to use different adapters with multiple view types inside the same ConcatAdapter?

Question

I have a ConcatAdapter that uses one PagingDataAdapter (wrapped by its own ConcatAdapter by calling withLoadStateFooter and a normal ListAdapter):

val concatAdapter = ConcatAdapter(
   newChatMessagesAdapter, // this is a ListAdapter
   chatMessagesHistoryAdapter.withLoadStateFooter(
       ListLoadStateAdapter(chatMessagesHistoryAdapter::retry)
   ) // this is another ConcatAdapter (that contains a PagingDataAdapter)
)

Both the newChatMessageAdapter and the chatMessageHistoryAdapter have multiple view types, but both adapters have the exact same ones:

// newChatMessagesAdapter and chatMessagesHistoryAdapter both contain this code:

override fun onBindViewHolder(holder: RecyclerView.ViewHolder, position: Int) {
        val message = getItem(position)
        if (message != null) {
            when (holder.itemViewType) {
                R.layout.item_chat_message_outgoing_first -> {
                    (holder as ChatMessageOutgoingFirstViewHolder).bind(message)
                }
                R.layout.item_chat_message_outgoing_following -> {
                    (holder as ChatMessageOutgoingFollowingViewHolder).bind(message)
                }
                R.layout.item_chat_message_incoming_first -> {
                    (holder as ChatMessageIncomingFirstViewHolder).bind(message, isGroupChat)
                }
                R.layout.item_chat_message_incoming_following -> {
                    (holder as ChatMessageIncomingFollowingViewHolder).bind(message)
                }
            }
        }
    }

    override fun getItemViewType(position: Int): Int {
        val currentItem = getItem(position)
        val previousItem = if (position != itemCount - 1) getItem(position + 1) else null

        return if (currentItem?.senderUid == currentUserUid) {
            if (previousItem == null || previousItem.senderUid != currentItem.senderUid) {
                R.layout.item_chat_message_outgoing_first
            } else {
                R.layout.item_chat_message_outgoing_following
            }
        } else {
            if (previousItem == null || previousItem.senderUid != currentItem?.senderUid) {
                R.layout.item_chat_message_incoming_first
            } else {
                R.layout.item_chat_message_incoming_following
            }
        }
    }

When I start my app, all views are empty, because no itemViewType matches in onBindViewHolder. It works if I remove withLoadStateFooter (which turns the 2nd ConcatAdapter into my own PagingDataAdapter) and add ConcatAdapter.Config.Builder().setIsolateViewTypes(false).build(), probably because both adapters have the exact same viewTypes. However, as soon as I add withLoadStateFooter this stops working as well and I am back at empty view holders.

So in a nutshell, my question is: How can I use different viewTypes in my adapters that are combined inside a ConcatAdapter?