First, customize the overflow button style in styles.xml:
<!-- Application theme. -->
<style name="AppTheme" parent="Theme.MaterialComponents.Light">
    <!-- Your theme customization... -->
    <item name="android:actionOverflowButtonStyle">@style/MyActionOverflowButtonStyle</item>
</style>
<!-- Here you choose the ID of the overflow icon. -->
<item type="id" name="myActionOverflowButtonID"/>
<!-- Style to replace action bar overflow icon. -->
<style name="MyActionOverflowButtonStyle" parent="Widget.AppCompat.ActionButton.Overflow">
    <item name="android:id">@id/myActionOverflowButtonID</item>
</style>
(inspired by How to change three dots button on android to other button, but with android:id instead of android:src in the style)
Then, add a badge:
// Get the button view
val view = findViewById<View>(R.id.myActionOverflowButtonID)
// Create and customize a badge, e.g:
val badgeDrawable = BadgeDrawable.create(context).apply {
    backgroundColor = getColor(R.color.primary)
    horizontalOffset = 30
    verticalOffset = 30
    number = 2
}
// Attach the badge to the button
// For this API, you need to update Material components dependency to 1.3.0 or later
BadgeUtils.attachBadgeDrawable(badgeDrawable, view)