I was struggling because of this for a very long time. Answer from adneal didn't get me right colors, most were just white. The trick I did is load icons, convert them to very low resolution bitmaps and then get color that is the most not-greyish (to reduce a chance of picking white/grey/black colors from an icon) out of them.
Here is the code:
fun getBestPrimaryColor(icon:Drawable): Int{
    val bitmap: Bitmap = icon.toBitmap(5, 5, Bitmap.Config.RGB_565)
    var best_diff_j = 0
    var best_diff_k = 0
    var best_diff = 0
    var current_diff = 0
    for(j in 0..4){
        for(k in 0..4){
            val c = bitmap.getPixel(j,k)
            current_diff = max(listOf(c.red,c.green,c.blue)) - min(listOf(c.red,c.green,c.blue))
            if(current_diff > best_diff){
                best_diff = current_diff
                best_diff_j = j
                best_diff_k = k
            }
        }
    }
    return bitmap.getPixel(best_diff_j, best_diff_k)
}
It turns out like this:

You can change size of the bitmap height and width to get more colors but I don't think it's really necessary.