As I see, you've already checked this article:
https://www.blackcj.com/blog/2016/03/30/building-a-custom-android-keyboard
It explains how to add a custom layout for the popup keyboard.
The only customization it allows is changing keys background and text color.
Also, you could use a KeyboardView subclass and then override its methods to achieve you goals, like onTouchEvent() to capture motion events or onDraw() to repaint keyboard regions:
<org.home.CustomPopupKeyboard
  android:id="@android:id/keyboardView"
  ...
/>
Also, check the Hacker's Keyboard source:
https://github.com/klausw/hackerskeyboard
Seems, it uses the LatinKeyboardBaseView extends View class for the popup keyboard, in the layout/keyboard_popup.xml.
EDIT:
According to the Hacker's Keyboard code, the only way I found to close the popup when a key is released is creating your own KeyboardView class, which extends the View directly, and then changing its onTouchEvent().
The Android's native KeyboardView class has PopupWindow mPopupKeyboard property which is private so you can't subclass it and call mPopupKeyboard.dismiss() to hide the popup.
The chain that goes from releasing a key to the popup closing is:
- LatinKeyboardBaseView::onTouchEvent()>- case MotionEvent.ACTION_UP: onUpEvent()>- tracker.onUpEvent()
- PointerTracker::onUpEvent()>- detectAndSendKey()>- listener.onCancel()
- LatinKeyboardBaseView::onCancel()(implementation of the- OnKeyboardActionListenerinterface) >- dismissPopupKeyboard()>- mMiniKeyboardPopup.dismiss()