Browsing Google Play Instant developer documentation a bit more, I found an explanation: 
Google Play Instant runs instant-enabled app bundles in a special kind of SELinux sandbox for added security. This sandbox permits a subset of permissions, as well as limited types of interactions with other apps.
And more specifically, under which circumstances other apps can be accessed:
Access to installed apps 
When developing an instant experience, keep in mind that it cannot interact with installed apps on a device unless one of the following is true:
- One or more activities within an installed app has set its
  android:visibleToInstantAppselement totrue– This element is
  available to apps running Android 8.0 (API level 26) or higher.
- An
  installed app contains an intent filter that includes
  CATEGORY_BROWSABLE.
- The instant experience is sending an intent using
  either the ACTION_SEND,ACTION_SENDTO, orACTION_SEND_MULTIPLEaction.
So that explains this and seems reasonable enough. 
I'll investigate if some of these works in my case and report back later...
Edit: yes, android:visibleToInstantApps works!
In my case, I was trying to launch another app of the same client company, meaning I have control over that codebase too. 
In AndroidManifest of the target app, add android:visibleToInstantApps="true" to one of the activities. This works on Android 8 and higher, and can be used even if the minSdkVersion is lower.