The flags you mention all set properties for the MSI, which you can then reference in your custom action, converting to whatever format you need for your setup.exe.
For instance:
/i corresponds to the value of Installed being false (if the product isn't already installed).
/r corresponds to a value of "Repair" on the WixUIInstallMode property.
/x corresponds to a value of "Remove" on the WixUIInstallMode property.
/q (and its modifiers like /qn and /qb) correspond to the UILEVEL property, which has the following values found from this link.
UILevel:
INSTALLUILEVEL_NONE 2 Completely silent installation.
INSTALLUILEVEL_BASIC 3 Simple progress and error handling.
INSTALLUILEVEL_REDUCED 4 Authored UI, wizard dialogs suppressed.
INSTALLUILEVEL_FULL 5 Authored UI with wizards, progress, errors.
You can see this answer if you're unfamiliar with passing values and referencing them in a custom action.