Insert prefix to a variable if not empty in shell

Question

I want just one statement to do something like:

pip3 install $([ -n "$PYPI_MIRROR"] && echo "--index-url $PYPI_MIRROR") -r requirements.txt

It means it will append --index-url <url> arguments only if the $PYPI_MIRROR variable is set.

Then I found that

pip3 install ${PYPI_MIRROR/#/--index-url } -r requirements.txt

It works with sh and bash but zsh, zsh will always got --index-url whether $PYPI_MIRROR is set or not.

And there are some other problems with the 2nd solution, such as PYPI_MIRROR is set but empty.

Is there a way to do this more reliable?