This answer might be outdated. I do not have time to investigate right now. I recommend checking the briefcase resources for more up-to-date information. For example this section of the doc might be relevant: https://briefcase.readthedocs.io/en/latest/reference/configuration.html#pep621-compatibility
As far as I can tell, briefcase isn't actually PEP 517 compatible (at least not by default). It uses a pyproject.toml file, but doesn't fill up the [build-system] section, so it should be possible to set an actual PEP 517 build backend in that file without causing conflict.
pyproject.toml
[build-system]
build-backend = 'setuptools.build_meta'
requires = [
    'setuptools',
    'toml',
]
[tool.briefcase.app.exampleapp]
formal_name = 'exampleapp'
description = 'something'
requires = ['PyQt5', 'qtconsole']
sources = ['exampleapp']
setup.py
#!/usr/bin/env python3
import pathlib
import pkg_resources
import setuptools
import toml
def _parse_briefcase_toml(pyproject_path, app_name):
    pyproject_text = pyproject_path.read_text()
    pyproject_data = toml.loads(pyproject_text)
    briefcase_data = pyproject_data['tool']['briefcase']
    app_data = briefcase_data['app'][app_name]
    setup_data = {
        'name': pkg_resources.safe_name(app_data['formal_name']),
        'version': briefcase_data['version'],
        'install_requires': app_data['requires'],
        # ...
    }
    return setup_data
def _setup():
    app_name = 'exampleapp'
    pyproject_path = pathlib.Path('pyproject.toml')
    setup_data = _parse_briefcase_toml(pyproject_path, app_name)
    setuptools.setup(**setup_data)
if __name__ == '__main__':
    _setup()
Then pip and other PEP 517-compatible frontends should be able to build and install the project by delegating to setuptools while taking care to correctly setup a build environment containing both setuptools and toml.
I guess it would be also possible to let briefcase handle the parsing of the pyproject.toml file (maybe with briefcase.config.parse_config(...)) but it's not documented so I don't know how stable these APIs are.