0

I am able to connect to a certain URL with cURL, after I installed the corresponding SSL certificates:

$ export MY_URL=https://www.infosubvenciones.es/bdnstrans/GE/es/convocatoria/616783
$ curl -vvvv $MY_URL  # Fails
$ sudo openssl x509 -inform pem -outform pem -in /tmp/custom-cert.pem -out /usr/local/share/ca-certificates/custom-cert.crt
$ sudo update-ca-certificates
$ curl -vvvv $MY_URL  # OK

However, requests (or httpx, or any other library I use) refuses to do so:

In [1]: import os
   ...: import requests
   ...: requests.get(os.environ["MY_URL"])
---------------------------------------------------------------------------
SSLCertVerificationError                  Traceback (most recent call last)
...

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

My understanding is that requests uses certifi and as such these custom certificates are not available here:

In [1]: import certifi

In [2]: certifi.where()
Out[2]: '/tmp/test_ca/.venv/lib/python3.10/site-packages/certifi/cacert.pem'

I have already tried a number of things, like trying to use the system CA bundle:

  • export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt same error
  • requests.get(..., verify="/etc/ssl/certs/ca-certificates.crt") same error
  • switched to httpx + a custom SSL context as explained in the docs, same error
  • attempted truststore as discussed in this httpx issue, same error

How can I make Python (requests, httpx, raw ssl, anything) use the same certificates that cURL is successfully using?

The only thing that worked so far, inspired by this hackish SO answer, is to do verify=False. But I don't want to do that.

In [9]: requests.get(
   ...:     my_url,
   ...:     verify=False,
   ...: )
/tmp/test_ca/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
astrojuanlu
  • 6,744
  • 8
  • 45
  • 105
  • Can you share with us both the website to which you're connecting and the CA bundle you're trying to use? – larsks Mar 30 '22 at 11:26
  • It's not a super secret website or bundle, but I'd prefer not to disclose it straight away because it's a bit sensitive, unless it's strictly necessary. It is my understanding that the problem is not related to a specific website, although I also acknowledge that having such website would help with the debugging. If there are no answers in a few days (and the question hasn't been closed), I'll add more info. – astrojuanlu Mar 30 '22 at 12:24

1 Answers1

3

i tried your stuff on my system (Manjaro Linux, python 3.10) i can make a connection. I downloaded the complete certificate chain from the website (with my browser). After that i can use it with:

r = requests.get(url=URL, verify=<path to pem file>)

and with

export REQUESTS_CA_BUNDLE=<path to pem>

r = requests.get(url=URL)

I tried the export within pyCharm.

So the python stuff is working and you may have a problem in your certificates. Without this stuff i get the ssl error (of course), because python does not use the system certs as you mentioned correct. In my pem-file i have 3 certificates. Maybe you have only 1 and the others are in the global store, so that curl does not need the complete chain, instead of python. You should try to download the complete chain with your browser and try again.

D-E-N
  • 1,242
  • 7
  • 14