Python's default SSL certificate context not working in requests method when behind proxy, works fine otherwise

Question

I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.

def reverse_lookup(lat, long):
    cafile=certifi.where()
    params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
            'accept-language' : 'en', 'addressdetails' : 1}
    response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
    #response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
    result = json.loads(response.text)
    return result['address']['country'], result['address']['state'], result['address']['city']

When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:

def reverse_lookup(lat, long):
    cafile=certifi.where()
    proxies = {"https" : "https://myproxy.com"}
    params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
            'accept-language' : 'en', 'addressdetails' : 1}
    response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
    result = json.loads(response.text)
    return result['address']['country'], result['address']['state'], result['address']['city']

But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():

CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER

Only time it works is when I completely bypass the SSL verification with verify=False

My questions are:

Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?

Any help is appreciated. Code tested in both Python 2.7.15 and 3.9

score 1 · Accepted Answer · answered Apr 10 '21 at 17:42

Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?

Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.

How to make the default context of SSL verification work in this case, when I'm behind proxy ?

The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.

But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER

It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).

Python's default SSL certificate context not working in requests method when behind proxy, works fine otherwise

1 Answers1