GAE Python URL Fetch throws InvalidURLError

Question

GAE Python URL Fetch throws InvalidURLError while the same URL works perfectly with Postman ( Google Chrome App ).

CODE

url = "https://abcdefgh:28dfd95928dfd95928dfd95928dfd95928dfd95928dfd959@twilix.exotel.in/v1/Accounts/abcdefgh/Sms/send" 
form_fields = {
  "From": "08039511111",
  "To": "+919844100000",
  "Body": "message for you"
} 
form_data = urllib.urlencode (form_fields) 

try: 
  result = urlfetch.fetch(url=url,
                        payload=form_data,
                        method=urlfetch.POST,
                        headers={'Content-Type': 'application/x-www-form-urlencoded' } 
  ) 
  logging.info ("result = ["+repr (result)+"] ") 
except Exception: 
  logging.error ("Exception. ["+traceback.format_exc ()+"] ")

OUTPUT LOGS

2016-01-21 15:48:23.368 +0530 E Exception. [ 
Traceback (most recent call last): File "main.py", line 27, in get method=urlfetch.POST, 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 271, in fetch return rpc.get_result() 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self) 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 389, in _get_fetch_result 'Invalid request URL: ' + url + error_detail) InvalidURLError: Invalid request URL: https://abcdefgh:28dfd95928dfd95928dfd95928dfd95928dfd95928dfd959@twilix.exotel.in/v1/Accounts/abcdefgh/Sms/send ]

For security purpose, I have replaced sensitive text in the URL with similar different characters.

score 0 · Answer 1 · edited May 23 '17 at 12:07

The code indicates an INVALID_URL RPC error code was received from the urlfetch service.

The most common occurence seems to be due to the URL length limit (check if your unedited URL hits that): Undocumented max length for urlfetch URL?

A long time ago it was also seen for very slow URLs (in Go land, but I suspect the urlfetch service itself is the same serving all language sandboxes) - unsure if this still stands, I also see a DEADLINE_EXCEEDED error code as well which might have been introduced specifically for such case in the meantime): Google App Engine Go HTTP request to a slow page

The failure might also be related to incorrect parsing of the rather unusual "host" portion of your URL foo:blah@hostname. Check if it you're getting the same error if dropping the foo:blah@ portion. If it's indeed the case you might want to file an issue with Google - the URL seems valid, works with curl as well.

Thanks Dan. I found the problem and the solution. We need to specify the HTTP auth info using headers. urlfetch.make_fetch_call ( rpc, url, method = urlfetch.POST, headers = { "Authorization" : "Basic %s" % base64.b64encode ( URL_USERNAME+":"+URL_PASSOWRD ) }, ) — gsinha, Jan 21 '16 at 14:21

score 0 · Accepted Answer · edited May 23 '17 at 12:23

I found the problem and the solution. We need to specify the HTTP auth info using headers.

        urlfetch.make_fetch_call (  rpc, 
                                    url, 
                                    method      = urlfetch.POST, 
                                    headers     = {     "Authorization"     : "Basic %s" % base64.b64encode ( URL_USERNAME+":"+URL_PASSOWRD ) }, 
        )

Courtesy
https://stackoverflow.com/a/8454580/1443563 by raugfer

GAE Python URL Fetch throws InvalidURLError

2 Answers2