Proxies in Python Requests

Question

So I used Proxy Broker to scrape some proxies. Sometimes the proxies are dead when scraped so I wanted to check them before I used them. So I wrote a program using Python Requests to check them. Here it is:

import time
import random
import requests
lines = open('not_checked.txt').read().splitlines()
check =random.choice(lines)
yaya = {
  check
}
for x in range(0 , 10):
    requests.get('https://reg.ebay.com/reg/PartialReg?ru=https%3A%2F%2Fwww.ebay.com%2F': proxies=yaya)
    r.status_code
    print(status_code)

    if status_code == 200:
        f=open("checked_proxies.txt", "a+")
        f.write(proxies)
    else:
       time.sleep(.001)

However this throws back the "set object has no attribute get". I looked the error up online and the it said it is because I used a comma instead of a colon. So then I tried:

requests.get('https://reg.ebay.com/reg/PartialReg?ru=https%3A%2F%2Fwww.ebay.com%2F': proxies=yaya)

to get a syntax error. What is going on?

score 3 · Answer 1 · answered Sep 08 '18 at 01:45

3

proxies needs to be a dict. It's right there in the docs:

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

Your yaya is a set not dict.

answered Sep 08 '18 at 01:45

John Zwinck

239,568
38
324
436

You do not have to hard-code the proxies. But you do need to create a `dict` where the keys are protocols. – John Zwinck Sep 08 '18 at 01:46
Got it. Let me try will get back to you. Thanks for the help – Kunwar Sodhi Sep 08 '18 at 01:47

score 2 · Answer 2 · answered Sep 08 '18 at 01:58

There will definitely be a comma instead of semicolon. Sample snippet

import time
import random
import requests
lines = open('proxies.txt').read().splitlines()
# check =random.choice(lines)
proxies = [
    {
      "http": "XXX.XXX.XXX.XXX:XXXX",
      "https": "XXX.XXX.XXX.XXX:XXXX",
    },
    {
      "http": "XXX.XXX.XXX.XXX:XXXX",
      "https": "XXX.XXX.XXX.XXX:XXXX",
    },
    {
      "http": "XXX.XXX.XXX.XXX:XXXX",
      "https": "XXX.XXX.XXX.XXX:XXXX",
    }
]
for proxy in proxies:
    print("Requesting with %s and %s"%(proxy['http'], proxy['https']))
    r = requests.get('https://reg.ebay.com/reg/PartialReg?ru=https%3A%2F%2Fwww.ebay.com%2F', proxies=proxy)
    print("Loaded")
    r.status_code
    print(r.status_code)
    if r.status_code == 200:
        f=open("checked_proxies.txt", "a+")
        f.write(proxy)
    else:
        time.sleep(.001)

Proxies in Python Requests

2 Answers2