0
Traceback (most recent call last):
  File "filter_quotes.py", line 451, in <module>
    stocks_filter.symbol_ids();
  File "filter_quotes.py", line 216, in symbol_ids
    self.symbol_ids_list = p.map(parallel_request, self.batch_result)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'FilterStocks.symbol_ids.<locals>.parallel_request'

I am struggling with the following class method. I coded the function parallel_request() in such a way I could use it with the Pool() class. However, I got the above traceback and I don't know how to fix it. How could I use parallel_request() function inside the class method symbol_ids() so that it will fix that previous AttributeError?

from multiprocessing import  Pool
def symbol_ids(self):
    p = Pool()
    def parallel_request(self, symbols):
        response = requests.get(''.join((self.uri, symbols)), headers=self.headers)
        return response.json()

    self.symbol_ids_list = p.map(parallel_request, self.batch_result)

    self.symbol_ids_result = [element['symbolId'] for element in self.symbol_ids_list]

UPDATE

The above function originated from the bottom test code.

I used the following class to request information from the Questrade API (http://www.questrade.com/api/documentation/rest-operations/market-calls/markets-quotes-id). I have over 11,000 stock symbols where I request the Questrade API with batches of 100 symbols. 

from multiprocessing import Pool
import requests
import logging
import ast
import re

TOKEN_FILE = '../../token.txt'

class Test: 

    def __init__(self, access_token, api_server):
        self.access_token = access_token
        self.api_server = api_server

        self.uri = '{}v1/symbols?names='.format(self.api_server)
        self.headers = {'Authorization': 'Bearer {}'.format(self.access_token)}

    def parallel_request(self, symbols):
        print(symbols)
        response = requests.get(''.join((self.uri, symbols)), headers=self.headers)
        return response.json()

    def process(self):
        with open('Output.txt', 'r') as f:
            batch_result = ast.literal_eval(f.read())

        # symbol_ids_list = []
        # for symbols in batch_result:
            # response = requests.get(''.join((self.uri, symbols)), headers=headers)
            # symbol_ids_list.extend(response.json().get('symbols'))

        p = Pool()
        test = p.map(self.parallel_request, batch_result)   

if __name__ == '__main__':
    with open(TOKEN_FILE, "r") as token_json:
        token_json = ast.literal_eval(re.search('{.+}', token_json.read()).group(0))
        access_token = token_json["access_token"]
        api_server = token_json["api_server"].replace("\\", "")

    obj = Test(access_token, api_server)
    obj.process()
Jeremie
  • 405
  • 1
  • 7
  • 20
  • Possible duplicate of [I can "pickle local objects" if I use a derived class?](https://stackoverflow.com/questions/36994839/i-can-pickle-local-objects-if-i-use-a-derived-class) – rammelmueller Mar 18 '18 at 17:56
  • you're defining a function inside of a function, which is then not top level and therefore not pickle-able. – rammelmueller Mar 18 '18 at 17:58
  • Even with the link other question you post, I am not even sure how to fix that problem. Are you up to make a full answer @rammelmueller? – Jeremie Mar 18 '18 at 18:33
  • let me try - could you post the full problem? I'm somehow unsure what you're doing. – rammelmueller Mar 18 '18 at 18:37
  • @rammelmueller Yes, of course, I will modify my question. – Jeremie Mar 18 '18 at 19:08
  • I'm sorry, I still don't have a working test case here. First of all: is this the exact piece of code, that produces the error? (I don't quite see the connection to the above snippet..) and second: what does the token.txt file look like? – rammelmueller Mar 18 '18 at 19:27
  • @rammelmueller I can't show you the content of token.txt. The exact piece of code make 500 lines, so I doubt it is a good idea to put it in the question ;). I was inspired by the test case code to build the little class method `symbol_ids()` with `parallel_request()` function inside. How could I easily fix that problem? – Jeremie Mar 18 '18 at 19:42
  • As far as I know you can not define `parallel_request()` within 'symbol_ids()` if you want to use it with pickle. Try to define it outside of the function, I believe this will fix your issue (sorry, hard to come up with more specific help without a working example..) – rammelmueller Mar 18 '18 at 20:05

0 Answers0