0

I'm trying to thread as described in this post, and also pass multiple arguments in Python 2.7 through a work-around described here.

Right now I have something like this, a function that is part of class pair_scraper:

def pool_threading(self):
        pool = ThreadPool(4)
        for username in self.username_list:
            master_list = pool.map(self.length_scraper2,
                itertools.izip(username*len(self.repo_list),
                itertools.repeat(self.repo_list)))

def length_scraper2(self, username, repo):
    #code

However, when I run my code I get the error:

TypeError: length_scraper2() takes exactly 3 arguments (2 given)

Which seems to be because it wants self passed as an argument, which is nonsensical given I'm using a class function within the class. Thoughts on how to fix?

Community
  • 1
  • 1
zthomas.nc
  • 3,689
  • 8
  • 35
  • 49

1 Answers1

1

itertools.izip(username*len(self.repo_list),itertools.repeat(self.repo_list)) yields a tuple.

You need to pass 2 arguments explicitly to your method (self is implicitly passed because it's a non-static method), but you only pass 1 tuple explicitly, plus the implicit self which makes 2 arguments, hence the confusing error message.

You have to use * to pass your tuple as 2 separate arguments, like this:

master_list = pool.map(self.length_scraper2,
     *itertools.izip(username*len(self.repo_list),itertools.repeat(self.repo_list)))

simple test using the classical map on a simple function:

def function(b,c):
    return (b,c)

print(list(map(function,zip([1,2],[4,5]))))

error:

  print(list(map(function,zip([1,2],[4,5]))))
 TypeError: function() missing 1 required positional argument: 'c'

now adding single asterisk to expand args:

print(list(map(function,*zip([1,2],[4,5]))))

works:

[(1, 2), (4, 5)]

same goes for class method:

class Foo:
    def function(self,b,c):
        return (b,c)

f = Foo()

print(list(map(f.function,*zip([1,2],[4,5]))))
Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219