python-boilerpipe hangs with multiprocessing

Question

I am trying to run boilerpipe with Python multiprocessing. Doing this to parse RSS feeds from multiple sources. The problem is it hangs in one of the threads after processing some links. The whole flow works if I remove the pool and run it in a loop.

Here is my multiprocessing code:

proc_pool = Pool(processes=4)
for each_link in data:
    proc_pool.apply_async(process_link_for_feeds, args=(each_link, ), callback=store_results_to_db)
proc_pool.close()
proc_pool.join()

This is my boilerpipe code which is being called inside process_link_for_feeds():

def parse_using_bp(in_url):
    extracted_html = ""
    if ContentParser.url_skip_p.match(in_url):
        return extracted_html
    try:
        extractor = Extractor(extractor='ArticleExtractor', url=in_url)
        extracted_html = extractor.getHTML()
        del extractor
    except BaseException as e:
        print "Something's wrong at Boilerpipe -->", in_url, "-->", e
        extracted_html = ""
    finally:
        return extracted_html

I am clueless on why it is hanging. Is there something wrong in the proc_pool code?

Can you provide more details. What is `ContentParser`, what is `Extractor`? - I tried to mimic your problem but in my case it works. — Jon, Dec 15 '13 at 14:03
Looking into it I found the source [here](https://github.com/misja/python-boilerpipe/blob/master/src/boilerpipe/extract/__init__.py). All the code is just thrown into __init__.py (edit - actually ContentParser doesn't seem to be in there) — KobeJohn, Dec 15 '13 at 14:14

score 1 · Accepted Answer · edited May 23 '17 at 11:56

1

Can you try threading instead? Multiprocessing is basically for when you are CPU bound. Also, boilerpipe already includes protection when using threading which suggests that it may need protection in multiprocessing also.

If you really need mp, I will try to figure out how to patch boilerpipe.

Here is what I guess will be a drop-in replacement using threading. It uses multiprocessing.pool.ThreadPool (which is a "fake" multiprocessing pool). The only change is from Pool(..) to multiprocessing.pool.ThreadPool(...) The problem is that I'm not sure the boilerpipe multithreading test will detect the thread pool () as having activeCount() > 1.

import multiprocessing
from multiprocessing.pool import ThreadPool  # hidden ThreadPool class

# ...
proc_pool = ThreadPool(processes=4)  # this is the only difference
for each_link in data:
    proc_pool.apply_async(process_link_for_feeds, args=(each_link, ), callback=store_results_to_db)
proc_pool.close()
proc_pool.join()

edited May 23 '17 at 11:56

Community

1
1

answered Dec 15 '13 at 14:34

KobeJohn

7,390
6
41
62

@dpatro, even if you actually need multiprocessing, could you try this? If this works, then it might be straightforward to patch boilerpipe for multiprocessing protection. – KobeJohn Dec 15 '13 at 18:40
I also thought of same (github code). Moved to Java a week back to make boilerpipe run. Will check it out today and update the thread. – dpatro Dec 16 '13 at 07:42
Checked and it is working for continuously over a day now. The processing is slowed down a bit though, may be due to the library. Thanks @kobejohn for the insight to the solution. – dpatro Dec 17 '13 at 06:11
@dpatro that's great news. Thanks for letting me/everyone know it works. If multiprocessing worked faster, then it may be boilerpipe or the other parts of your code that are faster. If you can remove the other parts of the code and demonstrate that boilerpipe is significantly faster with multiprocessing, that would be a great way to submit a feature request on github. – KobeJohn Dec 17 '13 at 06:33
But the problem is with multiprocessing, boilerpipe hangs. My problem in first place! – dpatro Dec 17 '13 at 12:15
@dpatro Indeed. Complete failure of logic on my part there! Let me restate that. If the speed is important to you AND you are convinced it would work faster with multiprocessing, you could post another question on SO (or to the ypthon boilerpipe maintainer) specifically asking how to patch that piece of boilerpipe to work with multiprocessing. Sorry for the silly comment. – KobeJohn Dec 17 '13 at 13:21
sure. I will do that once I get sometime after my work hours :) – dpatro Dec 18 '13 at 02:14
1

I got a chance to check things properly. It turns out that ThreadPool is faster than ProcessPool. My previous remark on speed was a mistake as I forgot to check the internet speed that day. – dpatro Dec 29 '13 at 15:44

python-boilerpipe hangs with multiprocessing

1 Answers1