How to extract json from script tag using beautiful soup?

Question

I want to extract the reviewCount from the script tag using beautiful soup. Tried different approach but didn't succeed.

<script type="application/json" data-initial-state="review-filter">
{"languages":[{"isoCode":"all","displayName":"Toutes les langues","reviewCount":"573"},{"isoCode":"fr","displayName":"français","reviewCount":"567"},{"isoCode":"en","displayName":"English","reviewCount":"6"}],"selectedLanguages":["all"],"selectedStars":null,"selectedLocationId":null}
</script>

_Tried different approach but didn't succeed._ Can you share those attempts? From the tag you shared, at seems that all you need to do is get the contents of the tag and parse the result. If you're struggling with extracting the content from the element, this is a duplicate of [Extract content within a tag with BeautifulSoup](https://stackoverflow.com/questions/5999407/extract-content-within-a-tag-with-beautifulsoup). If the issue is parsing the JSON, this is a duplicate of [How to parse JSON in Python?](https://stackoverflow.com/questions/7771011/how-to-parse-json-in-python). — AMC, Apr 14 '20 at 22:19

James Powis · Answer 1 · 2020-04-14T21:53:12.210

8

This should work, I am absolutely sure there is a more elegant approach:

import json
from bs4 import BeautifulSoup

html = '''
<script type="application/json" data-initial-state="review-filter">
{"languages":[{"isoCode":"all","displayName":"Toutes les langues","reviewCount":"573"},{"isoCode":"fr","displayName":"français","reviewCount":"567"},{"isoCode":"en","displayName":"English","reviewCount":"6"}],"selectedLanguages":["all"],"selectedStars":null,"selectedLocationId":null}
</script>
'''

soup = BeautifulSoup(html, 'html.parser')
res = soup.find('script')
json_object = json.loads(res.contents[0])

for language in json_object['languages']:
    print('{}: {}'.format(language['displayName'], language['reviewCount']))

output:

Toutes les langues: 573
français: 567
English: 6

edited Apr 14 '20 at 21:53

answered Apr 14 '20 at 21:39

James Powis

609
4
16

Thanks James. I tried the method you mentioned above. My main problem is to get the reviewCount number. – free_123 Apr 14 '20 at 21:44
TypeError: object of type 'Response' has no len() – GGEv Feb 06 '21 at 16:58
`json.loads(res.text)` worked too for me – AK91 Jan 25 '22 at 15:15

score 3 · Answer 2 · answered Apr 14 '20 at 21:52

Import json and load data into json and then iterarte to get all the reviewCount.

import json
html='''<script type="application/json" data-initial-state="review-filter">
{"languages":[{"isoCode":"all","displayName":"Toutes les langues","reviewCount":"573"},{"isoCode":"fr","displayName":"français","reviewCount":"567"},{"isoCode":"en","displayName":"English","reviewCount":"6"}],"selectedLanguages":["all"],"selectedStars":null,"selectedLocationId":null}
</script>'''

soup=BeautifulSoup(html,"html.parser")
item=soup.select_one('script[data-initial-state="review-filter"]').text
jsondata=json.loads(item)
for item in jsondata['languages']:
    print(item['reviewCount'])

Output:

573
567
6

score 2 · Answer 3 · answered Apr 14 '20 at 23:34

import re

html = '''<script type="application/json" data-initial-state="review-filter">
{"languages":[{"isoCode":"all","displayName":"Toutes les langues","reviewCount":"573"},{"isoCode":"fr","displayName":"français","reviewCount":"567"},{"isoCode":"en","displayName":"English","reviewCount":"6"}],"selectedLanguages":["all"],"selectedStars":null,"selectedLocationId":null}
</script>'''


match = [item.group(1) for item in re.finditer('reviewCount":"(.+?)"', html)]

print(match)

Output:

['573', '567', '6']

How to extract json from script tag using beautiful soup?

3 Answers3

Linked

Related