python beautifulsoup how to get data as json format?

Question

I want to get data as json format. Right now I am getting data as dictionary which little bit messy for me. Here is my code:

my_dict = {"job_title":[],"time_posted":[],"number_of_proposal":[],"page_link":[]};
for page_num in range(1, 12):
    time.sleep(3)
    url = (
        f'my_url').format(page_num)
    print(url)
    headers = requests.utils.default_headers()
    print(headers)
    headers.update(
        {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0', })
    print(headers)
    r = requests.get(url, headers=headers).text
    soup = BeautifulSoup(r, 'lxml')

    box = soup.select('.item__top_container⤍ListItem⤚3pRrO')
    for i in box:
        job_title = i.select('.item__title⤍ListItem⤚2FRMT')[0].text.lower()
        job_title = job_title.replace('opportunity', ' opportunity').replace(
            'urgent', ' urgent').strip()
        print(job_title)
        time_posted = i.select('time')[0].text.lower()
        remove_month_year = ["month", "year"]
        print(time_posted)
        proposal = i.select(
            '.item__info⤍ListItem⤚1ci50 li:nth-child(3)')[0].text.replace('Proposals', '').strip()
        keywords = ['scrap', 'data mining']
        if(any(key_words in job_title for key_words in keywords)):
            if(not any(remove_m_y in time_posted for remove_m_y in remove_month_year)):
                   my_dict["job_title"].append(job_title)
                   my_dict["time_posted"].append(time_posted)
                   my_dict["number_of_proposal"].append(proposal)
                   my_dict["page_link"].append(url)

my dictionary data look like this:

{'job_title': ['web scraping of product reviews', 'yell web scraping in python', 'google business scraping',],'time_posted': ['6 days ago', '9 days ago', '3 days ago'], 'page_link': ['url1','url2','url3']}

My expected result will be look like this:

{"job_title":"web scraping of product reviews","time_posted":"6 days ago","page_link":"url1"},{"job_title":"yell web scraping in python","time_posted":"9 days ago","page_link":"url2"}

Freddy Mcloughlan I don't know why it's invalid JSON. I just need the data as my expected result so it could be json or whatever — Farhan, May 12 '22 at 11:46
Does this help? https://stackoverflow.com/a/32824345/14141223 Use `json.dumps(dictionary)` — Freddy Mcloughlan, May 12 '22 at 11:47
what exactly are you trying to do? Do you want to print your dictionary as a JSON string? or the invalid (for JSON at least) formatted string in your example? — Freddy Mcloughlan, May 12 '22 at 11:51
This has nothing to do with JSON. You just want a different structure. — gre_gor, May 12 '22 at 11:54
Then instead of putting values into 4 different list, put them in a single dict and add that dict into a list. — gre_gor, May 12 '22 at 11:57

score 1 · Accepted Answer · answered May 12 '22 at 11:58

You can change the structure with the following code:

my_list = []

for i in range(len(my_dict["job_title"])):
    my_list.append({
        "job_title": my_dict["job_title"][i],
        "time_posted": my_dict["time_posted"][i],
        "number_of_proposal": my_dict["number_of_proposal"][i],
        "page_link": my_dict["page_link"][i]
    })

Even better would be to directly create the list in the first loop like you need it in the end.

my_list = []
for i in box:
    job_title = i.select('.item__title⤍ListItem⤚2FRMT')[0].text.lower()
    job_title = job_title.replace('opportunity', ' opportunity').replace(
        'urgent', ' urgent').strip()
    print(job_title)
    time_posted = i.select('time')[0].text.lower()
    remove_month_year = ["month", "year"]
    print(time_posted)
    proposal = i.select(
        '.item__info⤍ListItem⤚1ci50 li:nth-child(3)')[0].text.replace('Proposals', '').strip()
    keywords = ['scrap', 'data mining']
    if(any(key_words in job_title for key_words in keywords)):
        if(not any(remove_m_y in time_posted for remove_m_y in remove_month_year)):
            my_list.append({
                "job_title": job_title,
                "time_posted": time_posted,
                "number_of_proposal": number_of_proposal,
                "page_link": page_link
            })

@Farhan, you can't just ask for upvotes. You receive upvotes when your question is clear, well researched and useful. Please edit your question to be more so, if you would like upvotes — Freddy Mcloughlan, May 12 '22 at 12:17

score 0 · Answer 2 · answered May 12 '22 at 12:10

I think you are defining your data structure wrong. From your expected result I understand that you want: {"job_title": "title 1", "time_posted":"6 days ago" ... }, {"job_title": "title2"...}

So, a list of dictionaries. Now you have a dictionary with list type values.

You have two options:

1.- Process your dictionary to adquire your desired structure

final_list = []

for _ in range(len(my_dict["job_title"])):
    item_dict = {}
    for key in my_dict:
        item_dict[key] = my_dict[key].pop(0)
    final_list.append(item_dict)

print(final_list) 
# [{'job_title': 'web scraping of product reviews', 'time_posted': '6 days ago', 'page_link': 'url1'}, {'job_title': 'yell web scraping in python', 'time_posted': '9 days ago', 'page_link': 'url2'}, {'job_title': 'google business scraping', 'time_posted': '3 days ago', 'page_link': 'url3'}]

2.- Same as user jugi mentioned, It's the best option. He has answered while I was writing this, so I'll post this anyways because my option 1 is slightly different.

Freddy Mcloughlan · Answer 3 · 2022-05-12T12:08:15.140

-1

You can create a dictionary for each entry with a comprehension:

# Just using x because it's shorter. This does not create a copy
x = my_dict
x = [{'job_title': x['job_title'][i], 'time_posted': x['time_posted'][i],
      'page_link': x['page_link'][i]} for i in range(len(x['page_link']))]

>>> x
[{'job_title': 'web scraping of product reviews',
  'page_link': 'url1',
  'time_posted': '6 days ago'},
 {'job_title': 'yell web scraping in python',
  'page_link': 'url2',
  'time_posted': '9 days ago'},
 {'job_title': 'google business scraping',
  'page_link': 'url3',
  'time_posted': '3 days ago'}]

edited May 12 '22 at 12:08

answered May 12 '22 at 11:50

Freddy Mcloughlan

4,129
1
13
29

did you read my question? if not then see what will be my expected result. It's not just converting dic to json. – Farhan May 12 '22 at 11:54

python beautifulsoup how to get data as json format?

3 Answers3