Writing webscrapped data into CSV

Question

With the below code I am able to scrape product infromation from two websites. My goal is to write the scraped data into a CSV where column A is used for the class "label" and column B is used for the class "value"

Can anyone help me achieve the desired outcome?

from bs4 import BeautifulSoup
import requests
import pandas as pd

url_list = ["https://21shares.com/product/abtc", "https://21shares.com/product/aeth/"]


for link in url_list:
    r = requests.get(link)
    r.encoding = 'uft-8'
    html_content = r.text
    soup = BeautifulSoup(html_content, "lxml")
    datas = soup.find('div', {'class':'product-sidebar-container'})

    for data in datas:
        soup.findAll("span", {"class": "Label", "Value": True})
        print(data.getText(separator=("\n")))

Does this answer your question? [How do I read and write CSV files with Python?](https://stackoverflow.com/questions/41585078/how-do-i-read-and-write-csv-files-with-python) — esqew, Feb 16 '22 at 15:11
@esqew thanks for the reference, I am pretty new to coding therefore I do not quite understand how I need to write the code for pandas in order to export it to a CSV in the way I would like to have it. Could you perhaps show me how it is done with the data I have scraped via my code? — Lighto, Feb 16 '22 at 15:33
There are plenty of answers in the linked duplicate that illustrate how to write Python data structures to a CSV format without the added complexity of something like Pandas - can you elaborate as to what *specifically* about the answers in the linked duplicate are not clear to you? Unfortunately Stack Overflow isn't a place you can have your code written *for* you - if you don't know where to begin or have any idea on where to start in your attempt, you might benefit from taking a step back and seeking out more foundational materials to sharpen your fundamental Python skills before continuing — esqew, Feb 16 '22 at 15:38

chitown88 · Accepted Answer · 2022-02-16T20:29:55.173

Initialize an empty list and as you iterate through datas, create a dictionary to append to the list. Once you have that list, it's easy to convert to a dataframe and write to csv with pandas. Follow the example below which is your code modified.

from bs4 import BeautifulSoup
import requests
import pandas as pd

url_list = ["https://21shares.com/product/abtc", "https://21shares.com/product/aeth/"]

rows = []
for link in url_list:
    r = requests.get(link)
    r.encoding = 'uft-8'
    html_content = r.text
    soup = BeautifulSoup(html_content, "lxml")
    datas = soup.find('div', {'class':'product-sidebar-container'})

    for data in datas:
        for each in soup.findAll("span", {"class": "label"}):
            label = each.text
            value = each.find_next('span', {'class':'value'}).text
            
            row = {'label':label, 'value':value}
            rows.append(row)


df = pd.DataFrame(rows)
df.to_csv('file.csv', index=False)

score -2 · Answer 2 · answered Feb 16 '22 at 15:22

-2

You have a typo, should be 'utf-8'.

answered Feb 16 '22 at 15:22

Mossy82

115
1
9

1

Please provide more details. The question already contains `uft-8` – William Baker Morrison Feb 16 '22 at 16:48
Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – HPringles Feb 16 '22 at 23:00

Writing webscrapped data into CSV

2 Answers2