how to change date and month?

Question

this is a piece of code where I want to change the date and month in the link. I don't want to use the date-time module because I will be scraping and storing some information from the website for each day and for each month I will be saving the data in CSV so after every month I want to create a new CSV file also if there would be a for loop for a month I want to use it in naming the CSV file while saving it

        from selenium import webdriver
        import time
        path="C:\\Users\\Nihal\\Downloads\\chromedriver_win32\\chromedriver.exe"
        import numpy 
        driver = webdriver.Chrome(path)
        driver.get('https://www.wunderground.com/history/daily/in/mangalore/VOML/date/2013-4-14')

without using `datetime` module how do you plan to change the date? — deadshot, Sep 07 '20 at 15:54

score 0 · Answer 1 · answered Sep 07 '20 at 15:56

Plug the dates as tuples and use f-string to format them in URL:

from selenium import webdriver
import time
path="C:\\Users\\Nihal\\Downloads\\chromedriver_win32\\chromedriver.exe"
import numpy
driver = webdriver.Chrome(path)
dates = [(2013, 4, 14)]
for x in dates:    
    #https://www.wunderground.com/history/daily/in/mangalore/VOML/date/2013-4-14
    driver.get(f'https://www.wunderground.com/history/daily/in/mangalore/VOML/date/{x[0]}-{x[1]}-{x[2]}')

score 0 · Answer 2 · answered Sep 07 '20 at 16:22

I am not used to selenium, so I can't help you with the parsing that you do with it, but I am pretty sure that you can use the datetime module to iterate on every day of a month and on every month of the year. Here is an example of how you can iterate on this and generate the url that you need:

from datetime import timedelta, date

def daterange(start_date, end_date):
    for n in range(int((end_date - start_date).days)):
        yield start_date + timedelta(n)

start_date = date(2020, 8, 1)
end_date = date(2020, 8, 31)
days_to_scrape = []

for single_date in daterange(start_date, end_date):
    days_to_scrape.append(f'https://www.wunderground.com/history/daily/in/mangalore/VOML/date/{single_date.strftime("%Y-%m-%d")}')

#Iteration with driver.get

If I understand, you will have no choice but to send as many requests as the number of days from which you want the data. You can then iterate on the items of the list with your scraping command. If there is another reason why you think datetime module can't do what you need it to do, please, explain it.

L.R.

P.S. Thanks to vinzee who helped me to understand that kind of iteration with his answer: Iterating through a range of dates in Python

how to change date and month?

2 Answers2