Python: Run all files by date in order

Question

I have an existing automation that picks up "today's" file from a network drive, loads them locally on C drive, then loads the data from that files.

Each file has the same file name, but with the date added at the end. For example, FileName_20220315. How can I edit my existing automation to run every file, one after the other or one at a time between a past date to today or the last one?

For example...

Runs FileName_20220301.
Once completed, picks up the next file, FileName_20220302.
And continues until the last file.

    import time, glob, os, pysftp, datetime    
    from shutil import copyfile    
    from Utils.Utilities import Utilities    
    import win32com.client as win32    
    from datetime import date    
    from email.mime.multipart import MIMEMultipart    
    from email.mime.application import MIMEApplication    
    from email.mime.text import MIMEText    
    
    global Status, log_file_name, user_file_name, team_file_name    
    from email.mime.multipart import MIMEMultipart    
    from email.mime.text import MIMEText    
    from email.mime.base import MIMEBase    
    import mimetypes, smtplib    

    Status = ""    
    log_file_name = ""    
    today = date.today().strftime("%Y%m%d")    
    #today=20211122    
    now = datetime.datetime.now()    
    today_date_time = ('%s_%s_%s_%s_%s' % (now.month, now.day, now.year, now.hour, now.minute))    
    import getpass
    
    user = getpass.getuser().lower()    
    count = 30    
    Email_count = 0    
    file_name = "File_Name_%s.csv" % (today)

get string `20220301` from filename, convert to `datetime` and later you can add `timedelta(days=1)` to get next day. — furas, Mar 16 '22 at 01:12
or get all filenames, sort them, find position of starting filename and get sublist `list[start_pos:]` and use it with `for`-loop — furas, Mar 16 '22 at 01:16

score 0 · Answer 1 · answered Mar 16 '22 at 01:16

You just need to make a range of dates.

You can use something like the following:

numdays = 100
base = datetime.datetime.today()
date_list = [base - datetime.timedelta(days=x) for x in range(numdays)]

Then just check if the date from the filename is in "date_list".

I took this code from the following answer: Creating a range of dates in Python

furas · Answer 2 · 2022-03-16T01:30:06.157

If all filenames have the same name FileName with date Year Month Day then:

you can get all filenames

filenames = os.listdir()

you should sort them to make sure they are in correct order

filenames = sorted(filenames)

you can find starting filename on this list

pos = filenames.find("FileName_20220315")

and you can use this position to get only needed filenames

selected_filenames = filenames[ pos: ]

And later you can use for-loop to run your function with this list

for name in selected_filenames:
    your_function(name)

But will not work if you use Day Month Year

score 0 · Answer 3 · answered Mar 16 '22 at 01:35

You need to filter and sort the file names. Load the filenames into a list, then use a for loop with an argument such as (element > A and element < B), either to place the desired range of names into a new list or to delete the undesired range from the existing list.

After that use mylist.sort() if you need them in alphabetical order, then you can loop over mylist to do whatever it is you want. Because you have a strict filename format the alphabetical order should also be date order.

If you do not need the names in order then you can use a generator, or generator comprehension, to yield filenames and avoid the initial creation of a complete list. The generator can feed filenames one at a time directly into a for loop to test for date range without the need for a huge storage buffer to hold a list. (eg If you have millions of file names.)

Python: Run all files by date in order

3 Answers3