I need to find all text files in all subfolders created on a certain date, open them, and copy the contents into a single text file

Question

My goal is to take the contents of all text files in subfolders created today and move them to a single existing report.txt but I can't seem to find a good way to go about it. I'm not very experienced in coding so any help would be much appreciated. Here is what I have so far (I know it's rubbish):

if getmtime == today:
   with open(glob.iglob(drive + "://CADIQ//CADIQ_JOBS//?????????????????????")) as f:
      for line in f:
         content += line
   with open(reportFile, "a") as f:
      f.write(content)

score 1 · Answer 1 · answered Dec 22 '20 at 16:54

I would start by creating a desired_date object, which is a datetime.date. You can then format that date into a string, which makes up the pattern you want to look for in your glob. The glob pattern doesn't care about the time, just the date.

from pathlib import Path
import datetime

desired_date = datetime.date(year=2020, month=12, day=22)
pattern = "13.2.1_" + desired_date.strftime("%y_%m_%d") + "_*"

for path in Path("path/to/folders").glob(pattern):
    if not path.is_dir():
        continue
    print(path)

From there, you can visit each path, glob all text files in the current path, and accumulate the lines in each text file. Finally, write everything to one file.

score 1 · Answer 2 · answered Dec 22 '20 at 16:54

1

import glob

contents = b''
for file in glob.glob('./*/*.txt'): # u can change as per your directory
    fname = file.split(r'\\')[-1]
    with open(fname, 'rb') as f1:
        contents += f1.read()
with open('report.txt','wb') as rep:
    rep.write(contents)

Hope this helps so :) Better try to read or write files in terms of bytes because sometimes there may be a chance of corrupting data.

answered Dec 22 '20 at 16:54

Nanthakumar J J

860
1
7
22

What does the contents = b ' ' represent? – Dark Horse Dumpling Dec 23 '20 at 12:40
1

It just to define all contents will be in binary – Nanthakumar J J Dec 23 '20 at 13:15

smed · Accepted Answer · 2020-12-23T13:17:38.473

1

Try this, based on How do I list all files of a directory?

import os, time

def last_mod_today(path):
    '''
    return True if getmtime and time have year, mon, day coincinding in their localtime struct, False else
    '''
    t_s = time.localtime(os.path.getmtime(path))
    today = time.localtime(time.time())
    return t_s.tm_mday==today.tm_mday and t_s.tm_year == today.tm_year and t_s.tm_mon == today.tm_mon

name_to_path = lambda d,x:os.path.normpath(os.path.join(os.path.join(os.getcwd(), d),x))

def log_files(d):
    '''
    walking through the files in d
    log the content of f when last modif time for f is today
    WARNING : what happens when the file is a JPEG ?
    '''
    scand_dir = os.path.join(os.getcwd(), d)
    print(f"scanning {scand_dir}...")
    (_, _, filenames) = next(os.walk(scand_dir))
    log = open("log.txt", 'a')
    for f in filenames:
        if last_mod_today(name_to_path(d,f)):
            with open(name_to_path(d,f), 'r') as todays_file:
                log.write('##############################\n')
                log.write(f"file : {name_to_path(d,f)}\n")
                log.write(todays_file.read()) 
                log.write('\n')
                log.write('##############################\n')
    log.close()

#first scanning files in the current directory
(_, dirnames, _) = next(os.walk('./'))
log_files('./')
#then crawling through the subdirs (one level)
for d in dirnames:
    log_files(d)

edited Dec 23 '20 at 13:17

answered Dec 22 '20 at 17:21

smed

148
1
10

If you want me to elaborate on this to get a full answer, you're welcome. – smed Dec 22 '20 at 19:30
Trying this right now. I will let you know. – Dark Horse Dumpling Dec 22 '20 at 20:26
1

Have a look to the link above the code, since os.walk will also give you the directories through wich you want to loop. Let me know how this goes. You can do it ! – smed Dec 22 '20 at 20:38
Thank you man. I really appreciate it. So the first attempt returned the wrong files but it did return something so that is progress lol. – Dark Horse Dumpling Dec 22 '20 at 21:10
So currently it's targeting the directory the batch file is in but I need it to target all subdirectories within a different directory. – Dark Horse Dumpling Dec 22 '20 at 21:31
1

change the parameter of `os.walk()` – smed Dec 22 '20 at 21:38
Having a little trouble. I need to get all text files from inside the subfolders. I won't know the name of the subfolders because they change based on time created. What would I put in for a parameter of os.walk() to get that? – Dark Horse Dumpling Dec 22 '20 at 21:51
How many folders into folders into folders do you have ? (how deep shall we scan ? – smed Dec 23 '20 at 12:21
The text files are in the subfolders of the folder I have a the path for. FolderIKnow/SubfoldersIHaveToObtainRelativelyByDate/TextFiles – Dark Horse Dumpling Dec 23 '20 at 12:35
1

ok, i should have something that suits this – smed Dec 23 '20 at 13:12
1

I did some edits in the first answer above. Don't forget to mark it as accepted answer if this corresponds to your needs ! – smed Dec 23 '20 at 13:32
Checking this in just a moment. If it works I will absolutely mark it as accepted answer. Thank you so much for your help. You are awesome. – Dark Horse Dumpling Dec 23 '20 at 13:41
I didn't use your code to a T really but the content and knowledge that you shared definitely helped get me there so I'm marking your answer as the accepted answer. Thank you so much for your help! – Dark Horse Dumpling Dec 23 '20 at 14:43
My pleasure ! Good luck – smed Dec 23 '20 at 15:48

I need to find all text files in all subfolders created on a certain date, open them, and copy the contents into a single text file

3 Answers3