I need to monitor 1000+ specific links for about one month to see if their content has changed, and I wonder if I can automate this somehow. One idea was to simply download these websites now and again in one month, and compare the source files. If I go down this route, do you guys know of a tool (browser extension?) that would make such a download easy? I've tried HTTrack, but it fails after the first 100 links or so. Alternatively, a (free?) web service which can monitor a set of websites might work as well. I've used https://visualping.io/ before, but it's not really intended for thousands of links.
Asked
Active
Viewed 105 times
1 Answers
1
I wonder if I can automate this somehow.
Hardly necessary but yes you could write some simple scripts.
do you guys know of a tool ... that would make such a download easy?
wget, curl, etc
You could put the 1000 specific URLs into a text file, create two directories, cd into the first directory and use a tool such as wget with the -i option to read the list of URLs and fetch them. A month later repeat this in the second directory, use diff e.g. diff -r /directory1 /directory2 to find any changes.
Be careful about using recursive options, they can overwhelm the server and get you banned or can overload your computer.
I'd try with a small set of URLs first (e.g. 2, then 10, then 1000)
A lower-cost option may be to use HTTP HEAD requests and trust that the server knows if a resource has been changed.
RedGrittyBrick
- 85,717