Questions tagged [sitemap]
5 questions
6
votes
6 answers
3
votes
5 answers
Website crawler/spider to get site map
I need to retrieve a whole website map, in a format like :
http://example.org/
http://example.org/product/
http://example.org/service/
http://example.org/about/
http://example.org/product/viewproduct/
I need it to be linked-based (no file or dir…
ack__
- 117
2
votes
2 answers
Unrestricting a sub directory in robots.txt
OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is there any way to allow this one directory, other than…
xxl3ww
- 1,509
1
vote
2 answers
How can I create a Site Map for a Password Protected Site
Microsoft Visio has a feature where it will create a site map for an existing website. However, my website has a login and users must be authenticated to view the content.
Is there any way to enter credentials into Visio or provide Visio with an…
Bob
- 11
- 1
- 2
0
votes
1 answer
Download/update webpages listed in XML sitemap
I'm searching a FLOSS tool that downloads all pages (and embedded resources, e.g. images) linked in a XML sitemap (built according to http://www.sitemaps.org/).
The tool should "crawl" the sitemap regularly and look for new and deleted URLs and…
unor
- 3,196