Stack Overflow
Stack Overflow

Questions tagged [link-checking]

Use this tags for general questions about checking hyperlinks.

This tag is used for general questions about how to check hyperlinks. It can concern checking the validity of a hyperlink (syntax check) or testing if an URI is available.

There is some overlap with the [web-crawler] tag, but [link-checking] focuses primarily on whether a link points to a resource that is accessible.

17 questions
12
votes
5 answers

What is the best way to check if a URL exists in PHP?

What is the best way to see a URL exists and the response is not a 404 ?
user529649
3
votes
1 answer

Which HTTP status codes should we consider for dead links

We are executing below curl call from PHP. $url = $fullurl; if (isset($url)) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch , CURLOPT_RETURNTRANSFER, 1); $data =…
Hetal Sagar
  • 91
  • 5
3
votes
1 answer

Check link works and if not visually identify it as broken

I am working on a project which lists file sharing urls from the likes of Oron, filespost, depositfiles etc that reports sharing of copyrighted materials to identified content owners and rights holders in my network. To better improve the service,…
Ben
  • 190
  • 7
  • 21
2
votes
1 answer

BeautifulSoup Broken Link Checker / web crawler

I am trying to build a broken link checker based on this How-to: https://dev.to/arvindmehairjan/build-a-web-crawler-to-check-for-broken-links-with-python-beautifulsoup-39mg However, I'm having trouble with the line of code, since when I run the…
K C
  • 21
  • 1
2
votes
3 answers

Grails - Link checking as part of a continuous integration

So, we have a grails app set up with a Hudson CI build process. We're running unit tests, integration tests, and about to set up Selenium for some functional tests as well. However, are there any good ways of fully testing a sites links to make sure…
Reverend Gonzo
  • 39,701
  • 6
  • 59
  • 77
2
votes
1 answer

Use of link-Checker (ruby)

Has anyone used the link-checker gem? I don't want to use it in a project I want to write a small script to test links on a web app. I cant seem to figure out how to use it. Trying to require it doesn't work but saying gem 'link-checker' does…
Zach
  • 885
  • 2
  • 8
  • 27
1
vote
1 answer

Travis CI with GitHub Pages / Linkchecker

I am hosting pages on GitHub Pages. These use Jekyll, but that shouldn't matter. Since we have source code management for the web pages and even a publishing and hosting... why not add CI! Is there a best practice for implementing a link checker…
William Entriken
  • 37,208
  • 23
  • 149
  • 195
1
vote
0 answers

How to do 404 link testing through selenium rc for complete website?

How can i verify a complete website's link(mostly links that are redirecting to 404 page) by using Selenium RC. Previously I tried to do this thong by using xenu and web link validator but in there results most of the links are showing 500 internal…
user1726460
  • 107
  • 3
  • 10
0
votes
1 answer

Daily check all webpages from a list of websites

A guest has a website A has a partner program. Partner sites B have links to A. I need to check with a certain frequency (twice a day) all webpages of all the partner sites (5000 sites) and extract all the links from B to A. Then I have to check…
nulll
  • 1,465
  • 1
  • 17
  • 28
0
votes
1 answer

Maybe I need to export paths but not sure

I'm trying to use the check-html flag with linkchecker. It gives an error that the tidy module is not installed. If I say tidy at a command line I can use it. I have MacPorts installed and have installed just about every option for libtidy or tidy…
keith
  • 1
0
votes
0 answers

link check gets status bad result

We have some resources which contains links to external sites. However, we want to avoid dead links and have implemented a ping routine written in c# .net6. We loop through all links and do a HEAD and a GET request with HttpClient. Most sites return…
Bjarke
  • 1,283
  • 11
  • 36
0
votes
0 answers

How to check if Facebook links to user/page content are still active?

I have a list of various Facebook links of videos, photos, statuses, etc. posted by users with public profiles or public pages. I want to check if the links are still active and the content is still available (i.e. video wasn't taken down or…
Jaycee
  • 35
  • 6
0
votes
2 answers

Check site for my site files

Is there a program that crawls a specified website and will spit out if there is a reference to another website? I have images,video files,pdf's,etc. that I need to give to another developer to finish the port over to their new server. I just…
Rich
  • 11
  • 1
0
votes
1 answer

Which URL is considered valid while checking for broken links

I have a bunch of URLs and I'm trying to check if these URLs are valid (check for errors like 404 etc.). In such a scenario, which URLs are considered valid? Example scenarios: The server responds with HTTP 3XX (Moved permanently/temporarily) Are…
techno
  • 6,100
  • 16
  • 86
  • 192
0
votes
3 answers

Check Entire Website's Links

I know similar questions have been asked, but I'm not sure about the answers (and I can't easily test all of them), so before I go crazy continuing to search, I want to ask: Is there an easy way to crawl all the pages on a website and check them for…
Tom A
  • 1,662
  • 2
  • 23
  • 41
1
2