0

I have just started to learn how to use regular expressions to extract data from websites. The first goal of mine is to extract the title of a website. Here is what my code is like:

<?php 
    $data = file_get_contents('http://bctia.org');
    $regex = '/<title>(.+?)<\/title>/';
    preg_match($regex,$data,$match);
    var_dump($match); 
?>

The result of var_dump is empty:

array(0) { }

At first I thought, "maybe bctia.org does not have a title"? However, this is not the case, as I have checked the source of bctia.org, and it does have content between <title> and </title>.

Then I thought, maybe my code does not work? However, this is not the case either, as I have substituted bctia.org with other websites, say, bing.com, or apple.com, and they both returned correct results. For example, with apple.com I get the correct result

array(2) { [0]=> string(20) "" [1]=> string(5) "Apple" }

So I have to come to the conclusion that bctia.org is a very special website that prevents me from extracting its title...

I am wondering if that is actually the case? Or maybe my code has some problems that I have not identified?

Thank you in advance!

Bing
  • 21
  • 2
  • In this specific case, lack of `/s` modifier. – mario Jun 12 '13 at 00:16
  • 3
    I'll leave this here: http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags – styfle Jun 12 '13 at 00:18
  • Hi @mario, thanks for the suggestion. Unfortunately adding /s to the expression still does not work for bctia.org.. – Bing Jun 12 '13 at 00:34
  • @Bing I just tested your code with someone's example below, and it could be the fact that `` might be written in uppercase `<title>`, as was the case for me. My site has title in uppercase, which is why it didn't work in the first place. I asked myself the question and came up with that. Another possible reason could be that they're running some fancy JS. Look into other possible factors. Another reason is `redirection`, which is another possible factor, as this was something happening on my site. – Funk Forty Niner Jun 12 '13 at 00:48
  • @Bing To add, I tried your code the way it is, and replaced `` with uppercase letters, and it worked fine for me. Try and incorporate a function in there (`strtoupper`) that will take your lowercase `title` and render it in uppercase. I.e. `$regex = strtoupper($regex);` – Funk Forty Niner Jun 12 '13 at 00:50
  • @Fred Thank you for the comment. However, I double checked the source of bctia.org and the are in lowercase. Weird... – Bing Jun 12 '13 at 00:52
  • @Bing You're welcome. I do have another finding and upon viewing their source, found that the `...` was not setup in one line, but in `three lines`. When testing it on my side to do the same thing, it did not find the title. However when I put everything back onto one line, it found it. So, that may very well be the problem. Your regex doesn't take into account to find something in multiple lines. – Funk Forty Niner Jun 12 '13 at 01:01
  • @Bing To prove my theory, set it up on your server with 2 files. One with `This is the title of my page` on one line, then another test with `` on one line, then the actual page's title on the 2nd, then `` on a 3rd line with your existing/original code, and you'll see that I'm right. *P.s. Apple.com's title is on one line.* – Funk Forty Niner Jun 12 '13 at 01:07

3 Answers3

3

This specific website's server-side code assumes that the client sends a User-Agent header, and apparently, your PHP installation is not configured to send one. So a 500 Internal Server Error is returned, causing file_get_contents to return false.

Source Error:
Line 66: //LOAD: Compatibility Mode
Line 67: //<meta http-equiv="X-UA-Compatible" content="IE=7,IE=9" />
Line 68: string BrowserOS = Request.ServerVariables["HTTP_USER_AGENT"].ToString();
Line 69: HtmlMeta compMode = new HtmlMeta();
Line 70: compMode.Content = "IE=7,IE=9";


Source File: c:\inetpub\wwwroot\BCTIA\Website\bctia\layouts\Main Layout.aspx.cs   
Line: 68

Stack Trace:
[NullReferenceException: Object reference not set to an instance of an object.]
   Layouts.Main_Layout.Page_Load(Object sender, EventArgs e) in c:\inetpub\wwwroot\BCTIA\Website\bctia\layouts\Main Layout.aspx.cs:68
   System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +24
   System.Web.UI.Control.LoadRecursive() +70
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3063

To work around this issue, you can just set a user-agent string before making the request:

ini_set('user_agent', 'Mozilla/5.0 (compatible; Examplebot/0.1; +http://www.example.com/bot.html)');
PleaseStand
  • 31,641
  • 6
  • 68
  • 95
  • @Bing True, yet you'll have to re-configure your regex in regards to the `` problem. I used his workaround and it did find the `lang` as you suggested earlier, but not the `<title>`. In order for it to work, you'll have to use `$regex = '/<title>(.+?)<\/title>/s';`. It won't matter if they're on seperate lines. That worked, I just tried it. `$regex = '/<title>([^<]+)<\/title>/';` will also work. – Funk Forty Niner Jun 12 '13 at 05:06
0

Do not use regular expression .. !!

Instead Use xpath take a look at: xpath

regular expression will not work well.

Neta Meta
  • 4,001
  • 9
  • 42
  • 67
0

Parsing html code with regex isn't a good way because you may be surprised by his permissive structure.

The reason why your pattern don't work is that the dot don't match newlines.

If you want that the dot matches newlines use the s modifier at the end of the pattern, or don't use the dot:

$regex = '/<title>(.+?)<\/title>/s';

or 

$regex = '/<title>([^<]+)<\/title>/';

[^<] is a character class that contains all characters but <, as you can see with that you don't need to use a lazy quantifier: + instead of +?

Casimir et Hippolyte
  • 88,009
  • 5
  • 94
  • 125
  • Hi Casimir, thank you for your answer. However, unfortunately I was not able to get the correct result of extract bctia.org using either of the pattern you suggested :< Could we explore other possibilities together? – Bing Jun 12 '13 at 00:33
  • @Bing: the PleaseStand answer solve your first problem, after, using one of the two patten will solve your second. – Casimir et Hippolyte Jun 12 '13 at 04:40