How can I extract information from an HTML file using Perl regular expressions?

Question

I have two files, XML and an HTML and need to extract data from these on certain patterns.

My XML file is pretty well formatted and I can use readline to read a line and search data between tags.

if($line =~ /\<tag1\>$varvalue\<\/tag1\>/)`

However, for my HTML, it has one of the worst code I have seen and the file is like:

<div class="theater">
    <h2>
    <a href="/showtimes/university-village-3" >**University Village 3**</a></h2>
    <div class="address">
        <i>**3323 South Hoover Street, Los Angeles CA 90007 | (213) 748-6321**</i>
    </div>
</div>

<div class="mtitle">
    <a href="/movie/dream-house-2011"  title="Dream House" onmouseover="mB(event, 771204354);"  >**Dream House**</a>
    <span>**(PG-13 , 1 hr. 31 min.)**</span>
</div>

<div class="times">

    **1:00 PM,**
</div>

Now from this file I need to pick data which is shown in bold.

I can use Perl regular expression to search data from this file.

Answer 1

RegEx match open tags except XHTML self-contained tags

http://www.codinghorror.com/blog/2009/11/parsing-html-the-cthulhu-way.html

Using regular expressions to parse HTML: why not?

When you are done reading those come back :)

Edit : and to actually solve your problem take a look at this module :

http://perlmeme.org/tutorials/html_parser.html

Some sample to parse the an html file :

#!/usr/local/bin/perl

use HTML::TreeBuilder;

$tree = HTML::TreeBuilder->new;
$tree->parse_file('C:\Users\Stefanos\workspace\HTML_Parser_Test\test.html');

@divs = $tree->find('div');

$tree->delete;

In this example I just used your tags as the main body of an .html file. The divs are stored in the @divs array. Since I have no idea which text you want to find, because ** is not a element I can't help you further..

P.S. I have never used this module but I just did it in 5 minutes so it is not so hard to parse the html file and find whatever you want..

Regex to match any specific tag and store of contents result into $1:

if ($subject =~ m!<tagname[^>]*>(.*?)</tagname>!s) {
    # Successful match
}

Although you will soon realize the limitations of this approach when you have nested elements..

Replace tagname with actual tag.. e.g. in your case i, a, span, div although for div you will also get the contents of the first div which is not what you want..

Answer 2

Parsing XML and HTML using regular expressions is a fool's errand. There are many simple to use Perl modules for parsing HTML. Here is something using HTML::TokeParser::Simple. I've omitted the code to associate movies and showtimes with theaters (because I have no intention of building an appropriate input file):

#!/usr/bin/env perl

use strict; use warnings;
use HTML::TokeParser::Simple;

my $parser = HTML::TokeParser::Simple->new(handle => \*DATA);

my @theaters;

while (my $div = $parser->get_tag('div')) {
    my $class = $div->get_attr('class');
    next unless defined($class) and $class eq 'theater';

    my %record;

    $record{theater} = $parser->get_text('/a');
    $record{address} = $parser->get_text('/i');

    s{(?:^\s+)|(?:\s+\z)}{} for values %record;

    push @theaters, \%record;
}

use YAML;
print Dump \@theaters;

__DATA__
<div class="theater">
    <h2>
    <a href="/showtimes/university-village-3" >**University Village 3**</a></h2>
    <div class="address">
        <i>**3323 South Hoover Street, Los Angeles CA 90007 | (213) 748-6321**</i>
    </div>
</div>

<div class="mtitle">
    <a href="/movie/dream-house-2011"  title="Dream House" onmouseover="mB(event, 771204354);"  >**Dream House**</a>
    <span>**(PG-13 , 1 hr. 31 min.)**</span>
</div>

<div class="times">

    **1:00 PM,**
</div>

<div class="theater">
    <h2>
    <a href="/showtimes/university-village-3" >**Some other theater*</a></h2>
    <div class="address">
        <i>**1234 South Hoover Street, St Paul, MN 99999 | (999) 748-6321**</i>
    </div>
</div>

Output:

[sinan@macardy]:~/tmp> ./tt.pl
---
- address: '**3323 South Hoover Street, Los Angeles CA 90007 | (213) 748-6321**'
  theater: '**University Village 3**'
- address: '**1234 South Hoover Street, St Paul, MN 99999 | (999) 748-6321**'
  theater: '**Some other theater*'