Parse any number of subkeys in a perl hash

Question

I have a perl hash that is obtained from parsing JSON. The JSON could be anything a user defined API could generated. The goal is to obtain a date/time string and determine if that date/time is out of bounds according to a user defined threshold. The only issue I have is that perl seems a bit cumbersome when dealing with hash key/subkey iteration. How can I look through all the keys and determine if a key or subkey exists throughout the hash? I have read many threads throughout stackoverflow, but nothing that exactly meets my needs. I only started perl last week so I may be missing something... Let me know if that's the case.

Below is the "relevant" code/subs. For all code see: https://gitlab.com/Jedimaster0/check_http_freshness

use warnings;
use strict;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use DateTime;
use DateTime::Format::Strptime;

# Verify the content-type of the response is JSON
eval {
        assert_valid_json ($response->content);
};
if ( $@ ){
        print "[ERROR] Response isn't valid JSON. Please verify source data. \n$@";
        exit EXIT_UNKNOWN;
} else {
        # Convert the JSON data into a perl hashrefs
        $jsonDecoded = parse_json($response->content);
        if ($verbose){print "[SUCCESS] JSON FOUND -> ", $response->content , "\n";}

        if (defined $jsonDecoded->{$opts{K}}){
                if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $jsonDecoded->{$opts{K}}, "\n";}
                NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $jsonDecoded->{$opts{K}})));
        } else {
                print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
                exit EXIT_UNKNOWN;
        }
}




sub DATETIME_LOOKUP {
        my $dateFormat = $_[0];
        my $dateFromJSON = $_[1];

        my $strp = DateTime::Format::Strptime->new(
                pattern   => $dateFormat,
                time_zone => $opts{z},
                on_error  => sub { print "[ERROR] INVALID TIME FORMAT: $dateFormat OR TIME ZONE: $opts{z} \n$_[1] \n" ; HELP_MESSAGE(); exit EXIT_UNKNOWN; },
        );

        my $dt = $strp->parse_datetime($dateFromJSON);
        if (defined $dt){
                if ($verbose){print "[SUCCESS] Time formatted using -> $dateFormat\n", "[SUCCESS] JSON date converted -> $dt $opts{z}\n";}
                return $dt;
        } else {
                print "[ERROR] DATE VARIABLE IS NOT DEFINED. Pattern or timezone incorrect."; exit EXIT_UNKNOWN
        }
}




# Subtract JSON date/time from now and return delta
sub DATETIME_DIFFERENCE {
        my $dateInitial = $_[0];
        my $deltaDate;
        # Convert to UTC for standardization of computations and it's just easier to read when everything matches.
        $dateInitial->set_time_zone('UTC');

        $deltaDate = $dateNowUTC->delta_ms($dateInitial);
        if ($verbose){print "[SUCCESS] (NOW) $dateNowUTC UTC - (JSON DATE) $dateInitial ", $dateInitial->time_zone->short_name_for_datetime($dateInitial), " = ", $deltaDate->in_units($opts{u}), " $opts{u} \n";}

        return $deltaDate->in_units($opts{u});
}

Sample Data

{
  "localDate":"Wednesday 23rd November 2016 11:03:37 PM",
  "utcDate":"Wednesday 23rd November 2016 11:03:37 PM",
  "format":"l jS F Y h:i:s A",
  "returnType":"json",
  "timestamp":1479942217,
  "timezone":"UTC",
  "daylightSavingTime":false,
  "url":"http:\/\/www.convert-unix-time.com?t=1479942217",
  "subkey":{
    "altTimestamp":1479942217,
    "altSubkey":{
      "thirdTimestamp":1479942217
    }
  }
}

[SOLVED]

I have used the answer that @HåkonHægland provided. Here are the below code changes. Using the flatten module, I can use any input string that matches the JSON keys. I still have some work to do, but you can see the issue is resolved. Thanks @HåkonHægland.

use warnings;
use strict;
use Data::Dumper;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use Hash::Flatten qw(:all);
use DateTime;
use DateTime::Format::Strptime;

# Verify the content-type of the response is JSON
eval {
        assert_valid_json ($response->content);
};
if ( $@ ){
        print "[ERROR] Response isn't valid JSON. Please verify source data. \n$@";
        exit EXIT_UNKNOWN;
} else {
        # Convert the JSON data into a perl hashrefs
        my $jsonDecoded = parse_json($response->content);
        my $flatHash = flatten($jsonDecoded);

        if ($verbose){print "[SUCCESS] JSON FOUND -> ", Dumper($flatHash), "\n";}

        if (defined $flatHash->{$opts{K}}){
                if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $flatHash>{$opts{K}}, "\n";}
                NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $flatHash->{$opts{K}})));
        } else {
                print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
                exit EXIT_UNKNOWN;
        }
}

Example:

./check_http_freshness.pl -U http://bastion.mimir-tech.org/json.html -K result.creation_date -v
[SUCCESS] JSON FOUND -> $VAR1 = {
          'timestamp' => '20161122T200649',
          'result.data_version' => 'data_20161122T200649_data_news_topics',
          'result.source_version' => 'kg_release_20160509_r33',
          'result.seed_version' => 'seed_20161016',
          'success' => 1,
          'result.creation_date' => '20161122T200649',
          'result.data_id' => 'data_news_topics',
          'result.data_tgz_name' => 'data_news_topics_20161122T200649.tgz',
          'result.source_data_version' => 'seed_vtv: data_20161016T102932_seed_vtv',
          'result.data_digest' => '6b5bf1c2202d6f3983d62c275f689d51'
        };

Odd number of elements in anonymous hash at ./check_http_freshness.pl line 78, <DATA> line 1.
[SUCCESS] JSON KEY FOUND -> result.creation_date:
[SUCCESS] Time formatted using -> %Y%m%dT%H%M%S
[SUCCESS] JSON date converted -> 2016-11-22T20:06:49 UTC
[SUCCESS] (NOW) 2016-11-26T19:02:15 UTC - (JSON DATE) 2016-11-22T20:06:49 UTC = 94 hours
[CRITICAL] Delta hours (94) is >= (24) hours. Data is stale.

Answer 1

You could try use Hash::Flatten. For example:

use Hash::Flatten qw(flatten);

my $json_decoded = parse_json($json_str);
my $flat = flatten( $json_decoded );
say "found" if grep /(?:^|\.)\Q$key\E(?:\.?|$)/, keys %$flat;

Answer 2

You can use Data::Visitor::Callback to traverse the data structure. It lets you define callbacks for different kinds of data types inside your structure. Since we're only looking at a hash it's relatively simple.

The following program has a predefined list of keys to find (those would be user input in your case). I converted your example JSON to a Perl hashref and included it in the code because the conversion is not relevant. The program visits every hashref in this data structure (including the top level) and runs the callback.

Callbacks in Perl are code references. These can be created in two ways. We're doing the anonymous subroutine (sometimes called lambda function in other languages). The callback gets passed two arguments: the visitor object and the current data substructure.

We'll iterate all the keys we want to find and simply check if they exist in that current data structure. If we see one, we count it's existence in the %seen hash. Using a hash to store things we have seen is a common idiom in Perl.

We're using a postfix if here, which is convenient and easy to read. %seen is a hash, so we access the value behind the $key with $seen{$key}, while $data is a hash reference, so we use the dereferencing operator -> to access the value behind $key with $data->{$key}.

The callback needs us to return the $data again so it continues. The last line is just there, it's not important.

I've used Data::Printer to output the %seen hash because it's convenient. You can also use Data::Dumper if you want. In production, you will not need that.

use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;

my $from_json = {
    "localDate"  => "Wednesday 23rd November 2016 11:03:37 PM",
    "utcDate"    => "Wednesday 23rd November 2016 11:03:37 PM",
    "format"     => "l jS F Y h:i:s A",
    "returnType" => "json",
    "timestamp"  => 1479942217,
    "timezone"   => "UTC",
    "daylightSavingTime" =>
        0,    # this was false, I used 0 because that's a non-true value
    "url"    => "http:\/\/www.convert-unix-time.com?t=1479942217",
    "subkey" => {
        "altTimestamp" => 1479942217,
        "altSubkey"    => {
            "thirdTimestamp" => 1479942217
        }
    }
};

my @keys_to_find = qw(timestamp altTimestamp thirdTimestamp missingTimestamp);

my %seen;
my $visitor = Data::Visitor::Callback->new(
    hash => sub {
        my ( $visitor, $data ) = @_;

        foreach my $key (@keys_to_find) {
            $seen{$key}++ if exists $data->{$key};
        }

        return $data;
    },
);
$visitor->visit($from_json);

p %seen;

The program outputs the following. Note this is not a Perl data structure. Data::Printer is not a serializer, it's a tool to make data human readable in a convenient way.

{
    altTimestamp     1,
    thirdTimestamp   1,
    timestamp        1
}

---

Since you also wanted to constraint the input, here's an example how to do that. The following program is a modification of the one above. It allows to give a set of different constraints for every required key.

I've done that by using a dispatch table. Essentially, that's a hash that contains code references. Kind of like the callbacks we use for the Visitor.

The constraints I've included are doing some things with dates. An easy way to work with dates in Perl is the core module Time::Piece. There are lots of questions around here about various date things where Time::Piece is the answer.

I've only done one constraint per key, but you could easily include several checks in those code refs, or make a list of code refs and put them in an array ref (keys => [ sub(), sub(), sub() ]) and then iterate that later.

In the visitor callback we are now also keeping track of the keys that have %passed the constraints check. We're calling the coderef with $coderef->($arg). If a constraint check returns a true value, it gets noted in the hash.

use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
use Time::Piece;
use Time::Seconds; # for ONE_DAY

my $from_json = { ... }; # same as above

# prepare one of the constraints
# where I'm from, Christmas eve is considered Christmas
my $christmas = Time::Piece->strptime('24 Dec 2016', '%d %b %Y');

# set up the constraints per required key
my %constraints = (
    timestamp        => sub {
        my ($epoch) = @_;
        # not older than one day

        return $epoch < time && $epoch > time - ONE_DAY;
    },
    altTimestamp     => sub {
        my ($epoch) = @_;
        # epoch value should be an even number

        return ! $epoch % 2;
    },
    thirdTimestamp   => sub {
        my ($epoch) = @_;
        # before Christmas 2016

        return $epoch < $christmas;
    },
);

my %seen;
my %passed;
my $visitor = Data::Visitor::Callback->new(
    hash => sub {
        my ( $visitor, $data ) = @_;

        foreach my $key (%constraints) {
            if ( exists $data->{$key} ) {
                $seen{$key}++;
                $passed{$key}++ if $constraints{$key}->( $data->{$key} );
            }
        }

        return $data;
    },
);
$visitor->visit($from_json);

p %passed;

The output this time is:

{
    thirdTimestamp   1,
    timestamp        1
}

---

If you want to learn more about the dispatch tables, take a look at chapter two of the book Higher Order Perl by Mark Jason Dominus which is legally available for free here.