2

I created an array from a very large text file with more than 10.000 entries. This is how my textfile is formatted:

1232.cat=Fred
1232.age=13
1232.size=44
1232.food=chicken
1233.cat=Moe
1233.age=4
1233.size=23
1233.food=fish

This is my $animals array: 

 array(9435) {
  [0]=>
  array(5) {
    ["number"]=>
    string(4) "1232"
    ["cat"]=>
    string(4) "Fred"
    ["age"]=>
    string(2) "13"
    ["size"]=>
    string(2) "44"
    ["food"]=>
    string(7) "chicken"
   }
   ...(and so on)

I want to do various checks, for example if for each number cat exists.

    foreach($animals as $row) {
        if(empty($row['cat'])){
        echo "cat is missing in number: ".$row["number"];
       } 
    }

This is working very well for textfiles with around 3000 entries. But for larger files I cannot get any results. So my question is, what can I do. Obviously I cannot loop through very large arrays. But what is my alternative?

peace_love
  • 6,229
  • 11
  • 69
  • 157
  • 1
    not sure, but would the problem lies in `set_time_limit` also ?? http://php.net/manual/en/function.set-time-limit.php – Andrew Nov 23 '15 at 15:20
  • 3
    You could [read the file line by line](http://stackoverflow.com/questions/13246597/how-to-read-a-file-line-by-line-in-php) instead of the whole file. Given that you don't run out of memory when you read the whole file, you can use a [generator](http://php.net/manual/en/language.generators.syntax.php) and yield only what you need. Worth mentioning, @Andrew and myself are 2 different people. – Andrei Nov 23 '15 at 15:21
  • @Andrew just notice lol – Andrew Nov 23 '15 at 15:24
  • There is a limit how large files PHP can read. As @Andrew said, you can read the file partially line by line!] – Hasse Björk Nov 23 '15 at 15:25
  • 1
    @Andrew We should enter a thread and have 2 different opinions on something. That way people will think somebody is arguing with himself. :D – Andrei Nov 23 '15 at 15:25
  • @Andrew lol, I was thinking the same thing and it becomes an enormous infinite of loop – Andrew Nov 23 '15 at 15:28
  • @Andrew I tried `set_time_limit(60);` on the beginning of my page and after 54 seconds I got an output! That is really great. Do you know what is the largest time limit I can set? – peace_love Nov 23 '15 at 15:39
  • @Andrew I will now have a look at line by line – peace_love Nov 23 '15 at 15:40
  • you can use JudyArray for large data input/output https://secure.php.net/manual/fr/book.judy.php – Halayem Anis Nov 23 '15 at 15:42
  • @jarla Every time you call `set_time_limit` it will reset the timer so you can think of an reasonable amount of time a single line should have to parse in. – Maarten Bicknese Nov 23 '15 at 15:43
  • 1
    Have you tried using `array_filter` rather than iterating over your large array? Something like `$noCats = array_column(array_filter($arr, function($item) { return empty($item['cat']); }), 'number');` would give you an array of the 'number' fields with an empty 'cat'. – benJ Nov 23 '15 at 15:51

0 Answers0