PHP: Retrieving lines from the end of a large text file

Question

I've searched for an answer for quite a while, and haven't found anything that works correctly.

I have log files, some reaching 100MB in size, around 140,000 lines of text. With PHP, I am trying to get the last 500 lines of the file.

How would I get the 500 lines? With most functions, the file is read into memory, and that isn't a plausible case for this matter. I would preferably stay away from executing system commands.

[an answer that will help you](http://stackoverflow.com/a/3234857/612202) — dan-lee, Jun 17 '12 at 01:14

score 6 · Accepted Answer · answered Jun 17 '12 at 01:13

6

If you are on a 'nix machine, you should be able to use shell escaping and the tool 'tail'. It's been a while, but something like this:

$lastLines = `tail -n 500`;

notice the use of tick marks, which executes the string in BASH or similar and returns the results.

answered Jun 17 '12 at 01:13

Chris Trahey

18,202
1
42
55

http://php.net/manual/en/language.operators.execution.php: `The backtick operator is disabled when safe mode is enabled or shell_exec() is disabled.` This could be important for shared hosting. – Bailey Parker Jun 17 '12 at 01:25
Hmm. If shell execution is not available, then the solution may be a quick and tricky little algorithm utilizing fopen, fseek, and a loop; essentially reading the file backwards until you have 500 lines... – Chris Trahey Jun 17 '12 at 01:33
Yep. Although I do like your clever solution (+1), lots of people have shared hosting with safemode or shell_exec() disabled. – Bailey Parker Jun 17 '12 at 01:37
2

@hexacyanide, Why do you need to run it five times a second? Maybe you are better off using `proc_open()` and just listening to the output of `tail -f`. (Or maybe there's no need to use PHP to do whatever you are doing.) – Matthew Jun 17 '12 at 01:49
Yeah, it's a good question. Are you reading the same file 5 times per second, or just hoping to get through a pile of files in a certain amount of time? The former may benefit from a different approach to the problem. – Chris Trahey Jun 17 '12 at 01:55
1

It may be worth exploring a more stream-oriented approach, since the approach discussed here has two pitfalls: 1. over-examination of the file when changes are sparse, and 2. potentially under-examination when they are active. Any combination of line count and frequency is only a compromise between these two extremes, when there could be a solution which has parity with the quantity of actual changes (i.e. it 'hangs' until there is an actual change in the log.) If you want to explore that option, it merits a different SO post, at least. – Chris Trahey Jun 17 '12 at 04:06
1

After some experimentation, this seems to be the easiest and most reliable way to continuously tail a file: `$stream = popen("tail -n 500 -f $filename" , "r");` You can safely use `fgets()` without it eating up CPU. – Matthew Jun 18 '12 at 01:11

score 4 · Answer 2 · edited Sep 26 '16 at 13:14

I wrote this function which seems to work quite nicely to me. It returns an array of lines just like file. If you want it to return a string like file_get_contents, then just change the return statement to return implode('', array_reverse($lines));:

function file_get_tail($filename, $num_lines = 10){

    $file = fopen($filename, "r");

    fseek($file, -1, SEEK_END);

    for ($line = 0, $lines = array(); $line < $num_lines && false !== ($char = fgetc($file));) {
        if($char === "\n"){
            if(isset($lines[$line])){
                $lines[$line][] = $char;
                $lines[$line] = implode('', array_reverse($lines[$line]));
                $line++;
            }
        }else
            $lines[$line][] = $char;
        fseek($file, -2, SEEK_CUR);
    }
    fclose($file);

    if($line < $num_lines)
        $lines[$line] = implode('', array_reverse($lines[$line]));

    return array_reverse($lines);
}

Example:

file_get_tail('filename.txt', 500);

there is a small typo in your code -> ($): $file = fopen("filename", "r"); -> $file = fopen("$filename", "r"); Other than that, it woks fine. How about if i want to access a log file in remote server? — Mohammed Joraid, Apr 06 '13 at 13:20

Matthew · Answer 3 · 2012-06-17T01:51:02.907

If you want to do it in PHP:

<?php
/**
  Read last N lines from file.

  @param $filename string  path to file. must support seeking
  @param $n        int     number of lines to get.

  @return array            up to $n lines of text
*/
function tail($filename, $n)
{
  $buffer_size = 1024;

  $fp = fopen($filename, 'r');
  if (!$fp) return array();

  fseek($fp, 0, SEEK_END);
  $pos = ftell($fp);

  $input = '';
  $line_count = 0;

  while ($line_count < $n + 1)
  {
    // read the previous block of input
    $read_size = $pos >= $buffer_size ? $buffer_size : $pos;
    fseek($fp, $pos - $read_size, SEEK_SET);

    // prepend the current block, and count the new lines
    $input = fread($fp, $read_size).$input;
    $line_count = substr_count(ltrim($input), "\n");

    // if $pos is == 0 we are at start of file
    $pos -= $read_size;
    if (!$pos) break;
  }

  fclose($fp);

  // return the last 50 lines found  

  return array_slice(explode("\n", rtrim($input)), -$n);
}

var_dump(tail('/var/log/syslog', 50));

This is largely untested, but should be enough for you to get a fully working solution.

The buffer size is 1024, but can be changed to be bigger or larger. (You could even dynamically set it based on $n * estimate of line length.) This should be better than seeking character by character, although it does mean we need to do substr_count() to look for new lines.

For an untested solution, it works extremely well out of the box. Thank you my stuff finally working — rezizter, Oct 12 '15 at 10:15

PHP: Retrieving lines from the end of a large text file

3 Answers3

Linked