Fast read/write from file in delphi

Question

I am loading a file into a array in binary form this seems to take a while is there a better faster more efficent way to do this. i am using a similar method for writing back to the file.

procedure openfile(fname:string);
var
    myfile: file;
    filesizevalue,i:integer;
begin
  assignfile(myfile,fname);
  filesizevalue:=GetFileSize(fname); //my method
  SetLength(dataarray, filesizevalue);
  i:=0;
  Reset(myFile, 1);
  while not Eof(myFile) do
    begin
      BlockRead(myfile,dataarray[i], 1);
      i:=i+1;
    end;
  CloseFile(myfile);
end;

Please reformat, this is hard to read – schnaader Jan 18 '09 at 20:01 — schnaader, Jan 18 '09 at 20:01

Davy Landman · Answer 1 · 2009-01-19T08:23:19.510

If your really want to read a binary file fast, let windows worry about buffering ;-) by using Memory Mapped Files. Using this you can simple map a file to a memory location an read like it's an array.

Your function would become:

procedure openfile(fname:string);
var
    InputFile: TMappedFile;
begin
  InputFile := TMappedFile.Create;
  try
    InputFile.MapFile(fname);
    SetLength(dataarray, InputFile.Size);
    Move(PByteArray(InputFile.Content)[0], Result[0], InputFile.Size);
  finally
    InputFile.Free;
  end;
end;

But I would suggest not using the global variable dataarray, but either pass it as a var in the parameter, or use a function which returns the resulting array.

procedure ReadBytesFromFile(const AFileName : String; var ADestination : TByteArray);
var
    InputFile : TMappedFile;
begin
  InputFile := TMappedFile.Create;
  try
    InputFile.MapFile(AFileName);
    SetLength(ADestination, InputFile.Size);
    Move(PByteArray(InputFile.Content)[0], ADestination[0], InputFile.Size);
  finally
    InputFile.Free;
  end;
end;

The TMappedFile is from my article Fast reading of files using Memory Mapping, this article also contains an example of how to use it for more "advanced" binary files.

Have you ever measured the performance? I guess, memory mapped files should win hands down, but I'd like to see some figures before I start to rewrite any code. — maaartinus, Mar 12 '16 at 03:36

schnaader · Accepted Answer · 2009-01-19T08:24:29.943

16

You generally shouldn't read files byte for byte. Use BlockRead with a larger value (512 or 1024 often are best) and use its return value to find out how many bytes were read.

If the size isn't too large (and your use of SetLength seems to support this), you can also use one BlockRead call reading the complete file at once. So, modifying your approach, this would be:

AssignFile(myfile,fname);
filesizevalue := GetFileSize(fname);
Reset(myFile, 1);
SetLength(dataarray, filesizevalue);
BlockRead(myFile, dataarray[0], filesizevalue);
CloseFile(myfile);

Perhaps you could also change the procedure to a boolean function named OpenAndReadFile and return false if the file couldn't be opened or read.

edited Jan 19 '09 at 08:24

answered Jan 18 '09 at 20:03

schnaader

49,103
10
104
136

im sure you just made a typo but this does not work with i replaced as 1 and filesizevalue – Arthur Jan 18 '09 at 23:53
Yes, that's a typo. It should be filesizevalue, also be aware that your array should be an array of byte. – schnaader Jan 19 '09 at 07:49
Fixed now. Seems it has to be dataarray[0], too. – schnaader Jan 19 '09 at 08:25

score 5 · Answer 3 · answered Jan 18 '09 at 20:38

It depends on the file format. If it consists of several identical records, you can decide to create a file of that record type.

For example:

type
  TMyRecord = record
    fieldA: integer;

    ..
  end;
  TMyFile = file of TMyRecord;

  const
    cBufLen = 100 * sizeof(TMyRecord);
  var
    file: TMyFile;
    i : Integer;

  begin
    AssignFile(file, filename);
    Reset(file);
    i := 0;
    try
      while not Eof(file) do begin
        BlockRead(file, dataarray[i], cBufLen);
        Inc(i, cBufLen);
      end;
    finally
      CloseFile(file);
    end;
  end;

Good answer. I think you need to use the packed keyword when you're writing to files to be sure that you'll be easily able to retrieve the contents in future versions of delphi or from other languages as byte-alignment of the fields could be different. — RobS, Jan 19 '09 at 09:57

score 3 · Answer 4 · answered Jan 19 '09 at 05:49

If it's a long enough file that reading it this way takes a noticeable amount of time, I'd use a stream instead. The block read will be a lot faster, and there's no loops to worry about. Something like this:

procedure openfile(fname:string);
var
    myfile: TFileStream;
    filesizevalue:integer;
begin
  filesizevalue:=GetFileSize(fname); //my method
  SetLength(dataarray, filesizevalue);
  myFile := TFileStream.Create(fname);
  try
    myFile.seek(0, soFromBeginning);
    myFile.ReadBuffer(dataarray[0], filesizevalue);
  finally
     myFile.free;
  end;
end;

It appears from your code that your record size is 1 byte long. If not, then change the read line to:

  myFile.ReadBuffer(dataarray[0], filesizevalue * SIZE);

or something similar.

score 0 · Answer 5 · answered Jan 19 '09 at 09:32

0

Look for a buffered TStream descendant. It will make your code a lot faster as the disk read is done fast, but you can loop through the buffer easily. There are various about, or you can write your own.

answered Jan 19 '09 at 09:32

mj2008

6,647
2
38
56

score 0 · Answer 6 · answered Jan 19 '09 at 16:41

0

If you're feeling very bitheaded, you can bypass Win32 altogether and call the NT Native API function ZwOpenFile() which in my informal testing does shave a tiny bit off. Otherwise, I'd use Davy's Memory Mapped File solution above.

answered Jan 19 '09 at 16:41

Jesse C. Slicer

19,901
3
68
87

Fast read/write from file in delphi

6 Answers6

Linked