Why is writing data slow very slow in Delphi?

Question

I am designing, for my A2 Computing project, an application to simulate the response of a variety of filters. One problem I'm having is that the export data option is really slow.

Normally, when generating data to show on the display it's about 40000 - 80000 points/sec. When recording it to a file, it drops to about a fifth of that.

Initially I thought my problem was because I was calling writeln on every data point. So I wrote it such that it queued the data up into a string and every 1000 points it wrote it in one big operation. It made it slightly faster, but still around 4-5x slower as displaying it on the built in form.

Why might this be?

Here's the export code:

   for xx := 0 to npoints do
   begin
     freq := minfreq + ((xx / npoints) * maxfreq);
     ampl := GetAmplPoint(freq);
     phase := GetPhasePoint(freq);
     tempstr := tempstr + FormatFloat('#.#####', freq) + ',';
     tempstr := tempstr + FormatFloat('#.#####', ampl) + ',';
     tempstr := tempstr + FormatFloat('#.#####', phase) + sLineBreak;
     // Queue up to 1000 points, then write the data in one lump:
     // most of the time is spent in writeln waiting for IO which
     // slows down export.
     if xx mod 1000 = 0 then
     begin
       write(fileptr, tempstr);
       tempstr := '';
       ProgressBar.Position := 4 + Trunc((xx / npoints) * 96);
     end;
   end;

Access to disk takes more time than showing up info to screen — RBA, Dec 13 '11 at 12:54
take a look on this http://stackoverflow.com/questions/5639531/buffered-files-for-faster-disk-access — RBA, Dec 13 '11 at 12:58
Your problem is probably that there isn't a big enough buffer between you and the OS. I know that you can modify the way class Pascal I/O handles buffers but really you ought to be using streams. My answer here http://stackoverflow.com/questions/5639531/buffered-files-for-faster-disk-access/5639712#5639712 gives a very fast stream class for writing to disk. — David Heffernan, Dec 13 '11 at 12:59
I'm writing about 1 MB/sec to the file at the current rate. I'm only using old college computers, so it may well be that there is no easy way to go faster. — Thomas O, Dec 13 '11 at 13:28
that sounds very slow. Copy a large file to see what you should be shooting for. — David Heffernan, Dec 13 '11 at 13:48

score 4 · Accepted Answer · answered Dec 13 '11 at 13:21

4

Disk I/O is one of the slowest bottlenecks today, especially if you use slow disks (i.e. the 4200/5400 rpm disks found on many laptops).

Better perfomance can be obtained using buffered I/O (old pascal I/O functions were designed long ago, and may use small buffers, better to use one of the buffered streams available today in Delphi) or asynch I/O (you pass the buffer to write to the OS, the call returns immediately and later the OS will tell you when it had written the data).

answered Dec 13 '11 at 13:21

Mad Hatter

772
5
5

If the problem is lack of buffering then paragraph 1 is superfluous – David Heffernan Dec 13 '11 at 15:03
Buffering helps more with slow disks. Writing a lot of small pieces requires to move more the disk heads and wait the required sectors come there. Writing large chunks at once minimize such movements. The OS will buffer anyway, but when writing large files larger buffers may help. – Mad Hatter Dec 14 '11 at 15:12
With modern OS, the OS buffers like mad anyway. So buffering in apps doesn't actually help with disk access. What it does is avoid the overhead of hitting to the Windows API just to write a couple of bytes. That overhead is huge in a tight loop. – David Heffernan Dec 14 '11 at 15:19

score 2 · Answer 2 · answered Dec 13 '11 at 15:05

If I recall correctly.... you will have much better performance with binary files than text files. The contents don't matter, it's just the declaration. You don't show the declaration for fileptr. But if it's declared like this:

var fileptr : TextFile;

it will be slower than:

var fileptr : File;

or

var fileptr : File of <some record or type>;

Give that a try and see if it speeds things up. You can also use BlockWrite(). Note that your output may lag behind the program, due to buffering. You probably want to flush the file in any error handler.

It is better to start to forget the old Pascal I/O API and switch to a more modern one. If speed is very important calling directly OS APIs with the most useful parameters may help more than using an API designed for CP/M or DOS. — Mad Hatter, Dec 14 '11 at 15:14

Why is writing data slow very slow in Delphi?

2 Answers2