Monday, October 15, 2007

Effect of buffer size for reading files [Linux]

Reading a file entirely in memory is not a good idea as stated before, but how large should be the memory buffer?

I did a test under Linux (Ubuntu 7.04) reading the fpc2.2.0 installation file (29MB) using different buffer sizes.

Here's the result:

The time to read the file decreases as the buffer size increases until the buffer is 128kb then, as the buffer gets bigger, the trend inverts.

Some notes:
  • The test was executed three times for each buffer size. The results are expressed as the Median;
  • The Y axis is the time to read all the file in microseconds. The X axis is the buffer size in bytes;
  • The first time the file is read is significantly slower than subsequent reads. Probably this is an effect of the OS file system buffering (I did not find a way to skip it). This limits further analysis. However, excluding the first run, all other results are consistent across the same buffer size. All results can be browsed here.

No comments: