Creating data driven tests dynamically with FPTest/DUnit

I've been trying, whenever possible, to write tests along side new code i write. In fact, in one recent mid sized project, i created the tests before writing the code and the experience was broadly positive.

Given the personal need of a JSON Schema validator implemented in pascal, i decided to write one and, naturally, with a test driven approach.

The JSON Schema organization maintains a language agnostic test suite.  It's comprised of JSON files describing the specifications for each rule a validator must check.

I could write a program to convert the JSON specification to pascal units with the tests cases like i've done with mustache spec, but is far from optimal approach, imposing the need to recreate the test application each time a change is done in the spec.

So, i looked a way to create the tests dynamically reading directly the JSON files. A quick search lead me to the solution of creating a custom TTestCase class with a published method (named generically as Run) that implements the test. An instance of this class is created for each test, passing the appropriate data.

While it works, this approach has the drawback of the generic method name that would clutter the test runner output with meaningless information. To overcome this issue is possible to aggregate tests in one big test case, e.g., a unique test case for JSON Schema type rule, instead of creating one test case for each test description.

With the confidence that should exist a better solution, i digged into FPTest source code (a freepascal port of DUnit2) looking how i could have the best of two worlds, data driven dynamic tests with the granularity of handcraft tests.

Fortunately, i've got a way. The key is to subclass TTestProc and properly instantiate it .

  TJSONSchemaTestProc = class(TTestProc)
  private
    FData: TJSONObject;
    procedure ExecuteTest(SchemaData, TestData: TJSONObject);
    procedure ExecuteTests;
  public
    constructor Create(Data: TJSONObject);
  end;

constructor TJSONSchemaTestProc.Create(Data: TJSONObject);
begin
  inherited Create(@ExecuteTests, '', @ExecuteTests, Data.Get('description', 'jsonschema-test'));
  FData := Data;
end;

A TJSONObject with the test specification is passed in constructor. A not published method (ExecuteTests) is registered with the description of the test as name.

procedure TJSONSchemaTestProc.ExecuteTest(SchemaData, TestData: TJSONObject);
var
  Description: String;
  ValidateResult: Boolean;
begin
  Description := TestData.Get('description', '');
  ValidateResult := ValidateJSON(TestData.Elements['data'], SchemaData);
  if TestData.Booleans['valid'] then
    CheckTrue(ValidateResult, Description)
  else
    CheckFalse(ValidateResult, Description);
end;

procedure TJSONSchemaTestProc.ExecuteTests;
var
  SchemaData: TJSONObject;
  TestsData: TJSONArray;
  i: Integer;
begin
  SchemaData := FData.Objects['schema'];
  TestsData := FData.Arrays['tests'];
  for i := 0 to TestsData.Count - 1 do
    ExecuteTest(SchemaData, TestsData.Objects[i]);
end;

In ExecuteTests, the assertions (called in the specification tests) are executed one by one.

With this i get a comprehensive test suite that allows to effectively drive the development.

 

While took me some time to understand the FPTest/DUnit2 source code, the solution ended simpler and clearer than i think earlier. In a way that i foresee using this technique for testing other projects, not only third party specifications.

BTW: the test runner source code can be found here

Effect of using a constant parameter for string types (revisited)

Long eight years ago i wrote an post about using const for string parameters and effects in generated code. It showed benefits in using const for string types but was far from difference showed with similar test done with Delphi. I never bothered to replicate in Freepascal, i took it as granted.

As the discussion arose in forum, i decided to do a test equals to Delphi one, basically just using the parameter without modifying it, by the way, the most common usage.

I compared

procedure ByValueReadOnly(V: String);
begin
  DoIt(V);
end;

with

procedure ByReferenceReadOnly(const V: String);
begin
  DoIt(V);
end;    

The result talks by itself

Also compared

procedure ByValue(V: String);
begin
  V := V + 'x';
  DoIt(V);
end;

with

procedure ByReference(const V: String);
var
  S: String;
begin
  S := V + 'x';
  DoIt(S);
end;

The generated code is similar, size and performance wise.

For those that underestimate the impact of such differences, read this.

For the curious (or the wary), i uploaded the code.

The cost to supress a warning (and how not pay for it)

In the previous post, i pointed that passing a managed type (dynamic array) as a var parameter is more efficient than returning the value as a function result. However this technique have a known side effect: the compiler outputs a message  (Warning: Local variable "XXX" does not seem to be initialized) each time a call to the procedure is compiled.

The direct way to suppress the warning is change the parameter from var to out. Pretty simple but out does more than inhibit the compiler message. It implicitly initialize managed types parameters to nil or add a call FPC_INITIALIZE if the parameter is a record that has at least a field of a managed type. It does not add implicit code to simple types like Integer or class instances (TObject etc).

Although the performance impact is mostly negligible, is extra code anyway. In my case i initialize the parameter explicitly so out would add redundant code. There's an alternative to suppress the message: add the directive {%H-} in front of the variable that is being passed to the procedure. In the example of the previous post would be:

BuildRecArray({%H-}Result);

It can be annoying if the function is called often or the routine is part of a public API, otherwise is fine. At least for me.

Update: out does not generate initialization code for records that contains only fields which type is not automatically managed by the compiler, e.g., Integer.

Does it matter how dynamic arrays are passed/returned to/from a routine?

I was implementing a routine that should return a dynamic array and wondered if the produced code of a function and a procedure with a var parameter are different. So, i setup a simple test:

type
  TMyRec = record
    O: TObject;
    S: String;
  end;

  TMyRecArray = array of TMyRec;

function BuildRecArray: TMyRecArray;
begin
  SetLength(Result, 1);
  Result[0].O := nil;
  Result[0].S := 'x';
end;

procedure BuildRecArray(var Result: TMyRecArray);
begin
  SetLength(Result, 1);
  Result[0].O := nil;
  Result[0].S := 'x';
end;

var
  Result: TMyRecArray;

begin
  BuildRecArray(Result); //or Result := BuildRecArray
end.

Looking at the generated assembly revealed that the function version (returns the array in the result) leads to bigger code when compared with the procedure version (pass the array as a var parameter). More: the code difference is due to an implicit exception frame which is known to impact performance.

And what about the caller code? Again the function version generates more code (creates a temporary variable and calls FPC_DYNARRAY_DECR_REF).

In short: yes, it matters.

The cost of using generics

Since a few versions, fpc provides support for generics. It allows the developer to save some typing and also improves type safety at the time that prevents unsafe typecasts.

Unfortunately it's benefits is not for free.  Every time a generic is specialized, the whole implementation code is copied into the unit/program.

To be more clear, i created two examples that implements a list of a custom class (TMyObj): one uses a TFPList, the other specializes a TFPGList. The difference in usage is that the former needs a typecast.

I compiled both with fpc 2.6.0 under windows. The result is a difference in executable size of 2Kb, the generic version being bigger. Than i looked the generated asm: the code to use the list classes are the same, the difference comes from the copy of implementation of  TFPGList.

Many will say that code size is not a issue anymore given the availability of big hard drives, but i still think that is a good practice seeking smaller code. Regarding generics, it should be used, IMHO, when benefits are clear like classes that are instantiated many times in user (programmer) code and avoided in internal structures of e.g. RTL or third party libraries.

JSON to the rescue

Tired of creating boilerplate dialogs for every TDataSet i needed to edit, i developed a generic dialog with a data grid plus the basic edit buttons (add, delete). To edit a TDataset i just call a wrapper function that setup the grid with some configuration.

While most of the time i just needed to edit a single field, it was desirable to keep the ability to edit more than one field. Also it would be necessary also to set the column width for each field. So i was passing a record with the following structure to configure the dialog:

  TDataDialogInfo = record
    FieldNames: String;
    FieldWidths: array of Integer;
    Title: String;
  end;

Title stores the dialog title, FieldNames store the Field names in semicolon separated string and FieldWidths the width of corresponding field. Here is the first problem: without adding another record type with the field name and width there is noway to guarantee the correspondence between the values leading to error easily.

Another problem is that there's no way in fpc to create the record on the fly so i need to declare a constant somewhere. To use a dialog with Title "Edit Test Dataset" and editing field "Name" i need to do:

const
  Info: TDataDialogInfo = (
    FieldNames: 'Name';
    FieldWidths: nil;
    Title: 'Edit Test Dataset'
    );

Notice that even if i don't need/want to set the width i have to declare the record field Widths.

I used this way for some time, and is really a lot less work than creating a TForm and setting the components for each TDataset, until i needed to add an option to show more rows than the usual in the grid. How to add such option? Adding a field to the TDataDialogInfo record would make all previous defined constants unusable and i would need to fix than all. What if later i needed to add another option?

It was at this moment that i think: "How it would be good if pascal had the ability to define objects with arbitrary fields on the fly, like in JavaScript!". Well, in fact, although indirectly, is possible to do it in pascal thanks to the native fpc implementation of JSON.

So instead of using a record to pass the configuration now i use a string with a (valid) JSON definition. In the previous example (Title "Edit Test Dataset" and editing field "Name") now i do:

Info = '{"title": "Edit Test Dataset", "fields": {"name": "Name", "title": "Person"}}';

If i just need to define the field name:

Info = '"Name"';

If i want to edit two fields:

Info = '["Id", "Name"]';

And i can set different properties for each field

Info = '{"title": "Edit Test Dataset",'+
    '"fields": ["Id", {"name": "Name", "title": "Person"},' +
    ' { "name": "Phone", "width": 100}]}';

Notice that the format of the property (in this case "fields") vary from a single string to a JSON array or object. Better: you can mix in the same property different formats! Flexibility it's your name. All of this backward compatible: i can add another property and previous definitions will still work!

The cost of accessing object fields (part 2)

In the last post, we could see the benefits of using a temporary variable to access frequently used object fields. What if the object field is accessed only two times. The benefit would be maintained?
Let's see this example:
Before:

begin
  if FDataLink.Field <> nil then
    Caption := FDataLink.Field.DisplayText
  else
    Caption := '';
end; 

After:

var
  DataLinkField: TField;
begin
  DataLinkField := FDataLink.Field;
  if DataLinkField <> nil then
    Caption := DataLinkField.DisplayText
  else
    Caption := '';
end; 

It seems that yes, although very little (saves only two instructions). This is the kind of optimization to be done on only very sensitive areas.

Since the benefit was mainly due to the compiler saving the local variable in a register, a doubt that i had in mind was what would happen in a method with many parameters? The addition of the variable would still be beneficial?

So i tested the addition of a variable in a method with the following header

procedure DoIt(Sender, Sender2, Sender3: TObject);

As we can see, the version with the local variable is still smaller.

All in all, some like to say that less is more, but sometimes, as in this case, more is less!

The cost of accessing object fields (part 1)

The common sense make us believe that adding more code and/or more variables leads to bigger programs. Looking at the generated code of one example in the previous post, the addition of one variable made the executable smaller. This occurs because fpc is smart enough to reuse registers (in this case eax).

This week, while fixing one Lazarus bug i noticed the following pattern in the generated code of method TDBEdit.DataChange:

 movl 12(%ebx),%eax
 movl 24(%eax),%eax

Basically this is the code to access FDataLink.Field property (the first instruction get the FDataLink address and the second get the Field address). So what would happen if this field was "buffered" in a TField local variable?

Before:

procedure TDBEdit.DataChange(Sender: TObject);
begin
  if FDataLink.Field <> nil then begin
    Alignment := FDataLink.Field.Alignment; 
  [..]

After:

procedure TDBEdit.DataChange(Sender: TObject);
var
  DataLinkField: TField;
begin
  DataLinkField := FDataLink.Field;
  if DataLinkField <> nil then begin
    Alignment := DataLinkField.Alignment; 
  [..]

This simple change lead to these differences.

As expected the code became smaller but two things surprised me:

There's no increase in the temporary memory allocated

The variable assignment did cost nothing (not even one instruction)

The above test was done with a "clone" of TDBEdit.DataChange in a test project. To make sure there are no confounding factors i also tested with the original code to confirm the differences. Notice that in this case, although the code is also smaller, the addition of the variable increase the temporary memory allocated as well the variable assignment requires one extra instruction. Bad.

But there was one last hope: compile LCL with -O2 option (i assumed that LCL was already compiled with that optimization turned on). Seems that my assumption was wrong. The -O2 option did the trick: the same result as before.

In the next post i will play with a few more scenarios.

And remember: don't forget to put -O2 in LCL build options when doing a release, it makes difference.

Condition check versus a type map

Often, the programmer is faced with the need to translate from one type to another, e.g., given a boolean variable return a corresponding integer value. As a real world example see a piece of Lazarus code:

  if NewWordWrap then
    gtk_text_view_set_wrap_mode(AGtkTextView, GTK_WRAP_WORD)
  else
    gtk_text_view_set_wrap_mode(AGtkTextView, GTK_WRAP_NONE); 

NewWordWrap is a boolean variable, but the gtk function expects an integer. To translate from type to another a condition check is done.

Another way to handle this would be creating a map array with the type to be translated. Lazarus also has an example of this technique:

const
  WidgetDirection : array[boolean] of longint = (GTK_TEXT_DIR_LTR, GTK_TEXT_DIR_RTL);
[..]
  gtk_widget_set_direction(AGtkWidget, WidgetDirection[UseRightToLeftAlign]);

Here is the same pattern: UseRightToLeftAlign is a boolean variable and the gtk function expects a integer, but instead of checking for the variable value a boolean to integer map (WidgetDirection) is used.

While the map approach seems faster because avoids a check, it adds an additional constant. I decided to look at the generated code to see the actual benefits.

Check the condition code:

  if B then
    DoIt(CONST_1)
  else
    DoIt(CONST_2);

Map code:

const
  BoolMap: array[Boolean] of Integer = (CONST_2, CONST_1);

  DoIt(BoolMap[B])

Here is the generated code. This shows a clear advantage to the map approach. Notice that in this small example the size of executables were the same.

I also tested a more complex type than boolean: an enumerated.

Check the condition code:

  case E of
    EnumA: DoIt(CONST_1);
    EnumB: DoIt(CONST_2);
    EnumC: DoIt(CONST_3);
  end;

Map code:

const
  EnumMap: array[TMyEnum] of Integer = (CONST_1, CONST_2, CONST_3);

  DoIt(EnumMap[E])

The result.

Now with a slight optimized code for the condition check...

var
  I: Integer;

  case E of
    EnumA: I := CONST_1;
    EnumB: I := CONST_2;
    EnumC: I := CONST_3;
  end;
  DoIt(I);

... i got this.

The discover of RTTI

I never got much interest, or knowledge, by Delphi/fpc RTTI. But the recent fuzz about the new RTTI features introduced in Delphi 2010 raised my curiosity even if the fpc provides only the old style RTTI. 

The opportunity to learn, and use, RTTI came when i figured the possibility to enhance/clear some of my code.

To show a generic TForm i created a very imaginative simple function:

function ShowForm(FormClass: TFormClass; Owner: TWinControl): TModalResult;
var
  Form: TForm;
begin
  Form := FormClass.Create(Owner);
  try
    Result := Form.ShowModal;
  finally
    Form.Destroy;
  end;
end;

This little function works nice and save some boilerplate code but it's limited to forms that don't need to initialize a property before is called since i don't know class type before hand.

As stated before, messages can be used to notify with arbitrary information any TControl, so i created a variant of the ShowForm that takes two ordinal (LPARAM and WPARAM) parameters and send a CM_INIT message to the created TForm instance. The TForm descendant would need to add a CM_INIT message handler and interpret the TLMessage parameter.

function ShowForm(FormClass: TFormClass; Owner: TWinControl; WData: WPARAM = 0; LData: LPARAM = 0): TModalResult;
var
  Form: TForm;
begin
  Form := FormClass.Create(Owner);
  try
    Form.Perform(CM_INIT, WData, LData);
    Result := Form.ShowModal;
  finally
    Form.Destroy;
  end;
end;

So to set to true the MyBool field of a TForm descendant i would call:

ShowForm(TMyForm, nil, 1);

And in the CM_INIT handler:

procedure TMyForm.CMInit(var Msg: TLMessage);
begin
  MyBool := (Msg.lParam = 1);
end;

It worked nice for simple variables like a integer or boolean, but things started to look clumsy when i needed to pass a variable of string or TObject type. Furthermore there's the limitation of restricted number of variables and the danger of the need to assume the meaning of the Msg(TLMessage) fields

There's where the RTTI ability to set arbitrary properties came in hand. All i needed to do is publish a property in the TForm descendant and set it through RTTI functions. This way i got type safety, unlimited number of parameters/variables and clearer code.

The new ShowForm interface:

function ShowForm(FormClass: TFormClass; Owner: TWinControl; FormProperties: array of const): TModalResult;

FormProperties is a array of const where the even items are the property names and the odd items, the property values.

What about RTTI? Pretty simple and direct:

//stripped code (no type check, no array of const parsing)
uses typinfo;
[..]
  ClassInfo := Form.ClassInfo;
  PropInfo := GetPropInfo(ClassInfo, PropertyName);
  case PropInfo^.PropType^.Kind of
    tkAString, tkSString:
      SetStrProp(Form, PropInfo, StrPropertyValue);
    tkInteger:
      SetOrdProp(Form, PropInfo, IntPropertyValue);
    tkBool:
      SetOrdProp(Form, PropInfo, Integer(BoolPropertyValue));
  end; 
[..]

Now to set MyBool property of TMyForm i do:

ShowForm(TMyForm, nil, ['MyBool', True]);

A lot clearer no? ;-)

Convert database files from Ansi to UTF-8

While porting old Delphi projects that uses paradox files i faced the problem that the data was stored in an ANSI code page (the current Lazarus version expects UTF-8 encoded strings).

So i wrote an simple application that can convert the encoding of database files (sqlite3, dbf, paradox).

That program can be useful for more than legacy code: after converting a spreadsheet data to a sqlite3 file using Sqlite Data Wizard. The resulted file was in ANSI and there was no option (AFAIK) to convert directly to UTF-8.

I put it here in hope that can be used by someone else.

Using messages to notify events

Lately, i've been using frames extensively in my projects and recently i faced a little problem while working with them: how to notify the parent control (often a TForm) changes in the state of the frame controls/data?

I tried some approaches:

1) Add an event handler to the TDatasource.OnDataChange event.
This has the disadvantage of not being a generic solution since not always the frame has a TDataSource. Also this event is fired in changes to all fields of the respective TDataset, and often only a few fields need to be monitored, leading to some overhead. Yet is not possible to set the event at design time due to bug 14947.

2)Create handlers for events of child controls.
Works by setting in the parent form handlers for events, e.g. OnChange, of child controls of the frame. It has the drawback of needing more than one event handler in the case where is necessary to monitor more than one control/field.
Another problem is that such event handlers can be cleared by the IDE due to bug 14835. Not to say that sometimes the changes in the frame are not propagated to frame instances. In this case the workaround i found to sync the frames was to remove and add the frame again, loosing all properties set in the frame parent control.

So, i decided to implement a specific event for the frame. The first approach i thought was adding a TNotifyEvent property to the frame but this has almost the same drawbacks of approach 2. At this point the idea of using messages came in my mind.

AFAIK messages are used mostly in LCL and seldom in user projects so i had doubts if would work. Here is what i did:

1) Declare a custom message id constant
I declared the following constant in a shared unit:

LM_CHILDDATACHANGED = LM_USER + 1;

2) Create a method in the frame to send the message
I created a method SendDataChangeMsg with the following code:

var
Form: TCustomForm;
begin
Form := GetFirstParentForm(Self);
if Form <> nil then
Form.Perform(LM_CHILDDATACHANGED, 0, 0);
end;

The code is pretty straight. It get the parent form and then send the message through Perform method

3) Hook the events of the controls that should be monitored
Mostly OnChange events. Just called SendDataChangeMsg inside them

4) Add a property ValidData
This property checks if the data in the monitored controls are valid

5) Intercept the message at the frame parent (TForm)
In the TForm i put the frame, i created a method to intercept the message declared as follow:

procedure ChildDataChanged(var Msg: TLMessage); message
LM_CHILDDATACHANGED;

I'm using that to allow saving or not the data so i put a code to enable/disable the save button:

SaveButton.Enabled := Frame.ValidData;

That's all. It's working fine for me. In the end i got a pretty clear notify system. Now i can remove/add the frame as necessary without the need to reconnect the event handlers every time.

Effect of using a constant parameter for string types

Is not rare to find implementations of procedures/functions/methods that uses a value parameter for read only string arguments. While i always use constant parameters for such cases, the real benefit of this code practice was not clear. Until today.

I made a small application that implements two versions of a procedure identical except by the type of parameter (Value vs Constant)...

program asmConstParameter;

{$Mode ObjFpc}
{$H+}

uses
  SysUtils, Types;

procedure DoIt(V: String);
begin
  Writeln(V);
end;

procedure ByValue(V: String);
var
  S: String;
begin
  S := V;
  DoIt(S);
end;

procedure ByReference(const V: String);
var
  S: String;
begin
  S := V;
  DoIt(S);
end;

var
  X: String;

begin
  X := 'Test';
  ByValue(X);
  ByReference(X);
end.

...and examined the assembler output. See the difference yourself. Using optimizations through -O compiler options does not change the produced code.

So using a constant parameter has a practical effect, is not only a good code practice.

BTW: just for curiosity i put {$IMPLICITEXCEPTIONS OFF} in the program header. Not bad. Be aware that this info is here just for curiosity ;-) .

UPDATE: Using constant arguments also benefits ShortString types. See.

Memory layout (and size) of a object

After reading a article about memory layout of objects in Delphi i was curious about how fpc behaves. So i did some small tests:

Memory layout of objects (instances of a class)

At offset 0 resides the virtual method table. Starting from ofsset 4 comes the fields. Just like in Delphi.

Number of associated methods

The number of associated methods and if they are virtual does not influence the object size. Just like in Delphi.

Type of the fields

According to the cited article, Delphi reserves 4 bytes for each field even if the type has a size of 1 byte. Here comes the fun.

Take the following classes:

TOneFlagClass = class
Flag1: Boolean;
end;

TTwoFlagClass = class
Flag1: Boolean;
Flag2: Boolean;
end;

The size of TOneFlagClass and TTwoFlagClass are 5 and 6 bytes respectively (4 for the vmt and 1 for each field). The memory offsets of Flag1 and Flag2 are 4 and 5.

Delphi is a bit different here. The size of both classes are 8. The memory offsets of the fields are the same as fpc.

At this time i think: "In this case is better to place less than 4 bytes fields at the end of the class declaration to avoid subsequent fields to be accessed outside the dword boundary"

I was wrong. In fact half wrong:

Take the following classes:

TFlagFirstClass = class
Flag1: Boolean;
Int1: Integer;
end;

TFlagLastClass = class
Int1: Integer;
Flag1: Boolean;
end;

The size of TFlagFirstClass and TFlagLastClass are 12 and 9 respectively. The compiler allocates 4 bytes for the boolean field to maintain subsequent fields (that has a size of 4 bytes) aligned with the dword boundary.

If another boolean field (Flag2) is added just after Flag1, the instance size is not affected. In fact, grouping 4 boolean (or another 1 byte type) fields together will lead to the same instance size as only one boolean field if those are succeeded by Integer or Pointer like types.

In the end, my suggestion is still valid: put the "less than 4 bytes field types" at the end of the field declaration of the class (or group together in groups with 4 bytes in total). You will save some memory.

If you are not convinced compare size of a class with the following fields sequence:
Boolean, Integer, Boolean, Integer, Boolean, Integer, Boolean, Integer
Boolean, Boolean, Boolean, Boolean, Integer, Integer, Integer, Integer
Integer, Integer, Integer, Integer, Boolean, Boolean, Boolean, Boolean

Some notes:

• Object here is not referenced as the object type (that has the same memory layout of a record), but as the instance of a class
  
• There's no difference between mode delphi and objfpc
• It's valid only for i386 architeture. No idea how this works in ppc, amd64, arm
  

Effect of buffer size in deflate and md5

I tested the effect of buffer size in compressing a file using deflate procedure (paszlib unit) and calculating the md5 (using the functions of md5 unit).

I loaded a 30MB file in memory and did the compression/md5 calculation. The buffer size varied from 1024 to 512.000.

To my surprise no significantly difference was found, so no graph this time since is almost a plain line.

What Time Is It?

How many cairo clocks the world needs?

I don't think we have sufficient!

Effect of buffer size for reading files [Linux]

Reading a file entirely in memory is not a good idea as stated before, but how large should be the memory buffer?

I did a test under Linux (Ubuntu 7.04) reading the fpc2.2.0 installation file (29MB) using different buffer sizes.

Here's the result:


The time to read the file decreases as the buffer size increases until the buffer is 128kb then, as the buffer gets bigger, the trend inverts.

Some notes:

• The test was executed three times for each buffer size. The results are expressed as the Median;
  
• The Y axis is the time to read all the file in microseconds. The X axis is the buffer size in bytes;
• The first time the file is read is significantly slower than subsequent reads. Probably this is an effect of the OS file system buffering (I did not find a way to skip it). This limits further analysis. However, excluding the first run, all other results are consistent across the same buffer size. All results can be browsed here.
  

Update: Using Valgrind/massif with fpc

In a previous post i claimed that the option to hide memory allocation wrappers in massif was broken or not working with fpc.

The problem is that i was using only the pascal name of the function (CMEM_CGETMEM) instead of the mangled internal name (CMEM_CGETMEM$LONGINT$$POINTER).

Thanks to Michalis Kamburelis that gave the hint and also provided this script:

#!/bin/sh
set -eu

valgrind --tool=massif \
--alloc-fn='CMEM_CGETMEM$LONGINT$$POINTER' \
--alloc-fn='CMEM_CREALLOCMEM$POINTER$LONGINT$$POINTER' \
--alloc-fn='SYSTEM_GETMEM$LONGINT$$POINTER' \
--alloc-fn='SYSTEM_GETMEM$POINTER$LONGINT' \
--alloc-fn='SYSTEM_REALLOCMEM$POINTER$LONGINT$$POINTER' \
--format=html \
"$@"

I updated the previous charts so they show where in pascal code the memory is allocated.

PS: it's annoying that the function names are truncated in the charts.

Zlibar memory behavior compressing many small files

The previous analysis of zlibar memory behavior was done taking as example the compression of one big file. Let's with many small files (all *.pas files under lazarus/lcl dir).

Original (load the entire file in memory):
The heaptrc dump:
18573 memory blocks allocated : 496119560/496173048
18573 memory blocks freed : 496119560/496173048
0 unfreed memory blocks : 0
True heap size : 4423680
True free heap : 4423680


After memory optimization (load file in small buffer):
The heaptrc dump:
17446 memory blocks allocated : 439659064/439708048
17446 memory blocks freed : 439659064/439708048
0 unfreed memory blocks : 0
True heap size : 2326528
True free heap : 2326528



We can take some conclusions:

• The memory usage is almost equal over time (the graph scale does not help much here)
  UPDATE: the heaptrc dump shows that the original code really takes more memory.
  
• In the optimized build the memory is allocated in a continuous fashion, always growing. The original build the memory is allocated and freed all over time while still growing in the end. This can lead to more memory fragmentation.
  
• In the optimized build there's not the final peak. This is not really expected since the section of code responsible by the peak was not changed. Some options: 1) the peak exists but valgrind does not detect 2) a bug in the optimized code 3) an unexpected (and good) side effect

Reduce zlibar memory usage - step1

Previously we learned that zlibar uses 1.5MB of memory to compress a 1.1MB file. It compress each file in three passes: 1) loads all the file into memory 2) feed the deflate function with this data using a small buffer as a bridge 3) calculate the md5 signature transversing the file data again.

The option is to compress using only one pass: load the file data incrementally into a memory buffer and than feed deflate and md5 functions with it. It has two advantages: the memory usage (in this step) is constrained by the size of the buffer and you save the memory copy from the stream (that holds the file data) to the deflate buffer and to md5 buffer.

The modified InternalCompressStream function would be something like:

MD5Init(Context);
z.next_in := @input_buffer;
z.avail_in := FileRead(InStream, input_buffer, MAX_IN_BUF_SIZE);
MD5Update(Context, input_buffer, z.avail_in);
while z.avail_in > 0 do
begin
repeat
z.next_out := @output_buffer;
z.avail_out := MAX_OUT_BUF_SIZE;
err := deflate(z, Z_NO_FLUSH);
OutStream.Write(output_buffer, MAX_OUT_BUF_SIZE - z.avail_out);
until Z.avail_out > 0;
z.next_in := @input_buffer;
z.avail_in := FileRead(InStream, input_buffer, MAX_IN_BUF_SIZE);
MD5Update(Context, input_buffer, z.avail_in);
end;
MD5Final(Context, Result.Md5Sum);

Lets run valgrind/massif to see what we got:

Comparing with the previous graph we notice a great memory usage reduction: from 1.5MB to 0.5MB.

The heaptrc dump:
55 memory blocks allocated : 2811961/2812056
55 memory blocks freed : 2811961/2812056
0 unfreed memory blocks : 0
True heap size : 950272
True free heap : 950272

Let's do a deeper analysis:

• The memory used by deflate functions is close to the expected 256kb.
• The pink area is the memory used by the stream that holds the compressed data. It is allocated incrementally so the ascending angle.
• There's a peak after the deflate memory is freed and just before the program finishes. This represents the copy from the compressed stream to the output stream that doubles the data in memory. More on this later.

Someone may say that load a file using a small buffer is slower than reading all data directly in memory. This is true but it would be reasonable only with small files. In a general usage packer it would be a big limitation.