Instruments for Apple's Xcode is a tool for performance analysis of an iOS application. In 2018 Apple introduced Custom Instruments — an opportunity to expand the standard set of instruments for application profiling. When existing tools are not enough, you can create new ones yourself — it will collect, analyze and display the data the way you need.

Two years have passed, and there are almost no new published tools and information on the internet. So we decided to rectify the situation and share the experience of our Custom Instrument creation. New Instrument is based on the signpost technology and allows you to determine the reason for the weak isolation of unit tests quickly and accurately.

Theoretical minimum

To create a new tool for Xcode, you need an understanding of two theoretical blocks. For those who want to figure these blocks themselves, we are giving the necessary links right now:

The structure of .instrpkg tool markup file
CLIPS for tool logic programming.

So, below is a summary of the necessary topics.

First select File -> New -> Project -> macOS Category -> Instruments package. The created project includes a file with the extension .instrpkg, in which a new tool is declared in xml format. Let's get acquainted with it’s markup elements:

Element	Attributes	Description
Data schemas	interval-schema, point-schema, etc.	Describes the data structure as a table like sql schemes. Schemes are used in other markup elements to determine the type of data at the input and output of the model, for example, when describing a mapping (UI).
Import data schemas	import-schema	Importing predefined schemes. It allows you to use data structures tdat are defined by Apple.
Tool model	modeler	Associates tde tool witd a .clp file, in which tde logic of tde tool is defined, and announces tde expected data scheme at tde input and output of tde model.
Tool description	instrument	Describes tde data model and determines tde events displaying via UI. tde data model is described using tde attributes create-table, create-parameter, etc. Tool charts are defined by graph attributes, and tde parts table is defined by list, narrative, etc.

If we want to supplement the logic for the created tool, we need to create a .clp file. Basic CLIPS language entities:

“Fact” is a certain event registered in the system using the assert command;
“Rule” is an if-block with specific syntax that contains a condition under which a set of actions is performed.

Which rules and in what sequence them will be activated are determined by the CLIPS itself based on incoming facts, the priorities of the rules and the conflict resolution mechanism.

The language supports the creation of data types based on primitives, logical and arithmetical operations, functions and full-fledged object-oriented programming (OOP) with the definition of classes, message sending, multiple inheritances.

Let's look through the basic syntax of a language that will allow you to create logic for custom tools.

1. To create a fact, use the assert construction:

CLIPS> (assert (duck))

Thus, we get the duck entity in the fact table, which can be viewed using the facts command:

CLIPS> (facts)

To remove the fact, use the retract commandretract: (retract duck)

2. To create a rule, use the defrule construct:

CLIPS> (defrule duck) - create a rule named duck
(animal-is duck) </i> - if animal-is duck exists is in the fact table
=>
(assert (sound-is quack))) - a new fact sound-is quack is created

3. To create and use variables, the following syntax is used (put an obligatory sign "?" before the variable name ):

? <variable-name>

4. You can create new data types using:

CLIPS>
(deftemplate prospect
(slot name (type STRING) (default? DERIVE))
(slot assets (type SYMBOL) (default rich))
(slot age (type NUMBER) (default 80)))

So, we defined a structure with the name prospect with three attributes name, assets, and age with default value and type.

5. Arithmetic and logical operations have prefix syntax. To add 2 and 3, you should use the following construction:

CLIPS> (+ 2 3)

Or to compare two variables x and y:

 CLIPS> (> ?x ?y)

Practical example

In our case with unit-testing, we use the OCMock library to create stub objects. However, there are situations when a mock live longer than the test for which it was created, and affects the isolation of other tests. As a result, this leads to the “blinking” (instability) of unit tests. To track the lifetime of tests and mocks, we will create our tool.

Step 1. Making markup for signpost events

To detect defected mocks, we need two categories of interval events — mock's time of creation and destruction, the start and end time of the test. To get these events, go to the OCMock library and mark them up with signpost in the init and stopMocking methods of the OCClassMockObject class.

Next, go to your project, and do the markup in unit tests, setUp, and tearDown methods:

Step 2. Create a new tool from the Instrument Package template

First, we determine the data type of the input. To do this, we import the signpost scheme in the .instrpkg file. Now events created by signpost will be registered in the tool:

Next, we determine the type of data in the output. In this example, we will output simultaneous events. Each event will have its own time and description. To do this, declare the scheme:

Step 3. Describe the tool’s logic

We create a separate file with the extension .clp, in which we set the rules using the CLIPS language. To let the new tool know in which file the logic was defined, let’s add the modeler block:

In this block, using the production-system attribute we specify the relative path to the file with the logic. In the attributes output and required-input, we define the data schemes for input and output.

Step 4. We describe the specifics of the presentation of the tool (UI)

In the .instrpkg file, we describe the tool and the results displaying. Create a table for the data in the create-table attribute using the previously declared detected-mocks-narrative schema in the schema-ref attribute. And set up the type of information output — narrative (descriptive):

Step 5. We write the logic code

Let's move on to the .clp file, in which the logic of the expert system should be defined. The logic will be as follows: if the start of the test overlaps with the interval of the mock’s life, then we register the fact that this mock “came” from another test — and this violates the isolation of the current unit-test. To eventually create an event with discovered information, you need to do the following steps:

1. Define the mock and unitTest structures with fields — the time of the event, the event’s identifier, the name of the test and the class of the mock.

2. Define the rules that will create facts with the mock and unitTest types, based on the signpost’s incoming events:

These rules can be read as follows: if at the input we get a fact with os-signpost type with the expected subsystem, category, name, and event-type, then we create a new fact with the type that was defined above (unitTest or mock) and fill it with values. It is important to remember — CLIPS is a case-sensitive language and the values of the subsystem, category, name, and event-type must match what was used in the code of the project under study.

Variables from signpost events are passed as follows:

3. We define the rules that release completed events (they are redundant since they do not affect the result).

Step 6. Define the rule that will generate the results

You can read the rule like this.

If
1) there are unitTest and mock;
2) and the beginning of the test occurs after than the existing mock's life-time interval;
3) and there is a table for storing results with the detected-mocks-narrative schema;

then
4) create a new record;
5) fill in with time;
6)… and a description.

As a result, we see the following picture when using the new tool:

The source code for custom instrument and a sample project for using the instrument can be viewed on GitHub.

Tool debugging

The debugger allows you to:

1. See compiled code based on the description in instrpkg.
2. See detailed information about what happens at runtime.

3. Display a complete list and description of system data schemes that can be used as input in new instruments.

4. Execute arbitrary commands in the console. For example, display a list of rules with the list-defrules command or facts with the facts command.

Setup on CI server

You can run tools from the command line to profile the application during the execution of unit- or UI-tests on the CI-server. This will allow, for example, catching a memory leak as early as possible. To profile tests in the pipeline, use the following commands:

1. Running Instruments with attributes:

xcrun instruments -t <template_name> -l <average_duration_ms> -w <device_udid></a>

where template_name is the path to the template with Instruments or the name of the template. You can get the command xcrun instruments -s;
average_duration_ms — recording time in milliseconds, should be greater than/equal to the test run time;
evice_udid — simulator identifier. You can get the command xcrun instruments -s. Must match the identifier of the simulator on which the tests will be run.

2. Running tests on the same simulator with the command:

xcodebuild -workspace <path_to_workspace> -scheme <scheme_with_tests> -destination
<device> test-without-building

where path_to_workspace is the path to the Xcode workspace;
scheme_with_tests — a scheme with tests;
device — simulator identifier.

As a result, a report with the extension .trace will be created in the working directory, which can be opened by the Instruments application or by right-clicking on the file and selecting Show Package Contents.

Conclusion

We examined an example of upgrading signposts to a full-fledged tool and told how to automatically apply it on the “runs” of the CI server and use it in solving the problem of “blinking” (unstable) unit-tests.

As you dive into the possibilities of custom instruments, you will better understand in what other cases you can use it. For example, they also help us to understand the problems of multithreading — where and when to use thread-safe data access.

Creating a new tool was quite simple. But the main thing is that after spending several days of exploring the mechanics and documentation to create it today, you will be able to avoid several sleepless nights in attempts to fix bugs.

Sources

The article was written with regno, Anton Vlasov, an iOS developer

Signpost: When Breakpoints are not Enough