There is a story about an experience of using Actor Model in one interesting project of developing an automatic control system for a theatre. Below I'll tell my impressions, no more than that.

Not so much time ago I participated in one exciting task: modernization of automatic control system (ACS) for batten hoists, but in fact it was a development of a new ACS.

A modern theatre (especially if it is a big one) is a very complex organization. There are many people, various mechanisms and systems. One of such systems is ACS for the handling of lifting and setting down the scenery. Modern performances, like operas and ballets, use more and more technical means year over year. The scenery is actively used by show's directors and even play its own important role. It was fascinating to discover what's happening behind the curtains because ordinary spectators can view only actions on the scene.

But this is a technical article, and I want to share my experience of using the Actor Model for writing a control system. And share my impressions of using one of the actor frameworks for C++: SObjectizer.

Why did we choose this framework? We have been looking at it for a long time. There are plenty of articles in Russian, and it has wonderful documentation and a lot of examples. The project looks like a mature one. A brief look at examples has shown that SObjectizer's developers use the same terms (states, timers, events, etc.) and we didn't expect big problems studying and using it. And yet another important factor: SObjectizer's team is helpful and always ready to help us. So we decided to try.

What we're doing?

Let's talk about the target of our project. The system of batten hoists has 62 battens (metal tubes). Each batten is as long as the whole stage. They are suspended on ropes in parallel with gaps of 30-40cm, starting from the front edge of the stage. Every batten can be raised or lowered. Some of them are used in a show for scenery. The scenery is fixed on batten and is moved up/down during the performance. The commands from operators initiate the movement. A system of "engine-rope-counterbalance" is similar to one used in elevators in residential buildings. Engines are placed outside of the stage, so the spectators don't see them. All engines are divided into 8 groups, and each group has 3 frequency converters (FC). At most three engines can be used at the same time in a group, each of them is connected to a separate FC. So we have a system of 62 engines and 24 FCs, and we have to control this system.

Our task was to develop a human-machine interface (HMI) for controlling this system and to implement control algorithms. The system includes three control stations. Two of them are placed just above the stage, and one is in the engine room (this station is used by an electrician on duty). There are also control blocks with controllers in the engine room. These controllers perform control commands, do pulse-width modulation (PWM), turn engines on or off, control the position of battens. Two control stations above the stage have displays, system units, and trackballs as pointing devices. The control stations are connected via Ethernet. Every control station is connected with control blocks by RS485 channel. Both stations above the stage can be used to control the system at the same time, but only one station can be active. The active station is selected by an operator; the second station will be passive; the passive station has its RS485 channel disabled.

Why Actors?

From the algorithms' point of view, the system is built on top of events. Data from sensors, operator's actions, expiration of timers… These all are examples of events. The Actor Model works well for such algorithms: actors handle incoming events and form some outgoing actions depending on their current state. These mechanics are available in SObjectizer just out of the box.

The basic principles for such systems are: actors interact via asynchronous messages, actors have states and switch from one state to another, only messages which are meaningful for the current state are handled.

It's interesting that actors are decoupled from worker threads in SObjectizer. It means that you can implement and debug your actors first and only then decide what worker thread will be used for every actor. There are "Dispatchers" that implement various thread-related policies. For example, there is a dispatcher that provides a separate worker thread for each actor; there is a thread-pool dispatcher that provides a fixed-size pool of worker threads; there is a dispatcher that runs all actors on the same thread.

The presence of dispatchers provides a very flexible way of tuning an actor system for our needs. We can group some actors to work on the same context. We can change the type of dispatcher by just a single line of code. SObjectizer's developers say that writing a custom dispatcher is not a complex task. But there was no need to write our own dispatcher in this project; everything we needed was found in SObjectizer.

Yet another interesting feature is cooperations of actors. A cooperation is a group of actors that can exist if and only if all actors have started successfully. Cooperation cannot be started if at least one of its actors has failed to start. It seems that there is an analogy between SObjectizer's cooperations and pods from Kubernetes, but it also seems that SObjectizer's cooperations have appeared earlier...

When an actor is created it is added to cooperation (cooperation can contain just one actor) and is bound to some dispatcher. It is easy to create cooperations and actors dynamically and SObjectizer's developers say that it is a rather cheap operation.

All actors interact with each other via "message boxes" (mbox). It is yet another interesting and powerful SObjectizer's concept. It provides a flexible way of message processing.

At first, there can be more than one message receiver behind a mbox. It's quite helpful. For example, there can be a mbox that is used by sensors for publishing new data. Actors can create subscriptions for that mbox, and subscribed actors will receive the data they want. This allows working in a "Publish/Subscribe" fashion.

At second, the SObjectizer's developers have envisaged the possibility of custom mbox creation. It is relatively easy to create a custom mbox with special processing of incoming messages (like filtering or spreading between several subscribers based on the message's content).

There is also a personal mbox for every actor and actors can pass a reference to that mbox in messages to other actors (that allows to reply directly to a specific actor).

In our project we split all controlled objects into eight groups (one group for every control box). Three worker threads were created for every group (it is because of only three engines can work at the same time). It allowed us to have independence between groups of engines. It also allowed to work asynchronously with engines inside each group.

It is necessary to mention that SObjectizer-5 has no mechanisms for interprocess or/and network interaction. This is a conscious decision of SObjectizer's developers; they wanted to make SObjectizer as lightweight as possible. Moreover, the transparent support for networking had existed in some previous versions of SObjectizer but was removed. It didn't bother us because a mechanism for the networking is highly dependent on a task, protocols used and other conditions. There is no single universal solution for all cases.

In our case, we used our old library libuniset2 for network- and interprocess communications. As a result, libuniset2 supports communications with sensors and control blocks, and SObjectizer supports actors and interactions between actors inside a single process.

As I said earlier there are 62 engines. Every engine can be connected to a FC (frequency converter); a destination coordinate can be specified for the corresponding batten; the speed of the batten's movement can also be specified. And as an addition to that, every engine has the following states:

• ready to work;
• connected;
• working;
• malfunction;
• connecting (a transition state);
• disconnecting (a transition state);

Each engine is represented in the system by an actor that implements transition between states, handling data from sensors and issuing commands. It's not hard to create an actor in SObjectizer: just inherit your class from so_5::agent_t type. The first argument of actor's constructor should be of type context_t, all other arguments can be defined as a developer wants.

class Drive_A:
    public so_5::agent_t
{
    public:
       Drive_A( context_t ctx, ... );
...
}

I won't show the detailed description of classes and methods because it's not a tutorial. I just want to show how easy it all can be done in SObjectizer (in a few lines literally). Let me remind you that SObjectizer has excellent documentation and many examples.

What is the "state" of an actor? What are we talking about?

Usage of states and transition between them is a "native topic" for control systems. This concept it very good for event handling. This concept is supported in SObjectizer at the API level. States are declared inside the actor's class:

class Drive_A final:
        public so_5::agent_t
{
    public:
       Drive_A( context_t ctx, ... );
       virtual ~Drive_A();
       // состояния
       state_t st_base {this};
       state_t st_disabled{ initial_substate_of{st_base}, "disabled" };
       state_t st_preinit{ substate_of{st_base}, "preinit" };
       state_t st_off{ substate_of{st_base}, "off" };
       state_t st_connecting{ substate_of{st_base}, "connecting" };
       state_t st_disconnecting{ substate_of{st_base}, "disconnecting" };
       state_t st_connected{ substate_of{st_base}, "connected" };
...
}

and then event-handlers are defined for every state. Sometimes it is necessary to do something upon entering or exiting a state. This is also supported in SObjectizer through on_enter/on_exit handlers. It seems that SObjectizer's developers have background in the development of control systems.

Event handlers

An event handler is a place where your application logic is implemented. As I said earlier, a subscription is created for a particular mbox and a specific state. If an actor has no explicitly specified states it is in a special "default_state".

Different handlers can be defined for the same event in different states. If you don't define a handler for some event, then this event will be ignored (an actor won’t know about it).

There is a simple syntax to define event handlers. You specify a method, and there is no need to specify additional types or template parameters. For example:

so_subscribe(drv->so_mbox())
    .in(st_base)
    .event( &Drive_A::on_get_info )
    .event( &Drive_A::on_control )
    .event( &Drive_A::off_control );

It's an example of subscription on events from a specific mbox in the st_base state. It is worth mentioning that st_base is a base state for some other states and that subscription will be inherited by derived states. This approach allows to get rid of copy-and-paste for similar event handlers in different states. But the inherited event handler can be redefined for a particular state or an event can be completely disabled ("suppressed").

Another way to define event handlers is using lambda functions. It's a very convenient way because event handlers often contain just a line or two of code: a send of something to somewhere or a state change:

so_subscribe(drv->so_mbox())
    .in(st_disconnecting)
    .event([this](const msg_disconnected_t& m)
    {
        ...
        st_off.activate();
    })
    .event([this]( const msg_failure_t& m )
    {
        ...
        st_protection.activate();
    });

That syntax looks complex at the beginning, but it becomes familiar just after a couple of days of active coding and you even start to like it. It is because the whole logic of some actor can be concise and placed in one screen. In the example shown above, there are transitions from st_disconnected to st_off or st_protection. This code is easy to read.

BTW, for straightforward cases, where just a state transition is necessary, there is a special syntax:

auto mbox = drv->so_mbox();
st_off
    .just_switch_to<msg_connected_t>(mbox, st_connected)
    .just_switch_to<msg_failure_t>(mbox, st_protection)
    .just_switch_to<msg_on_limit_t>(mbox, st_protection)
    .just_switch_to<msg_on_t>(mbox, st_on);

The Control

How is the control organized? As mentioned above there are two control stations for controlling the movement of battens. Every control station has a display, a pointing device (trackball) and speed setter (and we don't count a computer inside the station and some additional accessories).

There are two control modes: manual and "scenario mode". "Scenario mode" will be discussed later, and now let’s talk about the manual mode. In this mode, an operator selects a batten, prepares it for movement (connects the engine to a FC), sets the target mark for the batten, and when the speed is set above zero, the batten starts to move.

The speed setter is a physical accessory in the form of a "potentiometer with a handle", but there is also a virtual one shown on the station's display. The more it is turned, the higher is the movement speed. The maximum speed is limited at 1.5 meters per second. The speed setter is one for all battens. It means that all selected battens move at the same speed. Battens can move in opposite directions (it depends on the operator's selection). It's apparent that it is hard for a human to control more than a few battens. Because of that only small groups of battens are handled in the manual mode. Operators can control battens from two control stations at the same time. So there is a separate speed setter for each station.

From the implementation's point of view, there is no specific logic in the manual mode. A "connect engine" command goes from the graphical interface, is transformed into a corresponding message to an actor, and then is being handled by that actor. The actor goes from "off" state to "connecting", and then to "connected" state. Similar things happen with commands for positioning a batten and setting the movement speed. All these commands are passed to an actor in the form of messages. But it is worth mentioning that "graphical interface" and "control process" are separate processes and libuniset2 is used for IPC.

The Scenario mode (are there Actors again?)

In practice, the manual mode is used only for very simple cases or during rehearsals. The main control mode is "scenario mode". In that mode, every batten is moved to a specific position with particular speed according to scenario settings. Two simple commands are available to an operator in that mode:

• prepare (a group of engines is being connected to FC);
• go (movement of the group starts).

The whole scenario is split into "agendas". An "agenda" describes a single movement of a group of battens. It means that an "agenda" includes some battens and contains target destinations and speeds for them. In reality, a scenario consists of acts, acts consist of pictures, picture consist of agendas, and agenda consist of targets for battens. But from the control's point of view, that doesn't matter, because only agendas contain the precise parameters of the batten's movement.

The Actor Model fits that case perfectly. We have developed a "scenario player" that spawns a group of special actors and starts them. We have developed two types of actors: executor actors (they control the movement of battens) and coordinator actors (they distribute tasks between executors). Executors are created on demand: when there are no free executors, a new executor will be created. Coordinator manages the pool of available executors. As a result, the control roughly looks like this:

• an operator loads a scenario;
• "scrolls" it until the required agenda;
• presses the "prepare" button at the appropriate time. At that moment a message is sent to a coordinator. This message contains data for every batten from the agenda;
• the coordinator reviews its pool of executors and distributes tasks between free executors (new executors are created, if needed);
• each executor receives a task and performs preparation actions (connects an engine to a FC, then waits for "go" command);
• the operator presses the "go" button at the appropriate moment;
• the "go" command goes to the coordinator, and it distributes the command between all executors that are currently in use.

There are some additional parameters in agendas. Like "start the movement only after N seconds delay" or "start the movement only after an additional command from an operator". Because of that, the list of states for an executor is quite long: "ready for the next command", "ready for movement", "delay of the movement", "awaiting operator command", "moving", "completed", "failure".

When a batten has successfully reached the target mark (or there is a failure) the executor reports to the coordinator about task completion. Coordinator replies with a command to turn the engine off (if the batten does not participate in the agenda anymore) or sends a new task to the executor. The executor either turns the engine off and switches to "waiting" state or starts processing the new command.

Because SObjectizer has a quite thoughtful and convenient API for working with states, the implementation code turned out to be quite concise. For example, a delay before movement is described by just a line of code:

st_delay.time_limit( std::chrono::milliseconds{target->delay()}, st_moving );
st_delay.activate();
...

The time_limit method specifies the amount of time to stay in the state and which state should be activated then (st_moving in that example).

Protection actors

Certainly, failures can occur. There are requirements to correctly handle these failures. Actors are used for such tasks too. Let's look at some examples:

• overcurrent protection;
• protection from sensor's malfunction;
• protection from movement in the opposite direction (it can happen if there is something wrong with sensors or actuators);
• protection from spontaneous movement (without a command);
• command execution control (the movement of a batten should be checked).

We can see that all those case are self-sufficient, but they should be controlled together, at the same time. It means that any failure can happen. But every check has its logic: sometimes it is necessary to check a timeout, sometimes it is required to analyze some previous values from a sensor. Because of that, protection is implemented in the form of small actors. These actors are added to cooperation to the main actor that implements control logic. This approach allows for easy addition of new protection cases: just add yet another protector actor to the cooperation. The code of such actor is usually concise and easy to understand, because it implements only one function.

Protector actors also have several states. Usually, they are turned on when an engine is turned on or when a batten starts its movement. When a protector detects a failure/malfunction it publishes a notification (with protection code and some additional details inside). The main actor reacts to that notification and performs necessary actions (like turning the engine off and switching to protected state).

As the conclusion...

...this article is not a breakthrough of course. The Actor Model is being used in multiple different systems for a quite long time. But it was my first experience of using the Actor Model for building an automatic control system in a rather small project. And this experience turned out to be quite successful. I hope I’ve shown that actors are a good fit for control algorithms: there are places for actors literally everywhere.

We had implemented something similar in previous projects (I mean states, message exchange, management of worker threads, and so on), but it wasn't a unified approach. By using SObjectizer we got a small, lightweight tool that solves a lot of problems. We no longer need to (explicitly) use low-level synchronization mechanisms (like mutexes), there is no manual thread management, no more handwritten statecharts. All these are provided by the framework, logically connected and expressed in the form of convenient API, but you don't lose the control on details. So it was an exciting experience. If you are still in doubt, then I recommend you to take a look at the Actor Model and SObjectizer in particular. It leaves positive emotions.

The Actor Model really works! Especially in the theatre.

Original article in Russian