In this series of articles, we explore the capabilities of the AutoHotkey language by demonstrating how to output colored text to the terminal for the Launcher console utility.

Part 1: Naive algorithm and its optimization
Part 2: Control sequences (you are here)
Part 3: Regular expressions
Part 4: Text styling and coloring library

In the previous part, we rewrote the algorithm from recursive to iterative and sped up its execution by almost 22 times. However, the final code lost support for nested colors. In this part, we will try to understand why this happened, what ANSI codes are, and how they are processed by the terminal.

Control sequences

Terminals

Today, a terminal is synonymous with a terminal emulator (e.g. Cmder, Windows Terminal, Ghostty, and many others), but once upon a time, terminals were quite simple devices: screen and keyboard for communicating with a remote server. These terminals received bytes of data but had no way to distinguish between different types of data received. There was no way to control the behavior of the terminal.

If you search for “ANSI codes” from the previous part, the first result you will see is the ANSI website (American National Standards Institute). The term "ANSI codes" does not exist, only the institute. ANSI created a standard for escaping characters in terminals, which was later replaced by the ISO 6429 standard. ANSI also reserved some bytes for manufacturer-specific escape codes.

Nowadays, these codes vary across different operating systems and terminal emulators. The emulator itself is responsible for interpreting ANSI. For example, in Cmder you can change the colors and termimnal behavior when it encounters ANSI.

In PowerShell 3.0+ (which is emulated by pwsh, Cmder, and Windows Terminal), the PSReadLine module is responsible for ANSI and interactive input. Since version 5.0, it allows changing text colors using $([char]0x1b)[91m (we'll explain what this means later), and since version 6.0, a more convenient syntax is available via $PSStyle:

$PSStyle.Formatting.Verbose = $PSStyle.Foreground.Cyan
$PSStyle.Formatting.Debug   = $PSStyle.Foreground.DarkGray

In Windows, there is a mechanism called Console Virtual Terminal Sequences at the console level. Starting with Windows 10, the console supports processing of VT100 sequences, which allows PowerShell to correctly interpret ANSI codes from the previous part. Thanks to this mechanism, we can output identically styled text from AutoHotkey to both cmd.exe and pwsh.exe (or any other emulator).

Escaped sequence

To tell the terminal that we want to use ANSI codes for text styling, we must use an escaped sequence. For this purpose, there is a special escape character. In Unix-like systems, the escape character is usually \e or \033. In PowerShell 6.0+ and AutoHotkey 2.0+, the escape character is `e. In AutoHotkey, we used esc := Chr(27) for reliability.

In PowerShell 5.1, you must use $([char]0x1B) or [char]27 instead of `e.

In this article, we will use the common notation \e.

Escaped sequences are mixed in with the main text when you type them. However, they disappear during output, since they provide information to the terminal, not to the reader of the message. Therefore, it is important to remember that the length of a message containing escaped sequence is always greater than the length of the output message!

The original message at the top contains more characters than the output message at the bottom

Control sequence

A control sequence begins with the combination of characters \e and [ - this sequence (combination) is called CSI (Control Sequence Introducer). It is followed by several bytes (remember that 0 is also a number). Each sequence has its own set of rules and conventions about what it does and what arguments it expects. Within the scope of this article, we will consider only one subset of control sequences - changing text color and style. They have the form CSI n1 [;n2 [; ...]] m, where m is the SGR function (Select Graphic Rendition), and each n1, n2, … is an SGR parameter. We are working with single-byte set (\e[n1;n2m...), so n1 is a number from 0 to 7 (text or cursor style); n1 is a number from 0 to 107 (color).

In the early ANSI standard, only 8 colors were available for changing text and background color. The n2 values 30-37 for the foreground color, and 40-47 for the background color. Later, high-intensity color codes appeared: 90-97 (foreground) and 100-107 (background). This set is called “single-byte” and it is what we used in the previous part.

The control sequence also disappears from the output message, since it is a form of escaped sequence.

256 colors

If 8 colors are not enough (which is quite possible), we can use a longer sequence for a palette of 256 colors. The ANSI sequence for using 8-bit color looks as follows: \e[<text | background code>;5;n, where the first code is 38 (text color) or 48 (background color); n is a color from 0 to 255. For example, the sequence \e[38;5;220m changes the text color to dark yellow. The range of 256 colors is divided into several sections: 0-7 are standard colors, 8-15 are bright colors, 16-231 is a 6×6×6 color cube, and 232-255 are grayscale shades.

Various examples of sequences can be found here.

Sequence reset

If the reset combination of the sequence is not specified, it will be applied to all subsequent text by our source code to the screen (doesn't matter if it's PowerShell script or AutoHotkey script). When running the code snippet below, you will notice that the text on the new line is still bold and inverted. Additionally, the first line break character in the prompt has changed.

Formatting affects the entire prompt and it doesn't updates

To reset the sequence, it is sufficient to add \e[0m at the end. This combination resets the previously applied style, color, and so on.

An alternative option is to reset each code separately. For example, the sequence \e[1;7m will make the font bold and invert the foreground and background colors. To reset inversion only, you should add \e[27m. The text will remain bold but not inverted. To reset the bold text, you need to add \e[22m. The text would no longer be inverted or bold.

Control Sequences

Now that we know that we need to add \e[n1;n2m at the beginning to change foreground color (e.g. \e[0;96m) and \e[0m at the end, it becomes obvious that the message consists of several control sequences:

\e[0;96m message \e[0m

Spaces are added for readability. Henceforth in the article, we will use the phrase control sequences, since \e[0;96m is one sequence; \e[0m is also one sequence; and the plural form indicates the presence of several such sequences in the message.

We have figured out what ANSI codes from the previous part actually are, so we can call them properly: control sequences.

Solution for Nested Colors

State Machine

Let's return to our console output algorithm and understand why nested colors are not working, i.e. changing the text (foreground) color inside already colored text.

Control sequences operate on the principle of a finite state machine with a single set of current attributes. The terminal stores exactly one set of active graphic properties in memory: text color, background color, and style (italic, underline, etc.). Each new sequence overwrites the corresponding attributes rather than pushing them onto a stack.

Suppose we have a message that we want to color as follows:

<cyan>text<green>and</green>text</cyan>

HTML tags are provided for readability and understanding of colors. We need to wrap the text in control sequences:

"text and text".Color(".+", "cyan").Color("and", "green")

The Color() method will apply control sequences in the following order:

\e[0;96m text \e[0;92m and \e[0m text \e[0m

Let’s break down step by step how the terminal interprets this:

Step	Sequence	Terminal state after application
1	`\e[0;96m`	Reset all attributes, then set the text color to cyan (96)
2	Output `text`	Cyan text color
3	`\e[0;92m`	Reset all attributes. Change the text color to green (92)
4	Output `and`	Green text color
5	`\e[0m`	Reset all attributes
6	Output `text`	Default color
7	`\e[0m`	Reset all attributes (redundant, the state has already been reset)

On the third step a full reset occurs. The terminal does not remember that the color was cyan before. Due to the lack of a stack, this information is lost. Unlike HTML/CSS, where each element has its own set of styles and the browser calculates the cascade, the terminal works with a global state.

Thus, the correct sequence looks like this:

\e[0;96m text \e[0;92m and \e[0;96m text \e[0m

You can test control sequences on the ansi101 and ansi.tools websites.

Stack

In previous part we developed our own syntax that tells Color() where to add colors. Let’s define pairs of characters that will change the text color to cyan and green. Let these be " " and * * respectively:

"text *and* text"

The idea of the correct algorithm is simple. Before each character, it is necessary to add some control sequence (color). If we are inside a pair of characters (e.g., " ") and have not found any new nested pair (* *), we simply add the opening and closing sequence before the opening quote and after the closing quote, respectively:

\e[96m"	# level 1	text and text
"\e[0m

For each nested pair * * found inside " " we switch the color: open a new one for the opening and switch to the previous one for ":

\e[96m  <- own color
"  
	; #1
	text 
    \e[92m   <- own color
    *    
   	  ; #2
      and 
    *
    \e[96m   <- previous color
    text
"
\e[0m   <- no color

This means that the closing color of each pair (the nesting problem we are trying to solve) depends on the nesting level of that pair: each subsequent level is closed with the color of the pair at the level above. And so on until the first top-level pair found that is not nested:

Level	Char	Color
1	"	Cyan
2	*	Green
2	*	Cyan
1	"	0

For a top-level pair, there is no previous level, so it can be closed with \e[0m or its own color \e[96m. This entire table can be simplified to a simple stack that only stores the found characters:

"
*
*
"

We can get the previous character from the top of the stack and switch to it’s color, that can be found in chrColors - a HashMap with "char-code" pairs:

" 96
* 92

The algorithm looks like this:

stack := []
while (pos <= len) {
    if !RegExMatch(msg, regex, &match, pos) {
        ; Remaining text
        clrMsg .= msg.Slice(pos)
        break
    }
    
    ; Normal text before the match
    clrMsg .= msg.Slice(pos, match.pos - pos)

    if (stack.Has(-1) && stack[-1] = match[1]) {
        clrMsg .= match[1] . end
        stack.Pop()
    } else {
        begin  := esc '[0;' chrColors[match[1]] 'm'
        clrMsg .= begin . match[1]
        stack.Push(match[1])
    }
    
    ; Move position forward
    pos := match.pos + match.len
}

You can read what is RegExMatch here. Here is a algorithm visualization. Created using staying.fun

On the ansi101, you can see how the final clrMsg message looks like.

Char-color pairs

Obviously, it is inconvenient to pass a HashMap with numeric codes to the Color() function. We want to pass "char-color" pairs:

msg.Color([ 
  '"',  'cyan',
  '*',  'green'
])

In this case, we pass an array instead of a HashMap for two reasons:

The order is important to us, as it determines the priority of each color.
This is an intermediate container that is only needed for iteration.

The array is intermediate because we convert it into a HashMap chrColors with "char-code" pairs that we can use later:

regex     := ''		 ; resulted regular expression
chars     := ''		 ; string of characters
chrColors := Map()	 ; there is no shortland for HashMap
 
index := 1
loop (aRegexColor.length / 2) {
	; `aRegexColor` - input array with "char-color" pairs
    str   := aRegexColor[index++]
    color := aRegexColor[index++]

    chars .= str
    chrColors[str] := colors[color]  ; `colors` - HashMap with "color-code" pairs
}

if chars
    regex .= '([' chars '])'
else
    regex := regex.RTrim('|')

For optimization, we combine all the passed characters into a RegEx set ["*] for better parsing performance. Then each character will be the key to extract the color from chrColors.

The source code of the text output algorithm is available on GitHub. And here you can read about some additional features of this algorithm.

Conclusion

Control sequences are a legacy from the days when terminals were physical machines rather than programs. Almost every text formatting library, such as {fmt}, adds these invisible characters to a message. Therefore, to work with text colors and styles, we need to understand them.

In this article, we've seen that it's possible to read control sequences and understand their meaning; we learned how the terminal interprets them, so we can "ask" it to do exactly what we need.

In the next article, we will add support for atomic expressions and explore some interesting capabilities of regular expressions. Check out my GitHub if you want to see the AutoHotkey capabilities in action!

Control Sequences (ANSI)