In this series of articles, we explore the capabilities of the AutoHotkey language by demonstrating how to output colored text to the terminal for the Launcher console utility.

Part 1: Naive algorithm and its optimization
Part 2: Control sequences
Part 3: Atomic Regular Expressions (you are here)
Part 4: Declarative Programming
Part 5: 256 Colors and Text Styling

In the previous part, we learned about control sequences and how to apply them in our algorithm to change the color of text. In this article, we will explore some of the capabilities of regular expressions that will allow us to change the color of an entire text region.

If you are not familiar with regular expressions at a basic level, I recommend reading this article. If you are not familiar with the AutoHotkey syntax, I recommend reading the first part. However, this is not necessary to understand this article ;)

Constructing Expressions

Paired Characters

In the previous part, we wrote an algorithm for processing character pairs: " " or * *. We pass the intermediate array aRegexColor consisting of “character-color” pairs, convert it into a HashMap chrColors containing “character-code” pairs, and then search for each pair and add the corresponding character from chrColors before the opening character:

Color(msg, aRegexColor) {
    static colors := Map(
        'black',    30,
        'red',      31,
        'orange',   33,
        'magenta',  35,
        'gray',     90,
        'crimson',  91,
        'green',    92,
        'yellow',   93,
        'blue',     94,
        'purple',   95,
        'cyan',     96,
    )
    
    static esc := Chr(27)
    static end := esc '[0m'
    
    regex     := ''		 ; final regular expression
    chars     := ''		 ; character string
    chrColors := Map()
    
    index := 1
    loop (aRegexColor.length / 2) {
        ; `aRegexColor` - input array of "char-color" pairs
        str   := aRegexColor[index++]
        color := aRegexColor[index++]
    
        chars .= str
        chrColors[str] := colors[color]  ; `colors` - HashMap of "color-code" pairs
    }
    
    if chars
        regex .= '([' chars '])'
    else
        regex := regex.RTrim('|')
        
    stack := []
    while (pos <= len) {
        if !RegExMatch(msg, regex, &match, pos) {
            ; Remaining text
            clrMsg .= msg.Slice(pos)
            break
        }
        
        ; Normal text before the found character
        clrMsg .= msg.Slice(pos, match.pos - pos)
        ; Move forward
        pos := match.pos + match.len
        
        if (stack.Has(-1) && stack[-1] = match[1]) {
            clrMsg .= match[1] . end
            stack.Pop()
        } else {
            begin  := esc '[0;' chrColors[match[1]] 'm'
            clrMsg .= begin . match[1]
            stack.Push(match[1])
        }
    }
    
    return clrMsg
}

RegExMatch searches for characters from the set (e.g., ["*]) and stores one of the found characters in match[1]. As a result, the algorithm processes one character per iteration. This allows us to correctly highlight text inside paired characters: "text and text". It also highlights text inside that text: "text *and* text".

However, we have a more complex help message in which we need to highlight specific constructs: --switches, @file, and variables like %AhkDir%. We need to improve the algorithm and add the ability to work with entire expressions, rather than just individual characters.

Key-Value

In the algorithm above, we passed a “character-color” array, where the key is a character and the value is a color code:

msg.Color([ 
  '"',  'cyan',
  '*',  'green'
])

This array was converted to chrColors:

" 96
* 92

As a result, we could call chrColors['"'] or chrColors['*'], where each character served simultaneously as a search pattern (the regular expression, the regex variable), the matched character (the match[1]), and a key in chrColors.

Now we’re going to pass “expression-color” pairs (for simplicity, let’s temporarily forget about characters):

msg.Color([
  '(@(file|list\.ini))',    'yellow',   ; list
  '(mainDir|AhkDir)(?=\=)', 'purple',   ; variables names
  '(%[^%]+%)',              'blue',     ; variables values
  '(\-+[\-\w]+)(?=[ =])',   'cyan',     ; switches
])

With this approach, each expression can no longer serve as both a search pattern and a matched expression (text) at the same time, and therefore cannot be a key in chrColors.

Every regular expression we pass to RegExMatch is plain text. On its own, it doesn’t search for anything; instead, it is first compiled into a set of instructions for the processor. In other words, Regular Expressions is something that we can call a low-level language that can be compiled, where every character, such as \w, is converted into a specific instruction.

In our case, this means that an expression like (%[^%]+%) is converted into code that searches for a text like %AhkDir%, %mainDir%, and so on. In other words, the input ((%[^%]+%)) is not equal to the output (%mainDir%), and as a result, information about the original regular expression itself - which allowed us to apply the desired color to the found text - is lost. Consequently, the expression itself cannot serve as a key in chrColors to retrieve the color.

Mark

As you may have noticed earlier, the match variable is somewhat similar to an array: match[0] stores everything that was found, and match[1] stores what capturing group 1 found. However, it is actually a match object, and it stores not only indices but also fields: match.pos, match.len, match.count. It also has a mark field. When RegExMatch() encounters the (*MARK:..) construct in the pattern, the mark field takes on the value following the colon : For example, after (*MARK:chr), the value of match.mark will be "chr".

Let’s look at another expression:

(%[^%]+%)(*MARK:1)|(~[^~]+~)(*MARK:2)

If group 1 is captured, match.mark will be equal to “1”. If group 2 is captured, match.mark will be equal to “2”.

We can use this feature to mark (index) the patterns: 1, 2, 3, … Each pattern will have a unique mark, a unique key that we’ll use to retrieve the color from chrColors:

begin  := esc '[0;' chrColors[match.mark] 'm'
clrMsg .= begin . match[1]

To do this, we need to add mark with the corresponding index to the end of each pattern in the aRegexColor array, and then combine all the patterns into a single regular expression using the OR operator |

	regex     := '' 	; final regular expression
    chars     := ''     ; separate expression with chars
    chrColors := Map()
    
    index     := 1	; array index
    idxColor  := 1  ; mark index
    
    loop (aRegexColor.length / 2) {
        str   := aRegexColor[index++]
        color := aRegexColor[index++]

        if (str.length = 1) {
            ; Characters are grouped into set like ["*"]
            chars .= str
            chrColors[str] := colors[color]		; char-code
        } else {
            ; Group patterns into one
            regex .= str '(*MARK:' idxColor ')|' 
            chrColors[String(idxColor++)] := colors[color]  ; index-code
        }
    }
    
    if chars
        regex .= '([' chars '])(*MARK:chr)'  ; Add the characters separately at the end
    else
        regex := regex.RTrim('|')			 ; Remove the extra OR |

The result will be a single large expression like (..)(*MARK:1)|(..)(*MARK:2)|...|([..])(*MARK:chr). For convenience, we’ll use the same HashMap that contains “index-code” pairs and the “character-code” pairs we’re already familiar with:

  1. Expressions: the key is match.mark (index).

  2. Characters: the key is the character itself, match[1].

Since each expression (whether a character or an indexed expression) has a different key, we need to distinguish between indexed expressions and character expressions using (*MARK:chr):

        if (match.mark != 'chr') {
            ; Indexed expression: wrap in color
            begin  := esc '[0;' chrColors[match.mark] 'm'
            clrMsg .= begin . match[1] . end
            continue
        }

        ; Character expression
        if (stack.Has(-1) && stack[-1] = match[1]) {
            clrMsg .= end
            stack.Pop()
        } else {
            begin  := esc '[0;' chrColors[match[1]] 'm'
            clrMsg .= begin
            stack.Push(match[1])
        }

Branch Reset

The constructed expression has a critical flaw: it has many capturing groups. As a result, each captured text fragment will be associated with its own capturing group: 1, 2, …

Let’s look at this code snippet:

"%variable% is ~gray~".Color([
    '(%[^%]+%)', 'blue', 
    '(~[^~]+~)', 'gray'
])

Color will construct the expression:

(%[^%]+%)(*MARK:1)|(~[^~]+~)(*MARK:2)

which contains 2 capturing groups:

Index

Group

Text

1

(%[^%]+%)

%variable%

2

(~[^~]+~)

~gray~

Accordingly, the text ~gray~ will be stored in match[2], while match[1] (which we use in the algorithm) will be empty. In other words, we need to know the group’s index to retrieve the text. But if the user has included multiple groups in each pattern, such as =(%([^@](\w+))%), it will be difficult to determine the index. We want the text or character to always be stored in match[1].

To solve this problem, we can use Branch Reset. Branch Reset is a (?|..) construct, which makes the index of each group relative. If we represent the expression ()|()|... as a list of items separated by |, then the index of each capturing group within each item will be relative to that item:

( () )|( () () )
1  2   1  2  3

In other words, we don’t look at the entire expression, but rather to its individual items, and then at groups within those items. Without Branch Reset, indexing is absolute:

( () )|( () () )
1  2   3  4  5

Let’s wrap the constructed expression in (?|..):

(?|(%[^%]+%)(*MARK:1)|(~[^~]+~)(*MARK:2))

It is important to understand that (?|..) is not a capturing group, but a “Branch Reset” construct. And (*MARK:..) is a “verb” (verb). The parentheses are merely part of the syntax.

As a result, we end up with the following indexing:

Index

Mark

Group

Text

1

1

(%[^%]+%)

%variable%

1

2

(~[^~]+~)

~gray~

match[1] will always contain the captured text from the 1st, 2nd, or 3rd group. At the same time, match.mark will still store the correct index:

    if chars
        regex .= '([' chars '])(*MARK:chr)'
    else
        regex := regex.RTrim('|')

    regex := '(?|' regex ')'

Color Switching

Currently, our algorithm simply wraps the found text fragment in the opening and closing colors:

    	 static esc := Chr(27)
    	 static end := esc '[0m'
         if (match.mark != 'chr') {
            ; Indexed expression: wrap in color
            begin  := esc '[0;' chrColors[match.mark] 'm'
            clrMsg .= begin . match[1] . end
            continue
        }

However, as you may remember from the previous part, control sequences “toggle” colors:

\e[96m"  <- own color
	# level 1
	text 
    \e[92m*   <- own color
   		# level 2
    	and 
   *\e[96m    <- previous color
   text
"\e[0m   <- no previous color

The closing color of each pair of characters depends on the nesting level of that pair. That is why we use a stack to keep track of which color to apply: the opening color (the begin variable) or the closing color \e[0m (the end variable).

The closing color does not depend on anything in the current algorithm. The text region is simply wrapped in color, and then the color is reset to white using \e[0m.

However, it would be more correct to check whether we are inside a pair of characters (e.g., " ") and switch the color to the previous one, opened by the paired character:

        if (match.mark != 'chr') {
            ; We take the `begin` color from chrColors
            begin  := esc '[0;' chrColors[match.mark] 'm'
            if (stack.Has(-1)) {
                ; Within a pair, we look for the closing color 
                ; using the "key" from the stack
                _end := esc '[0;' chrColors[stack[-1]] 'm'
                clrMsg .= begin . match[1] . _end
            } else {
                ; Independent - close the color
                clrMsg .= begin . match[1] . end
            }
            continue
        }

As a result, the algorithm, whose source code is available on GitHub, looks like this:

; Define add. method for readability
({}.DefineProp)(String.prototype, 'Match', {call: RegExMatch})

Color(msg, aRegexColor) {
	; The color name corresponds to the color code
    static colors := Map(
        'black',    30,
        'red',      31,
        'orange',   33,
        'magenta',  35,
        'gray',     90,
        'crimson',  91,
        'green',    92,
        'yellow',   93,
        'blue',     94,
        'purple',   95,
        'cyan',     96,
    )
    
    ; Constants
    static esc := Chr(27)
    static end := esc '[0m'

	; Transform the input array
    regex     := ''
    chars     := ''
    chrColors := Map()
    chrColors.capacity := aRegexColor.capacity
    
    index     := 1    ; array index
    idxColor  := 1    ; mark index
    
    loop (aRegexColor.length / 2) {
        str   := aRegexColor[index++]
        color := aRegexColor[index++]

        if (str.length = 1) {
            ; Group chars into a set [..]
            chars .= str
            chrColors[str] := colors[color]    ; char-code
        } else {
            ; Group atomic patterns using OR |
            regex .= str '(*MARK:' idxColor ')|' 
            chrColors[String(idxColor++)] := colors[color]    ; index-code
        }
    }
    
    if chars
        regex .= '([' chars '])(*MARK:chr)'
    else
        regex := regex.RTrim('|')
        
    ; Apply Branch Reset
    regex := options '(?|' regex ')'
        
    ; Color the message
    pos := 1
    len := msg.length
    clrMsg := ''
    
    stack := []
    stack.capacity := aRegexColor.capacity * 2

    while (pos <= len) {
        if !msg.Match(regex, &match, pos) {
            ; Remaining text
            clrMsg .= msg.Slice(pos)
            break
        }
        
        ; Regular text before the match
        clrMsg .= msg.Slice(pos, match.pos - pos)
        ; Move forward
        pos    := match.pos + match.len
        
        if (match.mark != 'chr') {
            ; Atomic pattern.
            ; Take the `begin` color from `chrColors`
            begin  := esc '[0;' chrColors[match.mark] 'm'
            if (stack.Has(-1)) {
                ; Within a pair, we look for the closing color 
                ; using the "key" from the stack
                _end := esc '[0;' chrColors[stack[-1]] 'm'
                clrMsg .= begin . match[1] . _end
            } else {
                ; Independent - close the color
                clrMsg .= begin . match[1] . end
            }
            continue
        }

        ; Paired char
        if (stack.Has(-1) && stack[-1] = match[1]) {
            ; Found the char that completes the pair. 
            ; Close the color
            clrMsg .= end
            stack.Pop()
        } else {
            ; Open matching color
            begin  := esc '[0;' chrColors[match[1]] 'm'
            clrMsg .= begin
            stack.Push(match[1])
        }
    }
    
    return clrMsg
}

In the first part, we began with the help message. Now we can color it completely:

PrintHelp(*) {
	; ...
    msg := 
      msg.Color([
        '(@(file|list\.ini))',    'yellow',   ; list
        '(mainDir|AhkDir)(?=\=)', 'purple',   ; variables names
        '(%[^%]+%)',              'blue',     ; variables values
        '(\-+[\-\w]+)(?=[ =])',   'cyan',     ; switches
        '\*\*([^\*]+)\*\*',       'crimson',  
        '__([^_]+)__',            'magenta',  
        '~',                      'gray',
        '``',                     'green', 
        '#',                      'orange'
      ]).Print()
}

Conclusion

Regular Expressions are very convenient tool to parse text. They are not so readable, but they allow us to find text regions using low-level code and process them using the high-level AutoHotkey language.

In this article, we saw that regular expressions have the ability to track the path and they have some reflection. Thanks to these capabilities, we were able to use them more effectively and write a concise and readable algorithm.

In the next article, we’ll explore how to describe text parsing in human-readable language using AutoHotkey and examine the language from the perspective of declarative programming. Check out my GitHub if you want to see the AutoHotkey capabilities in action!