Building a syntax highlighter with regex and python

I originally wrote this up just to organize my own thoughts, but figured it might be worth sharing. So here it is, cleaned up a bit and put online.

Let's go through a simple syntax highlighting program, based on regex patterns and HTML+CSS for some colorful output. The idea is straightforward:

• take a code snippet as input
• break it into tokens
• assign a tag to each token
• and then format the result as HTML

For example, for the snippet x = 37 we get the tokens ['x', ' ', '=', ' ', '37'], and then we produce the HTML:

x <span class="opas">=</span> <span class="nu">37</span>

Here, the tags opas (assignment operator) and nu (number) can be styled with the desired colors. Thanks to such CSS-classes we can customize color-scheme when needed, without re-running the highlighter program.

Approach

Since we’re dealing with source code rather than natural language, we can expect to find some recurring patterns. That’s why regex can actually get us pretty far.

Hopefully, I will get around to a second part of this post, and we'll look into some more advanced Machine Learning based methods. But for now, lets see what we can accomplish with a regex pattern. I'm hoping for:

A logical tokenization

Break the code into pieces, where each token has a single purpose.

Partial sequence tagging

Use regex to capture categories such as numbers, strings, and common operators, without relying on long contexts.

Language agnostic design

By using flexible regex patterns instead of language-specific lookup tables, the same highlighter should work for most programming languages.

Dependencies

For this project, we need the regex package. Compared to Python's built in re, it is more flexible, for instance supporting variable length lookbacks (perhaps at the cost of speed in some cases).

Tag patterns

So, let's look at some things we might be able to tag. I won't go into too many details here, rather focus on the overall approach. To learn more and experiment with regex patterns, check out regex101.com, with an excellent visualizer and explanations.

What is a number?

So the easiest variant would be an integer, simply \d+. To allow decimals, we can do something like \d+\.\d+. Now to catch either of those cases we can use an optional non-capturing group, like \d+(?:\.\d?). This will capture many numbers, like 1337, 0.1, 4.44. In addition, some languages allow separators and letter-suffixes, such as 1_000_000 or '123usize', so lets add [\d_]* in the middle and \w* at the end . Also, let's catch percentages. So, we append %? to our number pattern.

Scientific numbers, such as 1e-9 and 6.022e23, are not that bad either.

Let's also include hexadecimal and binary numbers for good measure. In the end, I settled for these patterns, and they seem to catch all numbers i could think of.

# numbers: scientific
r"(?<!\w)\d+(?:\.\d+)?+e-\d+"
# numbers: hex, bin,
r"(?<!\w)0x[0-9a-fA-F]+|0b[01]+"
# numbers: integer, decimal, percent
r"(?<!\w)\d[\d_]*(?:\.\d+)?\w*%?"

Note the look-back (?<!\w), this prevents accidentally catching numbers inside variable names and identifiers (such as user1 and MyClassV2). Also note the order; for example we need to apply scientific numbers before "regular" numbers, to get 1e-9 as a single number, rather than 2 numbers and a subtraction sign.

Operators

I believe we can capture the most common binary/unary operators with the following patterns (hopefully this doesn't break in some language i don't know):

# common binary operators
r"===|!==|==|!=|<<|>>|\*\*|\/\/|\.\^|\|\||&&|~\/"
# common unary operators
r"\+\+|--"

Note that some operators, like +, - and * are not included, since they could be used in both ways. Of course we could put them in a generic "operator" category, but why not save them for later.

Comments and strings

A simple string is given by something like: \"[^\"\n]*\". Then, we can do similar patterns for stuff like single quotes, back-tics, etc. For strings that can span multiple lines (such as Python's triple-quote strings) we can do "\"{3}[\s\S]*\"{3}". The middle part with [\s\S]* might be a bit ugly, but allows us to match anything, without enabling the dotall flag for the whole combined pattern.

Comments are a little more tricky. The pattern #.*$ is a safe bet for languages like Python and bash. Then \/\/.*$ is quite tempting for C-style comments, but what about // as an operator (integer division in python)? So unfortunately i ended up with some ugly heuristics for comments.

For example, the look-behind (?<=(?:^|[;,])\s*) is useful for many comments, since they typically come after a line start, or a "punctuation", with some whitespace in-between However, it is not fixed width (due to the \s* part), which is fine with the regex module in Python, but incompatible with many other regex engines.

Named groups & a chunk of regex

Now, we have a long list of sub-patterns, lets store them as tuples, with a short name per pattern:

patterns = [
    ("co", r"^(?:\/{2,3}|#|%).+$"),  # one full line comment
    # ...
    ("st", r"\"{3}[\s\S]*\"{3}"),
    ("st", r"'{3}[\s\S]*'{3}"),
    ("st", r"\"[^\"\n]*\""), # basic double-quote string
    # ...
    ("uk", r"\w+|[^\w\s]+?"),  # everything else
]

In total, I ended up with around 35 such tuples. we can then build our final pattern like this:

re.compile("|".join(f"(?P<{t}>{p})" for t, p in patterns), re.M)

By using the form (?P<{t}>{p}) we get named capture groups, giving us tokenization and tagging in a single operation. We also use the "Multi-line" flag (re.M) to make use of ^, $ to match the beginning and end of each line.

Here we process an input string:

tokens: list[str] = []
tags: list[str] = []
# Iterate over all matches
for m in re_token.finditer(text):
    tokens.append(m.group())  # matched token
    t = m.lastgroup # get tag from named group
    tags.append(t)

A note on performance

By building the regex in parts, I'm sure there are some redundant parts, and possible optimizations. However, despite the mess, regex is fast, and the process typically takes under 10 ms on a typical code snippet.

A note on maintainability

Now, since the regex prioritizes each sub-pattern in the union in the order given, we have to be careful when introducing a new rule in the middle. For instance comments and strings need to be early, since they can contain all kinds of other things. For example we want to tag "hello 777" as a single string-token. and the snippet 3.2 # or was it 3.1? should give one number and one comment.

So, to avoid breaking cases that already work we can write some small unit tests. The pytest library, or the standard library unittest work more than well for this. For example, here we test the tokenization for a single variable assignment:

class TestRegex:
    def test_single_stmt(self):
        tk, _ = process_regex("x = 3") # get tokens
        assert tk == ["x", " ", "=", " ", "3"]

Output formatting

HTML

So, the basic idea is to append a bunch of <span class="...">...</span> to a string. But we need to make sure our tokens don't contain any HTML special-characters, so we run them through this:

def html_specials(text: str) -> str:
    """Replace html reserved characters."""

    text = re.sub(r"&", r"&", text)
    text = re.sub(r"<", r"&lt;", text)
    text = re.sub(r">", r"&gt;", text)
    text = re.sub(r'"', r"&quot;", text)
    text = re.sub(r"'", r"&apos;", text)
    return text

Also, to avoid unnecessary markup we can skip the span and the class for the white-space and unknown categories. So our final formatting looks like:

# no need to tag whitespace/unknown tokens
exclude_tags = {"ws", "uk"}

tagged: list[str] = []
for token, tag in zip(tokens, tags):
    # fix html specials
    token_text = html_specials(token)
    if tag in exclude_tags:
        tagged.append(token_text) # only token
    else:
        s = f'<span class="{tag}">{token_text}</span>'
        tagged.append(s) # token inside span

# mush it all together
text = "".join(tagged)

# final code block:
# wrap with <pre> to preserve linebreaks and indentation
final_html = f'\n<pre>{text}</pre>\n'

We also add a head-element to the top of the document, to link the style-sheet.

<head>
    <link rel="stylesheet" type="text/css" href="style.css">
</head>

Nested brackets

There is one more thing we should do. To improve readability, code editors can color brackets by level of nesting. So we want to replace our brop and brcl tags with brX, where X represents the level of nesting. For example, the sequence (()[]) should get tags br0, br1, br1, br1, br1, br0. We can use this function:

def bracket_levels(tags: list[str]) -> tuple[list[str], list[int]]:
    """Rename bracket tags from br_op/cl to br{n}.

    Returns
    -------
    tags_new: list[str]
        modified tags
    """

    current_level = 0
    tags_new = tags.copy() # modify copy of original tags
    for i in range(len(tags_new)):
        if tags_new[i] == "brop":
            tags_new[i] = f"br{current_level}"
            # Increased nesting
            current_level += 1
        elif tags_new[i] == "brcl":
            # Decreased nesting
            current_level -= 1
            tags_new[i] = f"br{current_level}"
    return tags_new

CSS

Last but not least, lets choose some colors for each of our classes (or any other decorations as desired). The CSS might look like this:

body {
  color: white;
  background-color: black;
}
code {
  font-family: "Comic Mono", monospace;
}

.co {
  color: rgb(81, 115, 120);
  font-style: italic;
}
.st {
  color: rgb(194, 188, 108);
}
.nu {
  color: rgb(212, 111, 255);
}
.br0 {
  color: rgb(182, 255, 179);
}
.br1 {
  color: rgb(179, 231, 255);
}
/* and so on ... */

More examples

In addition to the previous examples, let's see what it looks like for a few more languages.

Java:

System.out.println("data: " + data.length + " samples");
System.out.println(Arrays.toString(data));

Rust:

let n = adj.len();

let mut q: VecDeque<usize> = VecDeque::new();
q.push_back(start);

PHP:

// Simulate creating user in past
$this->travel(-$this->lifespan)->years();
$u = new User;
$u->name = 'Old student';

Conclusion

Overall, it works, and while only a few tokens are actually tagged, we might be on track to make something actually useful. If you made it this far, thanks for checking it out, and i hope something was interesting.