In the previous post of this series we looked at how the New Dynamic Lexer Wizard generated a lexer for us that we could live test immediately. In today’s post, we’ll look more in-depth at the dynamic lexer that was generated.
What is a dynamic lexer?
A dynamic lexer is a lexer that uses patterns to determine how to tokenize text. Patterns can be explicit, which mean the pattern matches the exact characters in the pattern. Patterns can also use regular expressions. This allows for incredible flexibility that you won’t find in other products. And remember that dynamic lexers are only one type of lexer that are supported in SyntaxEditor.
The wizard results in-depth
We use the treeview on the left to navigate the parts of the lexer. In this screenshot, we have selected the Lexical States node.
All of the lexical states defined in the lexer are listed. One or more can be selected and have its properties be edited directly in the property grid on the right. Note that our wizard generated a default state along with states for strings and comments.
The ribbon contains several buttons that may be used for modifying list selections. In the screenshot above we have the mouse cursor over the Move Down button. The Move Up and Move Down buttons allow you to reorder items in a list. Ordering states and patterns properly in dynamic lexers is very important.
Now let’s click the Add button.
The Add Dynamic Lexical States dialog appears. A similar dialog appears when adding other features like lexical macros. In the dialog you enter one state key for each new lexical state you wish to create. Pressing OK adds new states with the specified keys and then you use the property grid to configure them.
Now let’s cancel and click the Child States node of the Default state.
Here we see a list of references to other states that can be transitioned to from the default state. Scopes are used to determine when states transition. For instance, a string state will transition when a double-quote character is seen.
Next, let’s click on Pattern Groups. This is where the meat of a lexer is. Each pattern group defines one or more patterns that can be recognized. The group can be designated as an explicit or regular expression pattern. Whenever a pattern is matched against text, a specified token ID/key are assigned to the token that is created. This token is also assigned a classification type key, which is used to drive syntax highlighting.
Pattern groups also can set their case sensitivity option (or inherit from parent) and can set various look-ahead and look-behind patterns that must be true for a pattern in the group to be matched.
Let’s right-click the Operator pattern group, which opens the Dynamic Lexical Patterns dialog.
This dialog gives us some high-level info about the pattern group being edited and lets us modify all its patterns. You simply enter one pattern on each line and click OK to save.
Next let’s click on the Scopes node of the PrimaryString state.
Here we see the scope for the PrimaryString state. It has a start and end pattern group. The start pattern group says that this state is entered when a double quote is seen. The end pattern group says this state is exited when a double-quote or line feed are seen. This state also has its own other pattern groups we could modify by going to its Pattern Groups node. Those pattern groups define how to tokenize text that is within the two double-quotes.
Next steps
Now that we have looked through what the wizard generated, our next post will focus on doing a project build, which does some really nice robust validation of our language project settings.