In the previous post, we added error handling to our Simple language grammar. This allowed the grammar parser to recover from any invalid syntax in a document being parsed, and to still produce an AST.
It concluded the three basic stages in building a grammar:
- Configure terminals, non-terminals, and EBNF productions
- Add customized tree constructors to productions that create a concise AST
- Add error handling to recover from invalid syntax
Now what happens when in the middle of the grammar design and something is not parsing as intended? It can be tricky to know what the parser is “thinking” since it’s a bit of a black box.
To solve this issue, we’ve built complete debugging capabilities directly into the parser and have even enhanced the Language Designer application to have a complete debugger UI!
Debugging our XML language grammar
As mentioned in some previous posts, we’ve been working on a port of our WinForms Web Languages Add-on for WPF. We hope to get it into the WPF Studio 2010.2 release or shortly after. We’ve been testing the grammar/AST framework with its advanced XML language implementation.
Here’s how the Language Designer looks once the LL Parser Debugger is opened and the assembly containing the XML parser is selected:
The middle area shows a textual EBNF representation of how our framework interprets the grammar that we created for our XML language. Above that is a ComboBox containing the list of non-terminals in the grammar.
You can select a non-terminal to automatically jump to it in the EBNF representation. Likewise, as you move the caret around in the EBNF representation, the ComboBox updates its selection appropriately.
The bottom selected tab shows the input we’ll use to debug.
Before we begin debugging, let’s also examine the Terminals tab. It shows all the terminals in the grammar, their related token ID / key values, and error aliases.
Now let’s start debugging by pressing the Step Into button (or hitting F11):
In the above screenshot we’ve stepped into a number of constructs already. The statusbar indicates we are paused at a * quantifier right now. It is the quantifier that takes zero or more processing instruction text tokens.
In the EBNF, we can see the current term highlighted in yellow. The input is also highlighting the current token in yellow and the previously-read token in green. These visual indicators make it easy to see what we’re debugging at this break.
The Non-Terminal Stack tab on the upper right is showing us which non-terminals we’ve traversed into. We’ve gone through the CompilationUnit non-terminal and are in the ProcessingInstruction one right now.
The Matches tab on the lower right shows us the AST matches that have been made thus far. For this particular non-terminal production scope, we have read the <? token so far.
In the toolbar we have a full array of step commands, the same as you’ll find when debugging your standard C#/VB code in Visual Studio. You can even set breakpoints on a non-terminal and code will auto-break when they are encountered.
Now let’s continue on a bit in our debugging…
In the above screenshot we have moved down further in the input and are at a < character. In the EBNF, we’re starting to evaluate nodes that can be within an element.
The Conditions tab shows us all the “first set” tokens and any can-match callbacks that are allowed to match at the current EBNF term.
Let’s press the Start button and parse through to the end now.
Parsing is complete and a text representation of the final AST result shows in the Matches tab.
Now let’s alter the input to be “<a> <b> </a>”, since we want to see if parse errors are caught.
When we run with this input, we see two parse errors were indicated. Our grammar has some advanced logic in it to recognize missing close tags, etc. You can see them reported in the Parse Errors tab. If you double-click on an error, it will jump to the proper location in the Input tab editor.
Since our XML grammar has error handling built-in, per the techniques mentioned in the past blog posts, you can see a proper AST is constructed even though there are syntax errors.
Summary
We’ve put a lot of work into the creation of a first-class debugger UI for our upcoming grammar/AST framework. We hope you will enjoy it once it’s released. This is an invaluable tool when working on grammars for your own languages.
The debugger UI and the grammar/AST framework should be released in WPF Studio 2010.2, which we hope to have out in September.