This post continues our series on new features coming to SyntaxEditor 2011.1, primarily centered around our LL(*) Parser Framework.
In our last post, we showed the new grammar syntax features that allow for type-specific AST nodes to be created. In today’s post, we’re going to look into enhancements made to can-match callbacks and error reporting.
Can-Match Callbacks
What Is a Can-Match Callback?
Our parser framework by default uses one token of look-ahead to determine when a non-terminal should be matched. This sort of parser is denoted LL(1) where the 1 means one token look-ahead.
However in complex languages, there are often scenarios where there is ambiguity between two possible non-terminal alternation options that can start with the same terminal. Sometimes it can take two or more tokens of look-ahead to determine which non-terminal to choose. Our grammar compiler warns you when these scenarios are encountered, allowing you to create a can-match callback to resolve the ambiguity. The general concept is described in more detail in this older post, but can-match callbacks allow you to do up to infinite look-ahead of tokens to determine if a non-terminal can be matched or note. This allows parsers made using our framework to be LL(*).
AreNext Helper Method
A lot of times, can-match callbacks just involve checking for a specific sequence of two or more tokens. We’ve added a new AreNext method to the ITokenReader interface where you pass two or more token IDs and the method returns if that exact sequence of tokens is next.
Let’s see an example:
1: tokenReader.AreNext(VBTokenId.OpenParenthesis, VBTokenId.Of)
The code above is for a VB type parameter list can-match callback. It will tell us if the next two tokens are a ( followed by the keyword Of, which is criteria we have for the type parameter list when it is used in ambiguous scenarios.
Lambda Can-Match Callbacks
While usage of lambdas isn’t something new in 2011.1, we thought it would be handy to show how short can-match callbacks can be specified in a single statement instead of making their own dedicated method:
1: typeParameterList.CanMatchCallback =
2: (state => state.TokenReader.AreNext(VBTokenId.OpenParenthesis, VBTokenId.Of));
Note how the can-match callback for type parameter list is fully specified by a lambda that calls AreNext.
Token Reader Push/Pop Methods
ITokenReader has gotten new Push and Pop methods for 2011.1. These methods are used in more complex can-match callbacks, where you need to do multiple token look-ahead and check for special conditions along the way.
Normally reading ahead through tokens in a can-match callback would consume the token, preventing it from being parsed. By calling Push we save the state of the token reader. We can advance through as many tokens as we wish and then Pop the old state back into place.
Here is an example of their usage:
1: private bool CanMatchLocalVariableDeclaration(IParserState state) {
2: state.TokenReader.Push();
3: try {
4: if (state.TokenReader.LookAheadToken.Id == CSharpTokenId.Var) {
5: // Advance past 'var'
6: state.TokenReader.Advance();
7: }
8: else if (!this.IsType(state))
9: return false;
10:
11: return identifierOrContextualKeyword.CanMatch(state);
12: }
13: finally {
14: state.TokenReader.Pop();
15: }
16: }
Note we are able to freely advance through tokens and do other calls (like to the IsType method) that also advance through more tokens to tell us if the conditions to start this non-terminal are met. The Push/Pop feature is a great enhancement for 2011.1.
These complex can-match callbacks are only needed in scenarios where there is a lot of ambiguity in certain parts of a grammar. Some grammars might not need can-match callbacks at all, while others like C# and VB do need a number of them to resolve ambiguities.
Token Can-Match Callbacks
In past versions, only non-terminals could have can-match callbacks. In 2011.1, we added can-match callback ability to terminals as well. This feature will only be used in rare scenarios. An example is say that you have some contextual keywords, meaning identifiers that turn into keywords but only in certain parser contexts.
In this case you could define a terminal that had a can-match callback to first make sure the look-ahead token is an identifier. If so, then check its text value to see if it matches the contextual keyword text. That is one way to handle contextual keywords like var or from in C#.
Error Reporting
All EBNF terms in our parser framework support callbacks for initialize, success, error, and completion. Success callbacks are often used to examine matches to make sure they are valid. For instance, in C# we make sure that the modifiers specified are valid for the type (class, interface, etc.) to which they are applied and report an error if not.
1: void ReportError(ParseErrorLevel level, string description);
2: void ReportError(ParseErrorLevel level, string description, IToken token);
3: void ReportError(IParseError parseError);
4: void ReportNonTerminalExpectedError(NonTerminal nonTerminal);
5: void ReportNonTerminalExpectedError(NonTerminal nonTerminal, IToken token);
6: void ReportTerminalExpectedError(Terminal terminal);
7: void ReportTerminalExpectedError(Terminal terminal, IToken token);
As you can see, the IParserState interface defines a full array of error reporting methods that are available for your use. Any errors that are reported bubble up to SyntaxEditor and can automatically show as squiggle lines. Mouse hovers over the squiggles show description tips containing the error message.
New error reporting helper methods have been added that let you indicate an IToken over which the error should appear. All the offset range calculation for the error is done for you.
Summary
This concludes our preview of upcoming LL(*) Parser Framework enhancements for the 2011.1 version of SyntaxEditor for WPF/Silverlight. We’re very excited about the updates and hope those of you looking to write a text parser will take advantage of them. 2011.1 will be available in the coming weeks.