The 2010.2 version of SyntaxEditor for WPF (and the same for the 2010.2 Silverlight Studio release) will get a new built-in language service called SquiggleTagQuickInfoProvider that you can register on your language.  When this service is registered and the mouse is moved over a squiggle, it checks the squiggle data to see if there is related content available that can be displayed in an IntelliPrompt popup.  If so, it displays it.  Here’s and example of quick info for a parse error:

In this sample, we’re using the Web Languages Add-on’s XML language.  The XML parser returned that there was a parse error because there is an unexpected duplicate end tag.  We already have a built-in tagger that automatically looks for parse errors from a parser and makes squiggle lines in the editor for them.  Now with this new quick info provider, mouse hovers over the squiggles will also show the text of the parse error.

If you have a parser on your language that is capable of returning parse errors, then it just takes two lines of code to register the parse error tagger (which renders the squiggles) and the new squiggle tag quick info provider, thereby giving you all this functionality.

In the previous post, we added error handling to our Simple language grammar.  This allowed the grammar parser to recover from any invalid syntax in a document being parsed, and to still produce an AST.

It concluded the three basic stages in building a grammar:

1. Configure terminals, non-terminals, and EBNF productions
2. Add customized tree constructors to productions that create a concise AST
3. Add error handling to recover from invalid syntax

Now what happens when in the middle of the grammar design and something is not parsing as intended?  It can be tricky to know what the parser is “thinking” since it’s a bit of a black box.

To solve this issue, we’ve built complete debugging capabilities directly into the parser and have even enhanced the Language Designer application to have a complete debugger UI!  More...

We’ve just published build 102 of Silverlight Studio.  Major updates include:

• A free Ruby language definition for SyntaxEditor
• SyntaxEditor Language Designer updates including token and lexical state ID generation for dynamic lexers
• Improvements to our WPF/Silverlight compatibility layer for dependency properties

Head over the announcement post for details of what’s new.

In the previous post, we saw how the grammar framework supports callbacks nearly everywhere in the EBNF terms.  You are able to inject custom code before and after any term is matched.  One of the most important injection points is the Error callback since that allows you to tell the parser how to handle errors.

In today’s post, we’ll examine the importance of adding error handling.  We’ll enhance our Simple language grammar to properly recover from nearly any invalid syntax in a document so that it can continue parsing the rest of the document.

Why add error handling?

An error occurs when one or more terminals is expected by the parser (per the grammar) but a different token is found at the current location in the document.

By default when this scenario occurs, the parser will report a parse error if a single terminal is expected.  If there is more than one terminal that could be next, and if the containing non-terminal has an error alias set, it will report a parse error instead.  This behavior is described in the previous post. 

Following any possible error reporting, the parser will immediately exit the current non-terminal and will continue up the non-terminal stack, exiting each one as it goes until the root non-terminal is reached and exited itself.

Thus the result is that if any invalid syntax is found in a document, parsing stops at that point and no AST will be built.  That’s definitely not good since we expect code being parsed from our SyntaxEditor control editing to be invalid most of the time.  End users are continuously typing in it.

As shown in the previous post, the grammar framework has a lot ways to recover from errors, and they are easy to add.  More...

In the previous post, we optimized the tree construction output of our Simple language to be very concise.  The next step in building a grammar is to make sure that it properly handles errors.  After all, since this grammar framework is intended to be used with SyntaxEditor, our code editor control, we have to assume that most of the time the document’s code passed to our grammar parser will be in an invalid state.  The user is continuously typing and modifying it.

In today’s post we will look at the various callbacks that are available to you, probably the most important of which are error handling callbacks.  We’ll also dig into error handling options.

What is a callback?

As we’ve seen in the previous posts in this series, our entire grammar is built directly in C# or VB code.  We do not do code generation like a lot of other parser generators do.  A benefit of this is that you can interact directly with objects in the grammar.  More...

In the previous post we gave an introduction to the powerful tree construction mechanism that is built into the grammar/AST framework and talked about each of the built-in tree construction nodes that are available.  We also mentioned that for very rare scenarios where advanced logic is needed to build a particular AST node, you can create and inject your own ITreeConstructionNode-based classes to handle the task.

In today’s post, we’re going to revisit the Simple language and will enhance our grammar productions so that we make the resulting AST very concise.  More...

In the previous post of this series, we walked through how to create a grammar for the Simple language (which is similar to a Javascript subset) using our new grammar/AST framework.  We showed how easy it was to write the grammar using EBNF-like notation but completely from within your native C#/VB code.

We also showed how the grammar automatically generates an AST of the parsed document for you.  The generated tree generally has much more information than what is useful though.  Today’s post is going to show how we can completely customize the AST output so that it focuses only on the information we care about.

Our grammar/AST framework has an extremely powerful tree construction mechanism with numerous built-in helper methods to do common tree construction tasks.  You also have the ability to create completely custom tree construction nodes via C#/VB code if you like.

Default tree construction

The default tree constructors will output nearly everything that is parsed.  Each AST node that is generated automatically contains the offset range of the text that created it.  More...

In the previous post we gave a detailed introduction to symbols, EBNF terms, and how you can translate a language’s EBNF specification to our grammar framework.

In today’s post we’re going to write a C#-based grammar using our new framework for a language called Simple.  The Simple language is basically a small subset of a Javascript-like language.  When we’re done, we’ll load it up into a SyntaxEditor and will look at the AST results that are generated for us based on some input code.

Finding the language specification

The first thing to do when building a Grammar for a language is to locate the language specification for the language.  Nearly all programming languages have some sort of formal EBNF specification that tell parsers and compilers how to interpret their code.  More...

In the previous post, we gave an overview about what we’ll be covering in this multi-part walkthrough of the next generation grammar/AST framework we’re developing for SyntaxEditor, our code editor control.

Today’s post will dive right into a grammar and the building blocks needed to create one.

Grammar

A grammar is a set of production rules for parsing text in a formal language, where there is a single starting symbol from which parsing should begin.  All code languages should have some sort of EBNF grammar specification available for reference on the Internet, so that compilers and other products know how to read the code and interpret it.

Symbols

A grammar is comprised of symbols, which are either terminals or non-terminals.

Terminals are the lowest-level fixed-input lexical element used in a grammar production rule.  In the case of SyntaxEditor’s framework, terminals equate to tokens that are read in from a lexer.

NonTerminals each have a production rule, that is comprised of some combination of terminal and non-terminal symbol references.

EBNF sample

Take this EBNF for a Simple language FunctionDeclaration non-terminal:

   1: FunctionDeclaration:

   2:     "function" "identifier" "(" FunctionParameterList? ")" Block

That is saying the non-terminal’s name is FunctionDeclaration and a function declaration is built by a concatenation of a function terminal, followed by an identifier terminal, followed by an open parenthesis terminal, followed by an optional FunctionParameterList non-terminal, followed by a close parenthesis terminal, and ending with a Block non-terminal.  More...

Last month we did a posting called “What features do you want to see in WPF/Silverlight SyntaxEditor’s grammar and AST framework?” where we described some conceptual ideas we had for building the next-generation grammar and AST framework for our SyntaxEditor code editor control.  Before continuing on this post, you may wish to quickly review that.

Today’s post begins a multi-part walkthrough on what we’ve accomplished with the framework and shows how to use it.

We’ve been testing the framework with a WPF version of an advanced XML language implementation that has a grammar-based parser which constructs an AST of an XML document.  This was shown in the previous “Progress on a SyntaxEditor Web Languages Add-on for WPF and Silverlight” post.

Quick summary of grammar/AST

A grammar/AST framework is the key to enhancing a SyntaxEditor editing experience because it gives us meaningful information about what is contained in the document.  This not only can be used to provide contextual information to the end user (what method is the caret in), but is used to help drive features like automated IntelliPrompt completion lists, parameter info, and quick info.

To quickly sum up, a grammar is a set of parsing rules defined via EBNF for how to derive structure from text.  In the case of SyntaxEditor usage, the text being examined is nearly always programming code.  When running a grammar-based parser on text, one output is typically an AST (abstract syntax tree).  Another output is a collection of parsing errors.  Other custom output can be created as well.

For example, if we define a grammar for Javascript, an end user should be able to load up a Javascript document in SyntaxEditor and our syntax language will have access to the document’s AST and errors collection.  Then with some additional coding, the syntax language can use information from the AST to provide automated IntelliPrompt for the end user.

What this multi-part walkthrough will cover

We’ll be doing multiple posts over the coming days that walk through the design of our new grammar/AST model.  We’ll step through and build a grammar for our “Simple” language that is like a slimmed down version of Javascript.

We’ll also show a preview of the grammar debugger we’re working on that we hope can be included in a future release of the Language Designer.

Here’s a list of the posts that will be part of this walkthrough series:

• Part 1: Overview
• Part 2: Introduction to symbols and EBNF terms
• Part 3: Creating a grammar for the Simple language
• Part 4: Introduction to customizing tree construction
• Part 5: Optimizing the Simple grammar’s AST output
• Part 6: Introduction to callbacks and error handling
• Part 7: Adding error handling to the Simple grammar
• Part 8: Grammar debugger preview

We’ll dig into everything starting with the next post.  Stay tuned!