SyntaxEditor updates to dynamic lexers

SyntaxEditor updates to dynamic lexers - feedback wanted

We’ve started some work on doing updates to what are known as “dynamic languages” in SyntaxEditor 4.0. These languages allow you to define a lexer in terms of explicit and regular expression pattern groups and make it very simple to get syntax highlighting working fast.

In the next-generation framework we’re prototyping out with the WPF version of SyntaxEditor, we’ve renamed these to be called “dynamic lexers” instead, since for the most part, they just control how the lexer tokenizes and highlights text. Thus they can really be used with any language.

Anyhow, while we’re pretty happy with the XML format for defining dynamic lexers right now, we’d love to get additional input on new features you’d like to see or ideas for improvement. Please post your comments or email them over.

The only thing I would add, potentially, is a separation between the content of pattern groups and the states in which they exist. For example, in my language for SQL I have [ and ] marked as delimiters for 'square string state'. However, that means I need to add tokens (e.g. a stored procedure name) to a pattern group both under the default state and the square string state. It would be slightly more elegant if I could define the pattern group once, outside of the context of a state, and then include that pattern group in the relevant states.

i.e. instead of:

<State Key="DefaultState">
  <PatternGroups>
    <ExplicitPatternGroup TokenKey="ProgrammableObjectToken">
      ... content ...
    </ExplicitPatternGroup>
  </PatternGroups>
</State>

<State Key="IdentifierState">
  <PatternGroups>
    <ExplicitPatternGroup TokenKey="ProgrammableObjectToken">
      ... content ...
    </ExplicitPatternGroup>
  </PatternGroups>
</State>

I might have:

<PatternGroups>
  <ExplicitPatternGroup Name="ProgrammableObjectToken" TokenKey="ProgrammableObjectToken">
    ... content ...
  </ExplicitPatternGroup>
</PatternGroups>

<State Key="DefaultState">
  <PatternGroups>
    <IncludedPatternGroup Name="ProgrammableObjectToken">
  </PatternGroups>
</State>

<State Key="IdentifierState">
  <PatternGroups>
    <IncludedPatternGroup Name="ProgrammableObjectToken">
  </PatternGroups>
</State>

July 19, 2009 at 14:17

I would be really helpful if the regular expressions pattern groups had complete support for regex groups and options like prefix/suffix exclude, prefix/suffix in not present, as few as possible, along with other typical regex abilities.

Paul Hasselfeldt

July 19, 2009 at 15:30

Hi Paul,

The regexes should support already nearly everything found in the .NET regex engine except for lazy qualifiers. Could you provide some detailed code examples on what you are specifically looking for? Thanks!

Bill Henning (Actipro)

July 20, 2009 at 07:21

If I recall correctly, Actipro regexes don't support Unicode categories like .NET does. I've had to work around that once and it wasn't particularly fun.

Jesper

July 20, 2009 at 07:34

We did some work recently on those and SyntaxEditor for WPF's regex engine should support "\p{name}" and "\P{name}" now.

July 20, 2009 at 07:35

Neat; I won't have to have long const strings to inline a valid C# identifier character (at the time of compilation, anyway) anymore.

July 22, 2009 at 12:03

Matt Whitfield

July 22, 2009 at 13:28

Thanks for the suggestion Matt, we'll think about this more. I wonder if we could alternatively do something like define the pattern group like normal in one state, give it a Key (already a property on pattern group), and then have a new tag like PatternGroupRef where you could indicate a Key and that would essentially be a reference to the first pattern group's data. So PatternGroupRef would be usable anywhere ExplicitPatternGroup or RegexPatternGroup are.

July 22, 2009 at 14:49

I think that would be even better, actually - good thought!

October 1, 2009 at 09:41

Actipro Blog 2009 Q3 posting summary

Actipro Blog 2009 Q3 posting summary

The Actipro Blog - WPF and WinForms Development

The Actipro Blog

Latest Twitter News

Month List

About Us