UDL with two keywords current-date and current-dateTime only colours current-date

[martin-honnen]

I am trying to define a UDL XML file for XQuery 3.1 where I try to include the built-in XQuery/XPath 3.1 functions as keywords to be colourized. Now there are XPath/XQuery 3.1 functions that overlap in name, i.e. there is current-date and there is current-dateTime.

It appears that instead of colourizing current-dateTime completely, I end up with current-date in current-dateTime being colourized, the T then seems to be getting an operator colour and the ime is not colourized.

Any ideas how to change that so that current-dateTime is completely colourized?

My UDL:

<NotepadPlus>
    <UserLang name="XQuery 3.1" ext="xq xqm xqy xquery xqs" udlVersion="2.1">
        <Settings>
            <Global caseIgnored="no" allowFoldOfComments="yes" foldCompact="no" forcePureLC="0" decimalSeparator="0" />
            <Prefix Keywords1="yes" Keywords2="yes" Keywords3="yes" Keywords4="yes" Keywords5="yes" Keywords6="yes" Keywords7="yes" Keywords8="yes" />
        </Settings>
        <KeywordLists>
            <Keywords name="Comments">00 01 02((EOL)) 03(: 04:) 03&lt;!-- 04--&gt;</Keywords>
            <Keywords name="Numbers, prefix1"></Keywords>
            <Keywords name="Numbers, prefix2"></Keywords>
            <Keywords name="Numbers, extras1">e E</Keywords>
            <Keywords name="Numbers, extras2">+</Keywords>
            <Keywords name="Numbers, suffix1"></Keywords>
            <Keywords name="Numbers, suffix2"></Keywords>
            <Keywords name="Numbers, range">to</Keywords>
            <Keywords name="Operators1">= != &lt; &lt;= &gt; &gt;= is &lt;&lt; &gt;&gt; + - * div idiv mod || | union intersect except . .. / // @ :: ( ) [ ] , ; ? := :=</Keywords>
            <Keywords name="Operators2">and or instance of treat as cast as castable as satisfies collation ascending descending empty greatest least stable ordered unordered</Keywords>
            <Keywords name="Folders in code1, open">{</Keywords>
            <Keywords name="Folders in code1, middle"></Keywords>
            <Keywords name="Folders in code1, close">}</Keywords>
            <Keywords name="Folders in code2, open"></Keywords>
            <Keywords name="Folders in code2, middle"></Keywords>
            <Keywords name="Folders in code2, close"></Keywords>
            <Keywords name="Folders in comment, open">(: &lt;!--</Keywords>
            <Keywords name="Folders in comment, middle"></Keywords>
            <Keywords name="Folders in comment, close">:) --&gt;</Keywords>
            <Keywords name="Keywords1">xquery version encoding module namespace declare default element function variable base-uri boundary-space construction ordering order empty greatest least copy-namespaces preserve no-preserve inherit no-inherit decimal-format decimal-separator grouping-separator infinity minus-sign NaN percent per-mille zero-digit digit pattern-separator exponent-separator option collation schema import at external context item as</Keywords>
            <Keywords name="Keywords2">for let return where group by order stable count allowing empty at in some every satisfies if then else switch case default try catch typeswitch</Keywords>
            <Keywords name="Keywords3">document element attribute namespace text comment processing-instruction node document-node schema-element schema-attribute element-node attribute-node namespace-node text-node comment-node</Keywords>
            <Keywords name="Keywords4">item function map array empty-sequence</Keywords>
            <Keywords name="Keywords5">child descendant attribute self descendant-or-self following-sibling following parent ancestor preceding-sibling preceding ancestor-or-self</Keywords>
            <Keywords name="Keywords6">validate lax strict type</Keywords>
            <Keywords name="Keywords7">ordered unordered ascending descending collation</Keywords>
            <Keywords name="Keywords8">abs acos analyze-string apply avg boolean ceiling codepoint-equal codepoints-to-string collations compare concat contains contains-token count current-dateTime current-date current-time data dateTime dateTime-equal day-from-date day-from-dateTime days-from-duration deep-equal default-collation distinct-values document-uri element-with-id empty encode-for-uri ends-with environment-variable error escape-html-uri exists false filter floor fold-left fold-right for-each for-each-pair format-date format-dateTime format-integer format-number format-time function-arity function-lookup function-name generate-id head hours-from-date hours-from-dateTime hours-from-duration hours-from-time id idref implicit-timezone index-of innermost insert-before iri-to-uri json-doc json-to-xml key lang last load-xquery-module local-name local-name-from-QName lower-case matches max min minutes-from-date minutes-from-dateTime minutes-from-duration minutes-from-time month-from-date month-from-dateTime months-from-duration name namespace-uri namespace-uri-for-prefix namespace-uri-from-QName nilled node-name normalize-space normalize-unicode not number one-or-more outermost parse-ietf-date path patient position pow prefix-from-QName random-number-generator remove replace resolve-QName resolve-uri reverse root round round-half-to-even seconds-from-date seconds-from-dateTime seconds-from-duration seconds-from-time snapshot sort starts-with static-base-uri string string-join string-length string-to-codepoints subsequence substring substring-after substring-before sum tail timezone-from-date timezone-from-dateTime timezone-from-time tokenize trace translate true unordered unparsed-text unparsed-text-available unparsed-text-lines upper-case uri-collection xml-to-json year-from-date year-from-dateTime years-from-duration map:contains map:entry map:find map:for-each map:get map:keys map:merge map:put map:remove map:size map:submap map:values array:append array:filter array:flatten array:fold-left array:fold-right array:for-each array:get array:head array:insert-before array:join array:put array:remove array:reverse array:size array:sort array:subarray array:tail math:acos math:asin math:atan math:atan2 math:cos math:exp math:exp10 math:log math:log10 math:pi math:pow math:sin math:sqrt math:tan</Keywords>
            <Keywords name="Delimiters">00&quot; 01 02&quot; 03&apos; 04 05&apos; 06(# 07 08#) 09``[ 10 11]`` 12&lt;![CDATA[ 13 14]]&gt; 15&lt; 16 17&gt; 18 19 20 21 22 23</Keywords>
        </KeywordLists>
        <Styles>
            <WordsStyle name="DEFAULT" fgColor="2B2B2B" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="COMMENTS" fgColor="808080" bgColor="FFFFFF" fontStyle="2" nesting="0" />
            <WordsStyle name="LINE COMMENTS" fgColor="808080" bgColor="FFFFFF" fontStyle="2" nesting="0" />
            <WordsStyle name="NUMBERS" fgColor="0080FF" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="KEYWORDS1" fgColor="7A3E9D" bgColor="FFFFFF" fontStyle="1" nesting="0" />
            <WordsStyle name="KEYWORDS2" fgColor="CC7832" bgColor="FFFFFF" fontStyle="1" nesting="0" />
            <WordsStyle name="KEYWORDS3" fgColor="0033B3" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="KEYWORDS4" fgColor="0B7A0B" bgColor="FFFFFF" fontStyle="1" nesting="0" />
            <WordsStyle name="KEYWORDS5" fgColor="1F7A99" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="KEYWORDS6" fgColor="B000B0" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="KEYWORDS7" fgColor="B07000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="KEYWORDS8" fgColor="800000" bgColor="FFFFFF" fontStyle="1" nesting="0" />
            <WordsStyle name="OPERATORS" fgColor="0066CC" bgColor="FFFFFF" fontStyle="1" nesting="0" />
            <WordsStyle name="FOLDER IN CODE1" fgColor="2B2B2B" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="FOLDER IN CODE2" fgColor="2B2B2B" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="FOLDER IN COMMENT" fgColor="808080" bgColor="FFFFFF" fontStyle="2" nesting="0" />
            <WordsStyle name="DELIMITERS1" fgColor="008000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS2" fgColor="008000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS3" fgColor="008000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS4" fgColor="008000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS5" fgColor="008000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS6" fgColor="800000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS7" fgColor="333333" bgColor="FFFFFF" fontStyle="0" nesting="0" />
            <WordsStyle name="DELIMITERS8" fgColor="333333" bgColor="FFFFFF" fontStyle="0" nesting="0" />
        </Styles>
    </UserLang>
</NotepadPlus>

[PeterJones]

@martin-honnen ,

Thanks for including the UDL definition. It would have been nice to have pasted the example text, so I didn’t have to retype it for myself, so here’s what I’m using as my testbed

declare namespace output = "http://.../";
<items>
{
fold-left(
  1 to 10,
  random-number-generator(current-dateTime()),
  function($a, $i) {
  head($a)?next(),
  tail($a),
  <item>{(head($a)?number * 100) => floor()}</item>
  }
) => tail()
}
</items>

Some things I noticed:

• your operators1 have some non-symbol operators; please note that the manual page on UDL > Operators specifically says, “these should be special non-alphanumeric characters”. Using alphanumeric in the operators1 might have unintended consequences
  • with is as an operator1, then the word mississippi has two operators embedded inside:
    
    if is were moved to operator2, then it would only be recognized with spaces around it, like one is two and not in mississippi:
    
• the - in the operators1 doesn’t require spaces… but that means that the - in current-dateTime could be interpreted as an operator, rather than part of the current-dateTime keyword8 entry.
  • this is likely the culprit: when I move - from operators1 to operators2 (separators required) , then current-dateTime
  • I think that trying to have - as both part of certain keywords and as the non-space-required operators1 is confusing the parser => I think it sees the - operator while doing the “forward search” for operators, but then it sees the current-date keyword8 (rather than the current-dateTime), and gets confused about where to place the keyword8 highlighting vs the operator highlighting, so it puts the operator highlighting after the current-date keyword instead of on the unseparated -.
• You have prefix mode enabled on all your keyword# lists. Are you sure that’s what you want?
  • for example, since abs is in Keywords8 with prefix mode, abs and absolute and absotivelyposilutely will all be shown as keywords.
  • with prefix mode, keyword8 containing current-date would be sufficient to match current-date and current-dateTime and current-dateFakeReallyThisShouldntBeHighlightedAnyMore
• I notice you have ( ) [ ] in operators1 – I think it would make more sense for them to be one of the remaining delimiter-pairs (like putting Delimiter 7 open = ( [ close = ) ] ), and allow nesting of some or all of the other categories…
• You also have < > in operators1 but also as delimiter6. Having the same exact token in multiple places in the UDL definition can confuse the parser

So my recommendations:

• move the - to operators2: this appears to fix your immediate problem
• move the words from operators1 to operators2
• don’t duplicate any tokens in multiple UDL fields
• don’t use prefix mode on keywords unless you really need it to match any alphanumerics that come after the prefixes listed in that specific keyword list

[PeterJones]

I said,

• I notice you have ( ) [ ] in operators1 – I think it would make more sense for them to be one of the remaining delimiter-pairs (like putting Delimiter 7 open = ( [ close = ) ] ), and allow nesting of some or all of the other categories…

I played around a bit with that. it messes up the folding, because the Nesting section of the Delimiter#'s Styler settings don’t allow nesting the folding keywords. So I no longer recommend moving the ( ) to a delimiter, and it’s doubtful that moving [ ] is overly helpful, either.

[martin-honnen]

@PeterJones, many thanks, great analysis, great suggestions.

[Lycan Thrope]

@PeterJones ,
This is the hardest part of doing the UDL’s I think. Even after we’ve done the original dBASE UDL you helped with, I’m finding that doing the earlier versions now, and fine-tuning the keywords, operators and the nesting precedents was the most confusing and finer control over the language. Doing the earlier versions I’ve figured out how to keep an apostrophe, from screwing up my code highlighting, while still keeping it as a delimiter by simply NOT allowing it to be included in the other delimiters nesting options. In turn, dBASE also has a JSON object, and with a little fine tuning, I was able to make the delimiters in dBASE also work to outline a JSON structure, so far. If there are other options in a JSON structure I’ll have to learn that as I go, but for now, the simple W3C version showing it, it works right, as well as the dBASE delimiters without messing things up.

Obviously a quick UDL is possible with just the categories to put things in, but I’m finding that the real power of the UDL system is in the finer use of the mechanisms like the nesting, differentiation of forward search/backward search, prefix mode, etc works much better when combined to get a desired result. It’s actually much more powerful than I originally figured. I wish I’d had a good explanation/example like you’ve done, back then…although, with so much to include to make a UDL almost a complete language system, it would have probably been too much to absorb at the time and would not have availed me much improvement at the time. It’s something that requires a lot of trial and error and understanding of the mechanisms and time playing with it, I think, to get things just right, if it’s at all possible to do. :-)