Tokenizer[edit]

Constructors

Constructs a new tokenizer based on the given rules and flags.

   

Constructs a new tokenizer based on the given rules and flags.

Arguments

rulesObjectRequired. The highlighting rules
flagStringRequired. Any additional regular expression flags to pass (like "i" for case insensitive)

Methods

    • Tokenizer.getLineTokens()

    Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

       

    Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.