</div></div></article><h3class="sectionHeader">Constructors</h3><articleid="Tokenizer.new"data-title="Tokenizer.new (constructor)"class="article"><divclass="section method"><divclass="memberContent"><divclass="title"><ulclass="signatures"><liclass="signature"><ul><liclass="signature-call"><spanclass="constructorIdentifier">new </span><spanid="Tokenizer.new"class="member-name methodClicker">Tokenizer</span>(<spanclass="sigArgList"><ahref="http://www.nodemanual.org/latest/js_doc/Object.html"class="argument methodClicker"title="Object"data-id="Object">Object</a> rules, <ahref="http://www.nodemanual.org/latest/js_doc/String.html"class="argument methodClicker"title="String"data-id="String">String</a> flag</span>)</li></ul><ulclass="metaInfo"></ul></li></ul></div><divclass="sideToggler"><divid="ellipsis_Tokenizer.new"class="ellipsis_description"><p>Constructs a new tokenizer based on the given rules and flags.</p>
</div><h3id="Tokenizer.new"class="methodToggle methodClicker inactive"> </h3><divclass="description"><p>Constructs a new tokenizer based on the given rules and flags.</p>
<h4>Arguments</h4><tableclass="argumentTable argument-list table table-striped table-bordered"><trclass="argumentRow "><tdclass="argName ">rules</td><tdclass="argType""><ahref="http://www.nodemanual.org/latest/js_doc/Object.html"class=""title="Object"data-id="Object">Object</a></td><tdclass="argDescription ">Required. The highlighting rules</td></tr><trclass="argumentRow "><tdclass="argName ">flag</td><tdclass="argType""><ahref="http://www.nodemanual.org/latest/js_doc/String.html"class=""title="String"data-id="String">String</a></td><tdclass="argDescription ">Required. Any additional regular expression flags to pass (like "i" for case insensitive)</td></tr></table></div></div></div></div></article><h3class="sectionHeader">Methods</h3><articleid="Tokenizer.getLineTokens"data-title="Tokenizer.getLineTokens (class method)"class="article"><divclass="section method"><divclass="memberContent"><divclass="title"><ulclass="signatures"><liclass="signature"><ul><liclass="signature-call"><spanid="Tokenizer.getLineTokens"class="member-name methodClicker"><spanclass="sigClassName">Tokenizer.</span><spanclass="sigMemberName">getLineTokens</span></span>(<spanclass="sigArgList"></span>)<liclass="signature-returns"><ulclass="argument-types"><liclass="argument-type"><ahref="http://www.nodemanual.org/latest/js_doc/Object.html"class="returnType "title="Object"data-id="Object">Object</a></li></ul></li></li></ul><ulclass="metaInfo"></ul></li></ul></div><divclass="sideToggler"><divid="ellipsis_Tokenizer.getLineTokens"class="ellipsis_description"><p>Returns an object containing two properties: <code>tokens</code>, which contains all the tokens; and <code>state</code>, the current state.</p>
</div><h3id="Tokenizer.getLineTokens"class="methodToggle methodClicker inactive"> </h3><divclass="description"><p>Returns an object containing two properties: <code>tokens</code>, which contains all the tokens; and <code>state</code>, the current state.</p>