Class DefaultTokenBuilder

Hierarchy (View Summary)

DefaultTokenBuilder
- IndentationAwareTokenBuilder

Implements

TokenBuilder

Index

Constructors

Properties

Methods

buildKeywordPattern buildKeywordToken buildKeywordTokens buildTerminalToken buildTerminalTokens buildTokens findLongerAlt flushLexingReport popDiagnostics regexPatternFunction requiresCustomPattern

Constructors

constructor

new DefaultTokenBuilder(): DefaultTokenBuilder

Returns DefaultTokenBuilder

Properties

`Protected`diagnostics

diagnostics: LexingDiagnostic[] = []

The list of diagnostics stored during the lexing process of a single text.

Methods

`Protected`buildKeywordPattern

buildKeywordPattern(
keyword: GrammarAST.Keyword,
caseInsensitive: boolean,
): TokenPattern
Parameters
- keyword: GrammarAST.Keyword
- caseInsensitive: boolean
Returns TokenPattern
- Defined in packages/langium/src/parser/token-builder.ts:142

`Protected`buildKeywordToken

buildKeywordToken(
    keyword: GrammarAST.Keyword,
    terminalTokens: TokenType[],
    caseInsensitive: boolean,
): TokenType
Parameters
- keyword: GrammarAST.Keyword
- terminalTokens: TokenType[]
- caseInsensitive: boolean
Returns TokenType
- Defined in packages/langium/src/parser/token-builder.ts:127

`Protected`buildKeywordTokens

buildKeywordTokens(
    rules: Stream<GrammarAST.AbstractRule>,
    terminalTokens: TokenType[],
    options?: TokenBuilderOptions,
): TokenType[]
Parameters
- rules: Stream<GrammarAST.AbstractRule>
- terminalTokens: TokenType[]
- Optionaloptions: TokenBuilderOptions
Returns TokenType[]
- Defined in packages/langium/src/parser/token-builder.ts:116

`Protected`buildTerminalToken

buildTerminalToken(terminal: GrammarAST.TerminalRule): TokenType
Parameters
- terminal: GrammarAST.TerminalRule
Returns TokenType
- Defined in packages/langium/src/parser/token-builder.ts:81

`Protected`buildTerminalTokens

buildTerminalTokens(rules: Stream<GrammarAST.AbstractRule>): TokenType[]
Parameters
- rules: Stream<GrammarAST.AbstractRule>
Returns TokenType[]
- Defined in packages/langium/src/parser/token-builder.ts:76

buildTokens

buildTokens(grammar: Grammar, options?: TokenBuilderOptions): TokenVocabulary
Parameters
- grammar: Grammar
- Optionaloptions: TokenBuilderOptions
Returns TokenVocabulary
Implementation of TokenBuilder.buildTokens
- Defined in packages/langium/src/parser/token-builder.ts:51

`Protected`findLongerAlt

findLongerAlt(
keyword: GrammarAST.Keyword,
terminalTokens: TokenType[],
): TokenType[]
Parameters
- keyword: GrammarAST.Keyword
- terminalTokens: TokenType[]
Returns TokenType[]
- Defined in packages/langium/src/parser/token-builder.ts:148

flushLexingReport

flushLexingReport(text: string): LexingReport
Produces a lexing report for the given text that was just tokenized using the tokens provided by this builder.
Parameters
- text: string
  The text that was tokenized.
Returns LexingReport
Implementation of TokenBuilder.flushLexingReport
- Defined in packages/langium/src/parser/token-builder.ts:66

`Protected`popDiagnostics

popDiagnostics(): LexingDiagnostic[]
Returns LexingDiagnostic[]
- Defined in packages/langium/src/parser/token-builder.ts:70

`Protected`regexPatternFunction

regexPatternFunction(regex: RegExp): CustomPatternMatcherFunc
Parameters
- regex: RegExp
Returns CustomPatternMatcherFunc
- Defined in packages/langium/src/parser/token-builder.ts:107

`Protected`requiresCustomPattern

requiresCustomPattern(regex: RegExp): boolean
Parameters
- regex: RegExp
Returns boolean
- Defined in packages/langium/src/parser/token-builder.ts:98