Class DefaultTokenBuilder

Hierarchy (view full)

DefaultTokenBuilder
- IndentationAwareTokenBuilder

Implements

TokenBuilder

Index

Constructors

Properties

Methods

buildKeywordPattern buildKeywordToken buildKeywordTokens buildTerminalToken buildTerminalTokens buildTokens findLongerAlt flushLexingReport popDiagnostics regexPatternFunction requiresCustomPattern

Constructors

constructor

new DefaultTokenBuilder(): DefaultTokenBuilder
Returns DefaultTokenBuilder

Properties

`Protected`diagnostics

diagnostics: LexingDiagnostic[] = []

The list of diagnostics stored during the lexing process of a single text.

Methods

`Protected`buildKeywordPattern

buildKeywordPattern(keyword, caseInsensitive): TokenPattern
Parameters
- keyword: GrammarAST.Keyword
- caseInsensitive: boolean
Returns TokenPattern
- Defined in packages/langium/src/parser/token-builder.ts:149

`Protected`buildKeywordToken

buildKeywordToken(keyword, terminalTokens, caseInsensitive): TokenType
Parameters
- keyword: GrammarAST.Keyword
- terminalTokens: TokenType[]
- caseInsensitive: boolean
Returns TokenType
- Defined in packages/langium/src/parser/token-builder.ts:134

`Protected`buildKeywordTokens

buildKeywordTokens(rules, terminalTokens, options?): TokenType[]
Parameters
- rules: Stream<GrammarAST.AbstractRule>
- terminalTokens: TokenType[]
- Optionaloptions: TokenBuilderOptions
Returns TokenType[]
- Defined in packages/langium/src/parser/token-builder.ts:123

`Protected`buildTerminalToken

buildTerminalToken(terminal): TokenType
Parameters
- terminal: GrammarAST.TerminalRule
Returns TokenType
- Defined in packages/langium/src/parser/token-builder.ts:85

`Protected`buildTerminalTokens

buildTerminalTokens(rules): TokenType[]
Parameters
- rules: Stream<GrammarAST.AbstractRule>
Returns TokenType[]
- Defined in packages/langium/src/parser/token-builder.ts:80

buildTokens

buildTokens(grammar, options?): TokenVocabulary
Parameters
- grammar: Grammar
- Optionaloptions: TokenBuilderOptions
Returns TokenVocabulary
Implementation of TokenBuilder.buildTokens
- Defined in packages/langium/src/parser/token-builder.ts:51

`Protected`findLongerAlt

findLongerAlt(keyword, terminalTokens): TokenType[]
Parameters
- keyword: GrammarAST.Keyword
- terminalTokens: TokenType[]
Returns TokenType[]
- Defined in packages/langium/src/parser/token-builder.ts:155

flushLexingReport

flushLexingReport(text): LexingReport
Produces a lexing report for the given text that was just tokenized using the tokens provided by this builder.
Parameters
- text: string
  The text that was tokenized.
Returns LexingReport
Implementation of TokenBuilder.flushLexingReport
- Defined in packages/langium/src/parser/token-builder.ts:70

`Protected`popDiagnostics

popDiagnostics(): LexingDiagnostic[]
Returns LexingDiagnostic[]
- Defined in packages/langium/src/parser/token-builder.ts:74

`Protected`regexPatternFunction

regexPatternFunction(regex): CustomPatternMatcherFunc
Parameters
- regex: RegExp
Returns CustomPatternMatcherFunc
- Defined in packages/langium/src/parser/token-builder.ts:114

`Protected`requiresCustomPattern

requiresCustomPattern(regex): boolean
Parameters
- regex: RegExp
Returns boolean
- Defined in packages/langium/src/parser/token-builder.ts:102