Package org.antlr.grammar.v2
Class AssignTokenTypesWalker
java.lang.Object
antlr.TreeParser
org.antlr.grammar.v2.AssignTokenTypesWalker
- All Implemented Interfaces:
AssignTokenTypesWalkerTokenTypes
- Direct Known Subclasses:
AssignTokenTypesBehavior
public class AssignTokenTypesWalker
extends antlr.TreeParser
implements AssignTokenTypesWalkerTokenTypes
[Warning: TJP says that this is probably out of date as of 11/19/2005,
but since it's probably still useful, I'll leave in. Don't have energy
to update at the moment.]
Compute the token types for all literals and rules etc.. There are
a few different cases to consider for grammar types and a few situations
within.
CASE 1 : pure parser grammar
a) Any reference to a token gets a token type.
b) The tokens section may alias a token name to a string or char
CASE 2 : pure lexer grammar
a) Import token vocabulary if available. Set token types for any new tokens
to values above last imported token type
b) token rule definitions get token types if not already defined
c) literals do NOT get token types
CASE 3 : merged parser / lexer grammar
a) Any char or string literal gets a token type in a parser rule
b) Any reference to a token gets a token type if not referencing
a fragment lexer rule
c) The tokens section may alias a token name to a string or char
which must add a rule to the lexer
d) token rule definitions get token types if not already defined
e) token rule definitions may also alias a token name to a literal.
E.g., Rule 'FOR : "for";' will alias FOR to "for" in the sense that
references to either in the parser grammar will yield the token type
What this pass does:
0. Collects basic info about the grammar like grammar name and type;
Oh, I have go get the options in case they affect the token types.
E.g., tokenVocab option.
Imports any token vocab name/type pairs into a local hashtable.
1. Finds a list of all literals and token names.
2. Finds a list of all token name rule definitions;
no token rules implies pure parser.
3. Finds a list of all simple token rule defs of form " : ;"
and aliases them.
4. Walks token names table and assign types to any unassigned
5. Walks aliases and assign types to referenced literals
6. Walks literals, assigning types if untyped
4. Informs the Grammar object of the type definitions such as:
g.defineToken(, ttype);
g.defineToken(, ttype);
g.defineToken(, ttype);
where some of the ttype values will be the same for aliases tokens.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final String[]
protected static GrammarAST
protected static GrammarAST
protected String
protected Grammar
protected static GrammarAST
protected static GrammarAST
Fields inherited from class antlr.TreeParser
_retTree, astFactory, ASTNULL, inputState, returnAST, tokenNames, traceDepth
Fields inherited from interface org.antlr.grammar.v2.AssignTokenTypesWalkerTokenTypes
ACTION, ACTION_CHAR_LITERAL, ACTION_ESC, ACTION_STRING_LITERAL, ALT, AMPERSAND, ARG, ARG_ACTION, ARGLIST, ASSIGN, BACKTRACK_SEMPRED, BANG, BLOCK, CHAR_LITERAL, CHAR_RANGE, CHARSET, CLOSE_ELEMENT_OPTION, CLOSURE, COLON, COMBINED_GRAMMAR, COMMA, COMMENT, DIGIT, DOC_COMMENT, DOLLAR, DOT, DOUBLE_ANGLE_STRING_LITERAL, DOUBLE_QUOTE_STRING_LITERAL, EOA, EOB, EOF, EOR, EPSILON, ESC, ETC, FORCED_ACTION, FRAGMENT, GATED_SEMPRED, ID, IMPLIES, IMPORT, INITACTION, INT, INTERNAL_RULE_REF, LABEL, LEXER, LEXER_GRAMMAR, LITERAL_catch, LITERAL_finally, LITERAL_grammar, LITERAL_lexer, LITERAL_private, LITERAL_protected, LITERAL_public, LITERAL_returns, LITERAL_throws, LITERAL_tree, LPAREN, ML_COMMENT, NESTED_ACTION, NESTED_ARG_ACTION, NOT, NULL_TREE_LOOKAHEAD, OPEN_ELEMENT_OPTION, OPTIONAL, OPTIONS, OR, PARSER, PARSER_GRAMMAR, PLUS, PLUS_ASSIGN, POSITIVE_CLOSURE, QUESTION, RANGE, RCURLY, RET, REWRITE, ROOT, RPAREN, RULE, RULE_REF, SCOPE, SEMI, SEMPRED, SL_COMMENT, SRC, STAR, STRAY_BRACKET, STRING_LITERAL, SYN_SEMPRED, SYNPRED, TEMPLATE, TOKEN_REF, TOKENS, TREE_BEGIN, TREE_GRAMMAR, WILDCARD, WS, WS_LOOP, WS_OPT, XDIGIT
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected void
alias
(GrammarAST t, GrammarAST s) protected void
final void
alternative
(antlr.collections.AST _t) protected void
assignStringTypes
(Grammar root) protected void
assignTokenIDTypes
(Grammar root) final void
ast_suffix
(antlr.collections.AST _t) final void
atom
(antlr.collections.AST _t) final void
attrScope
(antlr.collections.AST _t) final void
block
(antlr.collections.AST _t) final void
charSet
(antlr.collections.AST _t) final void
charSetElement
(antlr.collections.AST _t) protected void
protected void
void
defineTokens
(Grammar root) final void
delegateGrammars
(antlr.collections.AST _t) final void
ebnf
(antlr.collections.AST _t) final void
element
(antlr.collections.AST _t) final void
exceptionGroup
(antlr.collections.AST _t) final void
exceptionHandler
(antlr.collections.AST _t) final void
finallyClause
(antlr.collections.AST _t) final void
final void
grammarSpec
(antlr.collections.AST _t) protected void
protected void
final void
modifier
(antlr.collections.AST _t) final void
final Map
optionsSpec
(antlr.collections.AST _t) final Object
optionValue
(antlr.collections.AST _t) void
reportError
(antlr.RecognitionException ex) final void
rewrite
(antlr.collections.AST _t) final void
rule
(antlr.collections.AST _t) final void
rules
(antlr.collections.AST _t) final void
ruleScopeSpec
(antlr.collections.AST _t) final void
tokenSpec
(antlr.collections.AST _t) final void
tokensSpec
(antlr.collections.AST _t) protected void
protected void
protected void
trackTokenRule
(GrammarAST t, GrammarAST modifier, GrammarAST block) final void
tree
(antlr.collections.AST _t) Methods inherited from class antlr.TreeParser
getAST, getASTFactory, getTokenName, getTokenNames, match, match, matchNot, panic, reportError, reportWarning, setASTFactory, setASTNodeClass, setASTNodeType, traceIn, traceIndent, traceOut
-
Field Details
-
grammar
-
currentRuleName
-
stringAlias
-
charAlias
-
stringAlias2
-
charAlias2
-
_tokenNames
-
-
Constructor Details
-
AssignTokenTypesWalker
public AssignTokenTypesWalker()
-
-
Method Details
-
reportError
public void reportError(antlr.RecognitionException ex) - Overrides:
reportError
in classantlr.TreeParser
-
initASTPatterns
protected void initASTPatterns() -
trackString
-
trackToken
-
trackTokenRule
-
alias
-
defineTokens
-
defineStringLiteralsFromDelegates
protected void defineStringLiteralsFromDelegates() -
assignStringTypes
-
aliasTokenIDsAndLiterals
-
assignTokenIDTypes
-
defineTokenNamesAndLiteralsInGrammar
-
init
-
grammar
- Throws:
antlr.RecognitionException
-
grammarSpec
public final void grammarSpec(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
optionsSpec
- Throws:
antlr.RecognitionException
-
delegateGrammars
public final void delegateGrammars(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
tokensSpec
public final void tokensSpec(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
attrScope
public final void attrScope(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
rules
public final void rules(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
option
- Throws:
antlr.RecognitionException
-
optionValue
- Throws:
antlr.RecognitionException
-
charSet
public final void charSet(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
charSetElement
public final void charSetElement(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
tokenSpec
public final void tokenSpec(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
rule
public final void rule(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
modifier
public final void modifier(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
ruleScopeSpec
public final void ruleScopeSpec(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
block
public final void block(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
exceptionGroup
public final void exceptionGroup(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
alternative
public final void alternative(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
rewrite
public final void rewrite(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
element
public final void element(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
exceptionHandler
public final void exceptionHandler(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
finallyClause
public final void finallyClause(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
atom
public final void atom(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
ebnf
public final void ebnf(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
tree
public final void tree(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-
ast_suffix
public final void ast_suffix(antlr.collections.AST _t) throws antlr.RecognitionException - Throws:
antlr.RecognitionException
-