Package org.antlr.runtime
Class CommonTokenStream
java.lang.Object
org.antlr.runtime.BufferedTokenStream
org.antlr.runtime.CommonTokenStream
- All Implemented Interfaces:
IntStream
,TokenStream
- Direct Known Subclasses:
TokenRewriteStream
The most common stream of tokens where every token is buffered up
and tokens are filtered for a certain channel (the parser will only
see these tokens).
Even though it buffers all of the tokens, this token stream pulls tokens
from the tokens source on demand. In other words, until you ask for a
token using consume(), LT(), etc. the stream does not pull from the lexer.
The only difference between this stream and BufferedTokenStream superclass
is that this stream knows how to ignore off channel tokens. There may be
a performance advantage to using the superclass if you don't pass
whitespace and comments etc. to the parser on a hidden channel (i.e.,
you set $channel instead of calling skip() in lexer rules.)
- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected int
Skip tokens on any channel but this one; this is how we skip whitespace...Fields inherited from class org.antlr.runtime.BufferedTokenStream
lastMarker, p, range, tokens, tokenSource
-
Constructor Summary
ConstructorsConstructorDescriptionCommonTokenStream
(TokenSource tokenSource) CommonTokenStream
(TokenSource tokenSource, int channel) -
Method Summary
Modifier and TypeMethodDescriptionvoid
consume()
Always leave p on an on-channel token.int
Count EOF just once.protected Token
LB
(int k) LT
(int k) Get Token at current input pointer + i ahead where i=1 is next Token.void
reset()
void
setTokenSource
(TokenSource tokenSource) Reset this token stream by setting its token source.protected void
setup()
protected int
skipOffTokenChannels
(int i) Given a starting index, return the index of the first on-channel token.protected int
skipOffTokenChannelsReverse
(int i)
-
Field Details
-
channel
protected int channelSkip tokens on any channel but this one; this is how we skip whitespace...
-
-
Constructor Details
-
CommonTokenStream
public CommonTokenStream() -
CommonTokenStream
-
CommonTokenStream
-
-
Method Details
-
consume
public void consume()Always leave p on an on-channel token.- Specified by:
consume
in interfaceIntStream
- Overrides:
consume
in classBufferedTokenStream
-
LB
- Overrides:
LB
in classBufferedTokenStream
-
LT
Description copied from interface:TokenStream
Get Token at current input pointer + i ahead where i=1 is next Token. i<0 indicates tokens in the past. So -1 is previous token and -2 is two tokens ago. LT(0) is undefined. For i>=n, return Token.EOFToken. Return null for LT(0) and any index that results in an absolute address that is negative.- Specified by:
LT
in interfaceTokenStream
- Overrides:
LT
in classBufferedTokenStream
-
skipOffTokenChannels
protected int skipOffTokenChannels(int i) Given a starting index, return the index of the first on-channel token. -
skipOffTokenChannelsReverse
protected int skipOffTokenChannelsReverse(int i) -
reset
public void reset()- Overrides:
reset
in classBufferedTokenStream
-
setup
protected void setup()- Overrides:
setup
in classBufferedTokenStream
-
getNumberOfOnChannelTokens
public int getNumberOfOnChannelTokens()Count EOF just once. -
setTokenSource
Reset this token stream by setting its token source.- Overrides:
setTokenSource
in classBufferedTokenStream
-