genscript.syntax.tokens

Class Tokenizer

java.lang.Object

genscript.parse.ReaderStack

genscript.parse.IndexReader

genscript.syntax.tokens.Tokenizer

All Implemented Interfaces:

ReaderInterface

Direct Known Subclasses:

FlexibleTokenizer, PreScanTokenizer, ScanTokenizer

public abstract class Tokenizer
extends IndexReader

Field Summary

Fields

Modifier and Type	Field and Description
protected IndexReader	innerSource
protected VectorStack	tokenStack

Constructor Summary

Constructors

Constructor and Description
Tokenizer(IndexReader source)

Method Summary

Modifier and Type	Method and Description
void	clear() Deletes all the read information.
IndexReader	getBottomLevelSource()
java.lang.String	getFilename()
boolean	isTokenStackEmpty()
java.lang.Object	pollTokenValue()
void	pushTokenValue(java.lang.Object value)
java.lang.String	tokenStackToString()

Methods inherited from class genscript.parse.IndexReader

canStepBack, getBottomLevelIntervalStartPos, getBottomLevelPosition, getBottomLevelPosition, getBufferSize, getErrorPosition, getIntervalEndPos, getIntervalStartPos, getPosition, hasErrorPosition, intervalToString, intervalToString, isEndOfBuffer, newReader, parseStart, read, setErrorPosition, setIntervalEndPos, setIntervalPositions, setIntervalStartPos, setPosition, skipSpace, stepBack, switchParser, toTextPosition

Methods inherited from class genscript.parse.ReaderStack

getMarkedStackSize, getResultStackSize, getStackMark, popResult, pushResult, resetStack, resultStackToString, setStackMark

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokenStack

protected VectorStack tokenStack

innerSource

protected IndexReader innerSource

Constructor Detail

Tokenizer

public Tokenizer(IndexReader source)

Method Detail

isTokenStackEmpty

public boolean isTokenStackEmpty()

pollTokenValue

public java.lang.Object pollTokenValue()

pushTokenValue

public void pushTokenValue(java.lang.Object value)

getBottomLevelSource

public IndexReader getBottomLevelSource()

Specified by:

getBottomLevelSource in class IndexReader

clear

public void clear()

Description copied from class: IndexReader

Deletes all the read information. The read position remains the same.

Overrides:

clear in class IndexReader

tokenStackToString

public java.lang.String tokenStackToString()

getFilename

public final java.lang.String getFilename()

Specified by:

getFilename in interface ReaderInterface

Specified by:

getFilename in class IndexReader

Returns:

the filename of the source code.