genscript.defaultreaders

Class FlexibleTokenizer

java.lang.Object

genscript.parse.ReaderStack

genscript.parse.IndexReader

genscript.syntax.tokens.Tokenizer

genscript.defaultreaders.FlexibleTokenizer

All Implemented Interfaces:

ReaderInterface

public class FlexibleTokenizer
extends Tokenizer

Field Summary

Fields inherited from class genscript.syntax.tokens.Tokenizer

innerSource, tokenStack

Constructor Summary

Constructors

Constructor and Description
FlexibleTokenizer(IndexReader innerSource)

Method Summary

Modifier and Type	Method and Description
void	clear() Deletes all the read information.
int	getBottomLevelPosition(int topLevelPosition)
int	getBufferSize()
int	getIntervalEndPos()
int	getIntervalStartPos()
int	getPosition()
java.lang.String	intervalToString(int startShift, int endShift)
boolean	isEndOfBuffer()
IndexReader	newReader() Clones this Reader without copying the buffer itself
Token	read() Returns the next index in the source code and moves the marker.
void	setIntervalEndPos(int endPos)
void	setIntervalStartPos(int startPos)
void	setPosition(int position) Sets the Position of the inner Buffer.
void	setTokenSet(TokenSet tokenSet)
void	skipSpace()
void	stepBack() Makes one step back in the buffer.
void	switchParser(NTProductions newParser)
SourcePosition	toTextPosition(int position) Returns the position of the readable input source code.

Methods inherited from class genscript.syntax.tokens.Tokenizer

getBottomLevelSource, getFilename, isTokenStackEmpty, pollTokenValue, pushTokenValue, tokenStackToString

Methods inherited from class genscript.parse.IndexReader

canStepBack, getBottomLevelIntervalStartPos, getBottomLevelPosition, getErrorPosition, hasErrorPosition, intervalToString, parseStart, setErrorPosition, setIntervalPositions

Methods inherited from class genscript.parse.ReaderStack

getMarkedStackSize, getResultStackSize, getStackMark, popResult, pushResult, resetStack, resultStackToString, setStackMark

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlexibleTokenizer

public FlexibleTokenizer(IndexReader innerSource)

Method Detail

getPosition

public int getPosition()

Specified by:

getPosition in interface ReaderInterface

Specified by:

getPosition in class IndexReader

Returns:

the current position of the marker

setPosition

public void setPosition(int position)

Description copied from class: IndexReader

Sets the Position of the inner Buffer. The next read() call will start from the set position.

Specified by:

setPosition in interface ReaderInterface

Specified by:

setPosition in class IndexReader

Parameters:

position - the position to set the marker

isEndOfBuffer

public final boolean isEndOfBuffer()

Specified by:

isEndOfBuffer in interface ReaderInterface

Specified by:

isEndOfBuffer in class IndexReader

Returns:

True, if and only if the end of the inner Buffer is reached

skipSpace

public void skipSpace()

Specified by:

skipSpace in class IndexReader

setTokenSet

public void setTokenSet(TokenSet tokenSet)

read

public Token read()
           throws ParseException

Description copied from interface: ReaderInterface

Returns the next index in the source code and moves the marker.

Specified by:

read in interface ReaderInterface

Specified by:

read in class IndexReader

Returns:

the next index

Throws:

ParseException

stepBack

public void stepBack()

Description copied from class: IndexReader

Makes one step back in the buffer.

Specified by:

stepBack in class IndexReader

clear

public void clear()

Description copied from class: IndexReader

Deletes all the read information. The read position remains the same.

Overrides:

clear in class Tokenizer

switchParser

public void switchParser(NTProductions newParser)
                  throws ParseException

Overrides:

switchParser in class IndexReader

Throws:

ParseException

newReader

public IndexReader newReader()

Description copied from class: IndexReader

Clones this Reader without copying the buffer itself

Specified by:

newReader in class IndexReader

toTextPosition

public SourcePosition toTextPosition(int position)

Description copied from interface: ReaderInterface

Returns the position of the readable input source code.

Specified by:

toTextPosition in interface ReaderInterface

Specified by:

toTextPosition in class IndexReader

Parameters:

position - the position to transform into the text position

Returns:

the text position

getBottomLevelPosition

public int getBottomLevelPosition(int topLevelPosition)

Specified by:

getBottomLevelPosition in class IndexReader

intervalToString

public java.lang.String intervalToString(int startShift,
                                         int endShift)

Specified by:

intervalToString in class IndexReader

getIntervalStartPos

public final int getIntervalStartPos()

Specified by:

getIntervalStartPos in class IndexReader

setIntervalStartPos

public void setIntervalStartPos(int startPos)

Specified by:

setIntervalStartPos in class IndexReader

getIntervalEndPos

public final int getIntervalEndPos()

Specified by:

getIntervalEndPos in class IndexReader

setIntervalEndPos

public void setIntervalEndPos(int endPos)

Specified by:

setIntervalEndPos in class IndexReader

getBufferSize

public int getBufferSize()

Specified by:

getBufferSize in interface ReaderInterface

Specified by:

getBufferSize in class IndexReader

Returns:

the number of tokens/bytes of the buffer