Package org.apache.cassandra.index.sasi.analyzer

Interface StandardTokenizerInterface

public interface StandardTokenizerInterface
Internal interface for supporting versioned grammars.

  • Method Summary

    Modifier and Type
    Method
    Description
    char[]
    getArray()
     
    byte[]
    getBytes()
     
    int
    getNextToken()
    Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
    String
    getText()
     
    long
    yychar()
    Returns the current position.
    int
    yylength()
    Returns the length of the matched text region.
    void
    yyreset(Reader reader)
    Resets the scanner to read from a new input stream.

  • Method Details

    • getText

      String getText()

    • getArray

      char[] getArray()

    • getBytes

      byte[] getBytes()

    • yychar

      long yychar()
      Returns the current position.

    • yylength

      int yylength()
      Returns the length of the matched text region.

    • getNextToken

      int getNextToken() throws IOException
      Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
      Returns:
      the next token, StandardTokenizerImpl.YYEOF on end of stream
      Throws:
      IOException - if any I/O-Error occurs

    • yyreset

      void yyreset(Reader reader)
      Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.
      Parameters:
      reader - the new input stream