Class SynonymGraphFilter
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.TokenFilter
-
- org.apache.lucene.analysis.synonym.SynonymGraphFilter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class SynonymGraphFilter extends TokenFilter
Applies single- or multi-token synonyms from aSynonymMap
to an incomingTokenStream
, producing a fully correct graph output. This is a replacement forSynonymFilter
, which produces incorrect graphs for multi-token synonyms.However, if you use this during indexing, you must follow it with
FlattenGraphFilter
to squash tokens on top of one another likeSynonymFilter
, because the indexer can't directly consume a graph. To get fully correct positional queries when your synonym replacements are multiple tokens, you should instead apply synonyms using thisTokenFilter
at query time and translate the resulting graph to aTermAutomatonQuery
e.g. usingTokenStreamToTermAutomatonQuery
.NOTE: this cannot consume an incoming graph; results will be undefined.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) static class
SynonymGraphFilter.BufferedInputToken
(package private) static class
SynonymGraphFilter.BufferedOutputToken
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description private ByteArrayDataInput
bytesReader
private int
captureCount
private boolean
finished
private FST<BytesRef>
fst
private FST.BytesReader
fstReader
private boolean
ignoreCase
private int
lastNodeOut
private boolean
liveToken
private RollingBuffer<SynonymGraphFilter.BufferedInputToken>
lookahead
private int
lookaheadNextRead
private int
lookaheadNextWrite
private int
matchEndOffset
private int
matchStartOffset
private int
maxLookaheadUsed
private int
nextNodeOut
private OffsetAttribute
offsetAtt
private java.util.LinkedList<SynonymGraphFilter.BufferedOutputToken>
outputBuffer
private PositionIncrementAttribute
posIncrAtt
private PositionLengthAttribute
posLenAtt
private FST.Arc<BytesRef>
scratchArc
private BytesRef
scratchBytes
private CharsRefBuilder
scratchChars
private SynonymMap
synonyms
private CharTermAttribute
termAtt
static java.lang.String
TYPE_SYNONYM
private TypeAttribute
typeAtt
-
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description SynonymGraphFilter(TokenStream input, SynonymMap synonyms, boolean ignoreCase)
Apply previously built synonyms to incoming tokens.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description private void
bufferOutputTokens(BytesRef bytes, int matchInputLength)
Expands the output graph into the necessary tokens, adding synonyms as side paths parallel to the input tokens, and buffers them in the output token buffer.private void
capture()
Buffers the current input token into lookahead buffer.(package private) int
getCaptureCount()
(package private) int
getMaxLookaheadUsed()
boolean
incrementToken()
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.private boolean
parse()
Scans the next input token(s) to see if a synonym matches.private void
releaseBufferedToken()
void
reset()
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.-
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Field Detail
-
TYPE_SYNONYM
public static final java.lang.String TYPE_SYNONYM
- See Also:
- Constant Field Values
-
termAtt
private final CharTermAttribute termAtt
-
posIncrAtt
private final PositionIncrementAttribute posIncrAtt
-
posLenAtt
private final PositionLengthAttribute posLenAtt
-
typeAtt
private final TypeAttribute typeAtt
-
offsetAtt
private final OffsetAttribute offsetAtt
-
synonyms
private final SynonymMap synonyms
-
ignoreCase
private final boolean ignoreCase
-
fstReader
private final FST.BytesReader fstReader
-
bytesReader
private final ByteArrayDataInput bytesReader
-
scratchBytes
private final BytesRef scratchBytes
-
scratchChars
private final CharsRefBuilder scratchChars
-
outputBuffer
private final java.util.LinkedList<SynonymGraphFilter.BufferedOutputToken> outputBuffer
-
nextNodeOut
private int nextNodeOut
-
lastNodeOut
private int lastNodeOut
-
maxLookaheadUsed
private int maxLookaheadUsed
-
captureCount
private int captureCount
-
liveToken
private boolean liveToken
-
matchStartOffset
private int matchStartOffset
-
matchEndOffset
private int matchEndOffset
-
finished
private boolean finished
-
lookaheadNextRead
private int lookaheadNextRead
-
lookaheadNextWrite
private int lookaheadNextWrite
-
lookahead
private RollingBuffer<SynonymGraphFilter.BufferedInputToken> lookahead
-
-
Constructor Detail
-
SynonymGraphFilter
public SynonymGraphFilter(TokenStream input, SynonymMap synonyms, boolean ignoreCase)
Apply previously built synonyms to incoming tokens.- Parameters:
input
- input tokenstreamsynonyms
- synonym mapignoreCase
- case-folds input for matching withCharacter.toLowerCase(int)
. Note, if you set this to true, it's your responsibility to lowercase the input entries when you create theSynonymMap
-
-
Method Detail
-
incrementToken
public boolean incrementToken() throws java.io.IOException
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
java.io.IOException
-
releaseBufferedToken
private void releaseBufferedToken() throws java.io.IOException
- Throws:
java.io.IOException
-
parse
private boolean parse() throws java.io.IOException
Scans the next input token(s) to see if a synonym matches. Returns true if a match was found.- Throws:
java.io.IOException
-
bufferOutputTokens
private void bufferOutputTokens(BytesRef bytes, int matchInputLength)
Expands the output graph into the necessary tokens, adding synonyms as side paths parallel to the input tokens, and buffers them in the output token buffer.
-
capture
private void capture()
Buffers the current input token into lookahead buffer.
-
reset
public void reset() throws java.io.IOException
Description copied from class:TokenFilter
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call
super.reset()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on further usage).NOTE: The default implementation chains the call to the input TokenStream, so be sure to call
super.reset()
when overriding this method.- Overrides:
reset
in classTokenFilter
- Throws:
java.io.IOException
-
getCaptureCount
int getCaptureCount()
-
getMaxLookaheadUsed
int getMaxLookaheadUsed()
-
-