Class ICUTransformFilter
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.TokenFilter
-
- org.apache.lucene.analysis.icu.ICUTransformFilter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class ICUTransformFilter extends TokenFilter
ATokenFilter
that transforms text with ICU.ICU provides text-transformation functionality via its Transliteration API. Although script conversion is its most common use, a Transliterator can actually perform a more general class of tasks. In fact, Transliterator defines a very general API which specifies only that a segment of the input text is replaced by new text. The particulars of this conversion are determined entirely by subclasses of Transliterator.
Some useful transformations for search are built-in:
- Conversion from Traditional to Simplified Chinese characters
- Conversion from Hiragana to Katakana
- Conversion from Fullwidth to Halfwidth forms.
- Script conversions, for example Serbian Cyrillic to Latin
Example usage:
stream = new ICUTransformFilter(stream, Transliterator.getInstance("Traditional-Simplified"));
For more details, see the ICU User Guide.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) static class
ICUTransformFilter.ReplaceableTermAttribute
Wrap aCharTermAttribute
with the Replaceable API.-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description private com.ibm.icu.text.Transliterator.Position
position
private ICUTransformFilter.ReplaceableTermAttribute
replaceableAttribute
private CharTermAttribute
termAtt
private com.ibm.icu.text.Transliterator
transform
-
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description ICUTransformFilter(TokenStream input, com.ibm.icu.text.Transliterator transform)
Create a new ICUTransformFilter that transforms text on the given stream.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
incrementToken()
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.-
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end, reset
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Field Detail
-
transform
private final com.ibm.icu.text.Transliterator transform
-
position
private final com.ibm.icu.text.Transliterator.Position position
-
termAtt
private final CharTermAttribute termAtt
-
replaceableAttribute
private final ICUTransformFilter.ReplaceableTermAttribute replaceableAttribute
-
-
Constructor Detail
-
ICUTransformFilter
public ICUTransformFilter(TokenStream input, com.ibm.icu.text.Transliterator transform)
Create a new ICUTransformFilter that transforms text on the given stream.- Parameters:
input
-TokenStream
to filter.transform
- Transliterator to transform the text.
-
-
Method Detail
-
incrementToken
public boolean incrementToken() throws java.io.IOException
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
java.io.IOException
-
-