Class CompressingStoredFieldsWriter

    • Field Detail

      • FIELDS_EXTENSION

        public static final java.lang.String FIELDS_EXTENSION
        Extension of stored fields file
        See Also:
        Constant Field Values
      • INDEX_EXTENSION_PREFIX

        public static final java.lang.String INDEX_EXTENSION_PREFIX
        Extension of stored fields index
        See Also:
        Constant Field Values
      • INDEX_CODEC_NAME

        public static final java.lang.String INDEX_CODEC_NAME
        Codec name for the index.
        See Also:
        Constant Field Values
      • TYPE_BITS

        static final int TYPE_BITS
      • TYPE_MASK

        static final int TYPE_MASK
      • segment

        private final java.lang.String segment
      • chunkSize

        private final int chunkSize
      • maxDocsPerChunk

        private final int maxDocsPerChunk
      • numStoredFields

        private int[] numStoredFields
      • endOffsets

        private int[] endOffsets
      • docBase

        private int docBase
      • numBufferedDocs

        private int numBufferedDocs
      • numChunks

        private long numChunks
      • numDirtyChunks

        private long numDirtyChunks
      • numStoredFieldsInDoc

        private int numStoredFieldsInDoc
      • NEGATIVE_ZERO_FLOAT

        static final int NEGATIVE_ZERO_FLOAT
      • NEGATIVE_ZERO_DOUBLE

        static final long NEGATIVE_ZERO_DOUBLE
      • BULK_MERGE_ENABLED_SYSPROP

        static final java.lang.String BULK_MERGE_ENABLED_SYSPROP
      • BULK_MERGE_ENABLED

        static final boolean BULK_MERGE_ENABLED
    • Constructor Detail

      • CompressingStoredFieldsWriter

        CompressingStoredFieldsWriter​(Directory directory,
                                      SegmentInfo si,
                                      java.lang.String segmentSuffix,
                                      IOContext context,
                                      java.lang.String formatName,
                                      CompressionMode compressionMode,
                                      int chunkSize,
                                      int maxDocsPerChunk,
                                      int blockShift)
                               throws java.io.IOException
        Sole constructor.
        Throws:
        java.io.IOException
    • Method Detail

      • close

        public void close()
                   throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in class StoredFieldsWriter
        Throws:
        java.io.IOException
      • finishDocument

        public void finishDocument()
                            throws java.io.IOException
        Description copied from class: StoredFieldsWriter
        Called when a document and all its fields have been added.
        Overrides:
        finishDocument in class StoredFieldsWriter
        Throws:
        java.io.IOException
      • saveInts

        private static void saveInts​(int[] values,
                                     int length,
                                     DataOutput out)
                              throws java.io.IOException
        Throws:
        java.io.IOException
      • writeHeader

        private void writeHeader​(int docBase,
                                 int numBufferedDocs,
                                 int[] numStoredFields,
                                 int[] lengths,
                                 boolean sliced)
                          throws java.io.IOException
        Throws:
        java.io.IOException
      • triggerFlush

        private boolean triggerFlush()
      • flush

        private void flush()
                    throws java.io.IOException
        Throws:
        java.io.IOException
      • writeZFloat

        static void writeZFloat​(DataOutput out,
                                float f)
                         throws java.io.IOException
        Writes a float in a variable-length format. Writes between one and five bytes. Small integral values typically take fewer bytes.

        ZFloat --> Header, Bytes*?

        • Header --> Uint8. When it is equal to 0xFF then the value is negative and stored in the next 4 bytes. Otherwise if the first bit is set then the other bits in the header encode the value plus one and no other bytes are read. Otherwise, the value is a positive float value whose first byte is the header, and 3 bytes need to be read to complete it.
        • Bytes --> Potential additional bytes to read depending on the header.
        Throws:
        java.io.IOException
      • writeZDouble

        static void writeZDouble​(DataOutput out,
                                 double d)
                          throws java.io.IOException
        Writes a float in a variable-length format. Writes between one and five bytes. Small integral values typically take fewer bytes.

        ZFloat --> Header, Bytes*?

        • Header --> Uint8. When it is equal to 0xFF then the value is negative and stored in the next 8 bytes. When it is equal to 0xFE then the value is stored as a float in the next 4 bytes. Otherwise if the first bit is set then the other bits in the header encode the value plus one and no other bytes are read. Otherwise, the value is a positive float value whose first byte is the header, and 7 bytes need to be read to complete it.
        • Bytes --> Potential additional bytes to read depending on the header.
        Throws:
        java.io.IOException
      • writeTLong

        static void writeTLong​(DataOutput out,
                               long l)
                        throws java.io.IOException
        Writes a long in a variable-length format. Writes between one and ten bytes. Small values or values representing timestamps with day, hour or second precision typically require fewer bytes.

        ZLong --> Header, Bytes*?

        • Header --> The first two bits indicate the compression scheme:
          • 00 - uncompressed
          • 01 - multiple of 1000 (second)
          • 10 - multiple of 3600000 (hour)
          • 11 - multiple of 86400000 (day)
          Then the next bit is a continuation bit, indicating whether more bytes need to be read, and the last 5 bits are the lower bits of the encoded value. In order to reconstruct the value, you need to combine the 5 lower bits of the header with a vLong in the next bytes (if the continuation bit is set to 1). Then zigzag-decode it and finally multiply by the multiple corresponding to the compression scheme.
        • Bytes --> Potential additional bytes to read depending on the header.
        Throws:
        java.io.IOException
      • tooDirty

        boolean tooDirty​(CompressingStoredFieldsReader candidate)
        Returns true if we should recompress this reader, even though we could bulk merge compressed data

        The last chunk written for a segment is typically incomplete, so without recompressing, in some worst-case situations (e.g. frequent reopen with tiny flushes), over time the compression ratio can degrade. This is a safety switch.