Class IndexFileDeleter
- All Implemented Interfaces:
Closeable
,AutoCloseable
The same directory file may be referenced by more than one IndexCommit, i.e. more than one SegmentInfos. Therefore we count how many commits reference each file. When all the commits referencing a certain file have been deleted, the refcount for that file becomes zero, and the file is deleted.
A separate deletion policy interface (IndexDeletionPolicy) is consulted on creation (onInit) and once per commit (onCommit), to decide when a commit should be removed.
It is the business of the IndexDeletionPolicy to choose when to delete commit points. The actual mechanics of file deletion, retrying, etc, derived from the deletion of commit points is the business of the IndexFileDeleter.
The current default deletion policy is KeepOnlyLastCommitDeletionPolicy
, which removes
all prior commits when a new commit has completed. This matches the behavior before 2.2.
Note that you must hold the write.lock before instantiating this class. It opens segments_N file(s) directly with no retry logic.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static final class
Holds details for each commit point.private static final class
Tracks the reference count for a single index file: -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate List<IndexFileDeleter.CommitPoint>
private List<IndexFileDeleter.CommitPoint>
private final Directory
private final Directory
private final InfoStream
private SegmentInfos
private final IndexDeletionPolicy
private Map<String,
IndexFileDeleter.RefCount> (package private) final boolean
static boolean
Change to true to see details of reference counts when infoStream is enabledprivate final IndexWriter
-
Constructor Summary
ConstructorsConstructorDescriptionIndexFileDeleter
(String[] files, Directory directoryOrig, Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, boolean initialIndexExists, boolean isReaderInit) Initialize the deleter: find all previous commits in the Directory, incref the files they reference, call the policy to let it delete commits. -
Method Summary
Modifier and TypeMethodDescription(package private) boolean
void
checkpoint
(SegmentInfos segmentInfos, boolean isCommit) For definition of "check point" see IndexWriter comments: "Clarification: Check Points (and commits)".void
close()
private boolean
Returns true if the file should now be deleted.(package private) void
decRef
(Collection<String> files) Decrefs all provided files, even on exception; throws first exception hit, if any.(package private) void
decRef
(SegmentInfos segmentInfos) private void
Remove the CommitPoints in the commitsToDelete List by DecRef'ing all files from each SegmentInfos.private void
deleteFile
(String fileName) private void
deleteFiles
(Collection<String> names) (package private) void
deleteNewFiles
(Collection<String> files) Deletes the specified files, but only if they are new (have not yet been incref'd).(package private) void
boolean
private IndexFileDeleter.RefCount
getRefCount
(String fileName) (package private) void
(package private) void
incRef
(Collection<String> files) (package private) void
incRef
(SegmentInfos segmentInfos, boolean isCommit) (package private) static void
inflateGens
(SegmentInfos infos, Collection<String> files, InfoStream infoStream) Set all gens beyond what we currently see in the directory, to avoid double-write in cases where the previous IndexWriter did not gracefully close/rollback (e.g.(package private) boolean
isClosed()
private boolean
locked()
(package private) void
refresh()
Writer calls this when it has hit an error and had to roll back, to tell us that there may now be unreferenced files in the filesystem.(package private) void
Revisits theIndexDeletionPolicy
by calling itsIndexDeletionPolicy.onCommit(List)
again with the known commits.
-
Field Details
-
refCounts
-
commits
-
lastFiles
-
commitsToDelete
-
infoStream
-
directoryOrig
-
directory
-
policy
-
startingCommitDeleted
final boolean startingCommitDeleted -
lastSegmentInfos
-
VERBOSE_REF_COUNTS
public static boolean VERBOSE_REF_COUNTSChange to true to see details of reference counts when infoStream is enabled -
writer
-
-
Constructor Details
-
IndexFileDeleter
public IndexFileDeleter(String[] files, Directory directoryOrig, Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, boolean initialIndexExists, boolean isReaderInit) throws IOException Initialize the deleter: find all previous commits in the Directory, incref the files they reference, call the policy to let it delete commits. This will remove any files not referenced by any of the commits.- Throws:
IOException
- if there is a low-level IO error
-
-
Method Details
-
locked
private boolean locked() -
inflateGens
Set all gens beyond what we currently see in the directory, to avoid double-write in cases where the previous IndexWriter did not gracefully close/rollback (e.g. os/machine crashed or lost power). -
ensureOpen
- Throws:
AlreadyClosedException
-
isClosed
boolean isClosed() -
deleteCommits
Remove the CommitPoints in the commitsToDelete List by DecRef'ing all files from each SegmentInfos.- Throws:
IOException
-
refresh
Writer calls this when it has hit an error and had to roll back, to tell us that there may now be unreferenced files in the filesystem. So we re-list the filesystem and delete such files. If segmentName is non-null, we will only delete files corresponding to that segment.- Throws:
IOException
-
close
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
-
assertCommitsAreNotDeleted
-
revisitPolicy
Revisits theIndexDeletionPolicy
by calling itsIndexDeletionPolicy.onCommit(List)
again with the known commits. This is useful in cases where a deletion policy which holds onto index commits is used. The application may know that some commits are not held by the deletion policy anymore and callIndexWriter.deleteUnusedFiles()
, which will attempt to delete the unused commits again.- Throws:
IOException
-
checkpoint
For definition of "check point" see IndexWriter comments: "Clarification: Check Points (and commits)".Writer calls this when it has made a "consistent change" to the index, meaning new files are written to the index and the in-memory SegmentInfos have been modified to point to those files.
This may or may not be a commit (segments_N may or may not have been written).
We simply incref the files referenced by the new SegmentInfos and decref the files we had previously seen (if any).
If this is a commit, we also call the policy to give it a chance to remove other commits. If any commits are removed, we decref their files as well.
- Throws:
IOException
-
incRef
- Throws:
IOException
-
incRef
-
incRef
-
decRef
Decrefs all provided files, even on exception; throws first exception hit, if any.- Throws:
IOException
-
decRef
Returns true if the file should now be deleted. -
decRef
- Throws:
IOException
-
exists
-
getRefCount
-
deleteNewFiles
Deletes the specified files, but only if they are new (have not yet been incref'd).- Throws:
IOException
-
deleteFiles
- Throws:
IOException
-
deleteFile
- Throws:
IOException
-