The problem/use-case that the feature addresses
The current mechanism for defragging non-key data implicitly assumes that modules defrag their non-key data in a single call. Search simply can't do this without triggering unacceptably long periods of main thread unavailability.
In solving this problem, a large issue is the asynchronous nature of the search indexes. Modifying search indexes (in order to do defrag) is controlled by a time-slicing mechanism -- indexes are either writing or reading, never a mixture. During read periods, the indexes are immutable and query operations perform no fine-grained locking. Thus if the indexes are currently doing a read operation and the main thread decides that now is the time to perform defrag -- nothing can happen until the end of the current read period -- easily 10-20 mSecs. Having the main thread wait this long to start a defrag cycle is unacceptable.
Thus some degree of decoupling of the search index defrag from the database defrag is required. Decoupling has to be examined in at least two time horizons. The low-level locking-oriented decoupling described above must be addressed, but the higher level policy of the defrag state machine may also require revision in order to avoid unnecessary main thread defrag (main thread thinks more defrag is needed, but search is still defragging). Associated with the high-level decoupling is the issue of search CPU resource allocation, how much search CPU time should be devoted to defrag vs normal processing. Is this allocation controlled by the current defrag policy or is some other, independent, policy more appropriate?
The problem/use-case that the feature addresses
The current mechanism for defragging non-key data implicitly assumes that modules defrag their non-key data in a single call. Search simply can't do this without triggering unacceptably long periods of main thread unavailability.
In solving this problem, a large issue is the asynchronous nature of the search indexes. Modifying search indexes (in order to do defrag) is controlled by a time-slicing mechanism -- indexes are either writing or reading, never a mixture. During read periods, the indexes are immutable and query operations perform no fine-grained locking. Thus if the indexes are currently doing a read operation and the main thread decides that now is the time to perform defrag -- nothing can happen until the end of the current read period -- easily 10-20 mSecs. Having the main thread wait this long to start a defrag cycle is unacceptable.
Thus some degree of decoupling of the search index defrag from the database defrag is required. Decoupling has to be examined in at least two time horizons. The low-level locking-oriented decoupling described above must be addressed, but the higher level policy of the defrag state machine may also require revision in order to avoid unnecessary main thread defrag (main thread thinks more defrag is needed, but search is still defragging). Associated with the high-level decoupling is the issue of search CPU resource allocation, how much search CPU time should be devoted to defrag vs normal processing. Is this allocation controlled by the current defrag policy or is some other, independent, policy more appropriate?