From 4b588430c8e2f199957aa95b98ccf0c86786d5ce Mon Sep 17 00:00:00 2001 From: popalot2 <69579435+popalot2@users.noreply.github.com> Date: Fri, 21 Aug 2026 12:14:03 +0300 Subject: [PATCH 1/7] fix for indextool --apply-killlists crashes repeatedly with SIGSEGV #4837 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The main index was created with lookup format version 67, while the delta uses version 71. indextool --apply-killlists calls IndexFiles_c::CheckHeader() and then uses GetVersion(): manticoresearch/src/indextool.cpp:883 For JSON .sph headers, CheckHeader() previously returned success without reading index_format_version. The version therefore remained at its default—currently 71. The v67 .spt was consequently opened as v71. Version 71 added an 8-byte UUID lookup offset: manticoresearch/src/docidlookup.cpp:995 That shifted the checkpoint table by eight bytes. Every checkpoint was then interpreted as: - offset → base DocID - next checkpoint’s base DocID → data offset This explains the apparently valid DocID combined with the impossible row ID 4076938452. It also reproduces the bad .spt pointer offset seen in GDB. The atomic operation in DeadRowMap_c::Set() is only where the corrupted row ID finally causes an out-of-bounds access. killlist.h and the atomic operation are not the root cause. Also, m_iCount = 36330127 for the dead map is a count of DWORD elements, not bytes. Its actual size is approximately 145 MB. Row ID 4076938452 addresses approximately 510 MB from its base. ### Affected components - indextool --apply-killlists: directly affected, especially when processing indexes with different JSON-header format versions. - searchd: not affected by this specific bug. Normal index loading parses index_format_version correctly before constructing lookup readers. - indexer: not affected; it writes the current format and does not use this broken version-discovery path. - Index rotation calls CheckHeader(), but only to validate which header exists; it does not consume the stale version in the failing manner. - indextool --check: passes because its checker independently parses the JSON header version correctly at manticoresearch/src/indexcheck.cpp:646. ### Fix IndexFiles_c::CheckHeader() now parses JSON headers and stores the actual index_format_version: manticoresearch/src/indexfiles.cpp:267 It also validates that the version is supported. Therefore the v67 target lookup is read as v67 and the v71 delta as v71. No reindexing of the .spt files is required. They are structurally valid. However, because the failed run had the .spm dead-row map writable and memory-mapped, restore the target .spm from a pre-crash backup before rerunning if possible. The crashing write itself was outside the mapping, but earlier incorrect in-range row IDs could theoretically have set wrong dead bits. is IndexFiles_c::CheckHeader() only used in indextool? why this problem didn't happen in other tools? • No. IndexFiles_c::CheckHeader() has two active callers: 1. indextool --apply-killlists 2. Searchd’s index-rotation file checker The key difference is what callers do with the result. ### Why indextool crashed ApplyKilllists() uses both CheckHeader() and the version stored by it: IndexFiles_c tIndexFiles (...); if ( !tIndexFiles.CheckHeader() ) continue; tIndex.m_uVersion = tIndexFiles.GetVersion(); manticoresearch/src/indextool.cpp:883 Before the fix, a JSON header caused CheckHeader() to return true without updating m_uVersion. It remained at the compiled-in current version, 71. That incorrect version was passed directly into: LookupReaderIterator_c ( lookupData, tIndex.m_uVersion ); The v67 lookup was therefore decoded using the v71 layout, producing the corrupt row ID. ### Why searchd rotation did not hit it The other caller is: manticoresearch/src/index_rotator.cpp:20 It effectively does: IndexFiles_c(sPath).CheckHeader() IndexFiles_c(sPath).CheckHeader(".new") Here IndexFiles_c is temporary. Rotation only needs the boolean result to determine whether the current and/or .new header exists and is acceptable. It does not call GetVersion() on that object. Therefore the stale version value was discarded immediately. The fix still improves rotation because it now performs real JSON version validation, but rotation was not feeding that version into a lookup reader. ### Why normal searchd operations worked Searchd does not depend on IndexFiles_c::CheckHeader() to determine the loaded index version. Its normal index loader parses the complete JSON .sph header independently: manticoresearch/src/sphinx.cpp:9767 It correctly executes: m_uVersion = (DWORD)Int ( tBson.ChildByName ( "index_format_version" ) ); Later, searchd creates lookup readers using that correctly loaded m_uVersion. Consequently, searchd can apply kills internally with KillByLookup() safely—the same underlying kill logic is used, but the supplied format version is correct. ### Why indextool --check worked The index checker has another independent JSON-header parser: manticoresearch/src/indexcheck.cpp:646 It also reads index_format_version correctly. This is why both indexes passed --check: the files were valid, and the checker did not use the defective version-discovery path. ### Why indexer was unaffected indexer creates current-format indexes. It does not use this CheckHeader() → GetVersion() sequence to reopen mixed-version lookup files. So the precise vulnerable pattern was: JSON header → IndexFiles_c::CheckHeader() → GetVersion() → LookupReaderIterator_c Among active callers, only indextool --apply-killlists used that complete sequence. --- src/indexfiles.cpp | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/src/indexfiles.cpp b/src/indexfiles.cpp index c7c2ee38b2..1dbd7dc33a 100644 --- a/src/indexfiles.cpp +++ b/src/indexfiles.cpp @@ -16,6 +16,7 @@ #include "fileio.h" #include "fileutils.h" #include "sphinxint.h" +#include "sphinxjson.h" #include "tokenizer/tokenizer.h" static IndexFileExt_t g_dIndexFilesExts[SPH_EXT_TOTAL] = @@ -272,10 +273,46 @@ bool IndexFiles_c::CheckHeader ( const char * sType ) if ( !rdHeader.Open ( sPath, m_sLastError ) ) return false; - // check magic header + // Check the legacy binary magic first. JSON headers do not have that magic; + // they are identified by their opening brace below. auto uMagic = rdHeader.GetDword(); - if ( dBuffer[0] == '{' ) // that is new style json header, no need to check further... + if ( dBuffer[0] == '{' ) // new style JSON header + { + // Do not merely report that a JSON header exists. CheckHeader() also owns + // the invariant that GetVersion() returns the version read from disk. + // In particular, indextool --apply-killlists passes that value to the + // lookup reader. If m_uVersion is left at INDEX_FORMAT_VERSION, an older + // lookup can be decoded with the current layout. Version 71 added an + // SphOffset_t to the .spt preamble, so making that mistake shifts every + // checkpoint and eventually produces invalid row IDs and memory offsets. + CSphVector dData; + if ( !sphJsonParse ( dData, sPath, m_sLastError ) ) + return false; + + bson::Bson_c tBson ( dData ); + if ( tBson.IsEmpty() || !tBson.IsAssoc() ) + { + m_sLastError.SetSprintf ( "invalid JSON index header %s", sPath.cstr() ); + return false; + } + + // Keep the field name and layout-version rules in sync with the full header + // readers in sphinx.cpp and indexcheck.cpp. A missing, invalid, or future + // version must fail instead of falling back to the running binary's version. + // This is only the broad format check; callers may impose a newer minimum. + m_uVersion = (DWORD)bson::Int ( tBson.ChildByName ( "index_format_version" ) ); + if ( m_uVersion<=1 || m_uVersion>INDEX_FORMAT_VERSION ) + { + m_sLastError.SetSprintf ( "%s is v.%u, binary is v.%u", sPath.cstr(), m_uVersion, INDEX_FORMAT_VERSION ); + return false; + } + + // JSON detection currently assumes that '{' is the first byte. If headers + // ever permit a BOM or leading whitespace, update this probe together with + // the initial read above; otherwise a valid JSON header will be handled as + // a legacy binary header. return true; + } const char* sMsg = CheckFmtMagic ( uMagic ); if ( sMsg ) From 18d9b24061c13e72f30c5deb2d8d769fc3317427 Mon Sep 17 00:00:00 2001 From: popalot2 <69579435+popalot2@users.noreply.github.com> Date: Fri, 21 Aug 2026 12:52:24 +0300 Subject: [PATCH 2/7] 2nd try to fix for indextool --apply-killlists crashes repeatedly with SIGSEGV #4837 sphJsonParse in main returns an enum, instead of bool in the latest release, fixed sphJsonParse ( dData, sPath, m_sLastError )!=JsonFileParse_e::OK --- src/indexfiles.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/indexfiles.cpp b/src/indexfiles.cpp index 1dbd7dc33a..afacdfd923 100644 --- a/src/indexfiles.cpp +++ b/src/indexfiles.cpp @@ -286,7 +286,7 @@ bool IndexFiles_c::CheckHeader ( const char * sType ) // SphOffset_t to the .spt preamble, so making that mistake shifts every // checkpoint and eventually produces invalid row IDs and memory offsets. CSphVector dData; - if ( !sphJsonParse ( dData, sPath, m_sLastError ) ) + if ( sphJsonParse ( dData, sPath, m_sLastError )!=JsonFileParse_e::OK ) return false; bson::Bson_c tBson ( dData ); From aa62148c54d835393ecc3352a97574de24aa4eee Mon Sep 17 00:00:00 2001 From: Sergey Nikolaev Date: Fri, 21 Aug 2026 18:45:16 +0700 Subject: [PATCH 3/7] test: cover JSON index header version detection --- src/gtests/gtests_json.cpp | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/gtests/gtests_json.cpp b/src/gtests/gtests_json.cpp index d03a636da0..d0144d42ba 100644 --- a/src/gtests/gtests_json.cpp +++ b/src/gtests/gtests_json.cpp @@ -15,6 +15,7 @@ #include #include "fileio.h" +#include "indexfiles.h" #include "json/cJSON.h" #include "sphinx.h" #include "sphinxjson.h" @@ -63,6 +64,28 @@ TEST_F ( JsonFileParseTest, ValidJson ) } +TEST ( IndexFiles, ReadsVersionFromJsonHeader ) +{ + CSphString sBase; + sBase.SetSprintf ( "__indexfiles_%d_json_header", GetOsProcessId() ); + CSphString sHeader; + sHeader.SetSprintf ( "%s.sph", sBase.cstr() ); + + CSphString sError; + CSphWriterNonThrottled tWriter; + ASSERT_TRUE ( tWriter.OpenFile ( sHeader, sError ) ) << sError.cstr(); + tWriter.PutBytes ( R"({"index_format_version":67})", strlen ( R"({"index_format_version":67})" ) ); + tWriter.CloseFile(); + ASSERT_FALSE ( tWriter.IsError() ); + + IndexFiles_c tFiles ( sBase ); + ASSERT_TRUE ( tFiles.CheckHeader() ) << tFiles.ErrorMsg(); + EXPECT_EQ ( tFiles.GetVersion(), 67U ); + + unlink ( sHeader.cstr() ); +} + + TEST_F ( JsonFileParseTest, NonJsonFormat ) { Write ( "not json" ); From a05dcad7b600e1a3e22ce113a445285915ae68d0 Mon Sep 17 00:00:00 2001 From: Sergey Nikolaev Date: Fri, 21 Aug 2026 21:43:15 +0700 Subject: [PATCH 4/7] fix: lazily read JSON index header version --- src/gtests/gtests_json.cpp | 4 ++- src/indexcheck.cpp | 43 ++++++++++++++--------- src/indexcheck.h | 3 ++ src/indexfiles.cpp | 72 ++++++++++++++------------------------ src/indexfiles.h | 14 +++++--- src/indextool.cpp | 10 ++++-- 6 files changed, 76 insertions(+), 70 deletions(-) diff --git a/src/gtests/gtests_json.cpp b/src/gtests/gtests_json.cpp index d0144d42ba..e087c7f2f4 100644 --- a/src/gtests/gtests_json.cpp +++ b/src/gtests/gtests_json.cpp @@ -80,7 +80,9 @@ TEST ( IndexFiles, ReadsVersionFromJsonHeader ) IndexFiles_c tFiles ( sBase ); ASSERT_TRUE ( tFiles.CheckHeader() ) << tFiles.ErrorMsg(); - EXPECT_EQ ( tFiles.GetVersion(), 67U ); + DWORD uVersion; + ASSERT_TRUE ( tFiles.GetVersion ( uVersion ) ) << tFiles.ErrorMsg(); + EXPECT_EQ ( uVersion, 67U ); unlink ( sHeader.cstr() ); } diff --git a/src/indexcheck.cpp b/src/indexcheck.cpp index 90d1baa641..497bff5730 100644 --- a/src/indexcheck.cpp +++ b/src/indexcheck.cpp @@ -622,6 +622,31 @@ bool DiskIndexChecker_c::Impl_c::ReadLegacyHeader ( CSphString& sError ) } +bool ReadIndexJsonHeaderVersion ( CSphVector & dData, const CSphString & sHeader, DWORD & uVersion, CSphString & sError ) +{ + using namespace bson; + + if ( sphJsonParse ( dData, sHeader, sError )!=JsonFileParse_e::OK ) + return false; + + Bson_c tBson ( dData ); + if ( tBson.IsEmpty() || !tBson.IsAssoc() ) + { + sError = "Something wrong read from json header - it is either empty, either not root object."; + return false; + } + + uVersion = (DWORD)Int ( tBson.ChildByName ( "index_format_version" ) ); + if ( uVersion<=1 || uVersion>INDEX_FORMAT_VERSION ) + { + sError.SetSprintf ( "%s is v.%u, binary is v.%u", sHeader.cstr(), uVersion, INDEX_FORMAT_VERSION ); + return false; + } + + return true; +} + + bool DiskIndexChecker_c::Impl_c::ReadHeader ( CSphString& sError ) { bool bHeaderIsJson; @@ -640,33 +665,19 @@ bool DiskIndexChecker_c::Impl_c::ReadHeader ( CSphString& sError ) auto sHeader = GetFilename ( SPH_EXT_SPH ); - const char* szHeader = sHeader.scstr(); using namespace bson; CSphVector dData; - if ( sphJsonParse ( dData, GetFilename ( SPH_EXT_SPH ), sError )!=JsonFileParse_e::OK ) + if ( !ReadIndexJsonHeaderVersion ( dData, sHeader, m_uVersion, sError ) ) return false; Bson_c tBson ( dData ); - if ( tBson.IsEmpty() || !tBson.IsAssoc() ) - { - sError = "Something wrong read from json header - it is either empty, either not root object."; - return false; - } - - // version - m_uVersion = (DWORD)Int ( tBson.ChildByName ( "index_format_version" ) ); - if ( m_uVersion <= 1 || m_uVersion > INDEX_FORMAT_VERSION ) - { - sError.SetSprintf ( "%s is v.%u, binary is v.%u", szHeader, m_uVersion, INDEX_FORMAT_VERSION ); - return false; - } // we don't support anything prior to v64 with json format DWORD uMinFormatVer = 64; if ( m_uVersion < uMinFormatVer ) { - sError.SetSprintf ( "tables prior to v.%u are no longer supported (use index_converter tool); %s is v.%u", uMinFormatVer, szHeader, m_uVersion ); + sError.SetSprintf ( "tables prior to v.%u are no longer supported (use index_converter tool); %s is v.%u", uMinFormatVer, sHeader.cstr(), m_uVersion ); return false; } diff --git a/src/indexcheck.h b/src/indexcheck.h index b0b341c0d4..ea51b87279 100644 --- a/src/indexcheck.h +++ b/src/indexcheck.h @@ -44,6 +44,9 @@ class DebugCheckError_i DebugCheckError_i* MakeDebugCheckError ( FILE* fp, DocID_t* pExtract ); +// Read and validate the format version from a JSON plain-index header. +bool ReadIndexJsonHeaderVersion ( CSphVector & dData, const CSphString & sHeader, DWORD & uVersion, CSphString & sError ); + // disk index checker class DiskIndexChecker_c { diff --git a/src/indexfiles.cpp b/src/indexfiles.cpp index afacdfd923..ab7c8beee2 100644 --- a/src/indexfiles.cpp +++ b/src/indexfiles.cpp @@ -16,7 +16,7 @@ #include "fileio.h" #include "fileutils.h" #include "sphinxint.h" -#include "sphinxjson.h" +#include "indexcheck.h" #include "tokenizer/tokenizer.h" static IndexFileExt_t g_dIndexFilesExts[SPH_EXT_TOTAL] = @@ -104,7 +104,7 @@ bool IndexFiles_c::HasAllFiles ( const char * sType ) { for ( const auto & dExt : g_dIndexFilesExts ) { - if ( m_uVersion dData; - if ( sphJsonParse ( dData, sPath, m_sLastError )!=JsonFileParse_e::OK ) - return false; - - bson::Bson_c tBson ( dData ); - if ( tBson.IsEmpty() || !tBson.IsAssoc() ) - { - m_sLastError.SetSprintf ( "invalid JSON index header %s", sPath.cstr() ); - return false; - } - - // Keep the field name and layout-version rules in sync with the full header - // readers in sphinx.cpp and indexcheck.cpp. A missing, invalid, or future - // version must fail instead of falling back to the running binary's version. - // This is only the broad format check; callers may impose a newer minimum. - m_uVersion = (DWORD)bson::Int ( tBson.ChildByName ( "index_format_version" ) ); - if ( m_uVersion<=1 || m_uVersion>INDEX_FORMAT_VERSION ) - { - m_sLastError.SetSprintf ( "%s is v.%u, binary is v.%u", sPath.cstr(), m_uVersion, INDEX_FORMAT_VERSION ); - return false; - } - - // JSON detection currently assumes that '{' is the first byte. If headers - // ever permit a BOM or leading whitespace, update this probe together with - // the initial read above; otherwise a valid JSON header will be handled as - // a legacy binary header. return true; - } const char* sMsg = CheckFmtMagic ( uMagic ); if ( sMsg ) { - m_sLastError.SetSprintf ( sMsg, sPath.cstr() ); + m_sLastError.SetSprintf ( sMsg, m_sHeaderPath.cstr() ); return false; } - // get version DWORD uVersion = rdHeader.GetDword (); if ( uVersion==0 || uVersion>INDEX_FORMAT_VERSION ) { - m_sLastError.SetSprintf ( "%s is v.%u, binary is v.%u", sPath.cstr(), uVersion, INDEX_FORMAT_VERSION ); + m_sLastError.SetSprintf ( "%s is v.%u, binary is v.%u", m_sHeaderPath.cstr(), uVersion, INDEX_FORMAT_VERSION ); return false; } m_uVersion = uVersion; @@ -333,6 +296,25 @@ bool IndexFiles_c::CheckHeader ( const char * sType ) } +bool IndexFiles_c::GetVersion ( DWORD & uVersion ) +{ + if ( !m_uVersion ) + { + if ( m_sHeaderPath.IsEmpty() && !CheckHeader() ) + return false; + + CSphVector dData; + DWORD uHeaderVersion; + if ( !ReadIndexJsonHeaderVersion ( dData, m_sHeaderPath, uHeaderVersion, m_sLastError ) ) + return false; + m_uVersion = uHeaderVersion; + } + + uVersion = *m_uVersion; + return true; +} + + bool IndexFiles_c::ReadKlistTargets ( StrVec_t & dTargets, const char * szType ) { CSphString sPath = FullPath ( sphGetExt(SPH_EXT_SPK), szType ); diff --git a/src/indexfiles.h b/src/indexfiles.h index 38b8cde46e..95b1ccbb48 100644 --- a/src/indexfiles.h +++ b/src/indexfiles.h @@ -16,6 +16,7 @@ #include "sphinxint.h" #include "indexfilebase.h" +#include #include enum ESphExt : BYTE @@ -59,16 +60,18 @@ const char* sphGetExt ( ESphExt eExt ); /// encapsulates all common actions over index files in general (copy/rename/delete etc.) class IndexFiles_c : public IndexFileBase_c { - DWORD m_uVersion = INDEX_FORMAT_VERSION; - CSphString m_sIndexName; // used for information purposes (logs) + std::optional m_uVersion; + CSphString m_sHeaderPath; + CSphString m_sIndexName; // used for information purposes (logs) CSphString m_sLastError; bool m_bFatal = false; // if fatal fail happened (unable to rename during rollback) CSphString FullPath ( const char * szExt, const CSphString& sSuffix = "", const CSphString& sBase = "" ); + DWORD GetVersionForFiles() const { return m_uVersion.value_or ( INDEX_FORMAT_VERSION ); } inline void SetName ( CSphString sIndex ) { m_sIndexName = std::move(sIndex); } public: IndexFiles_c() = default; - explicit IndexFiles_c ( CSphString sBase, const char* sIndex=nullptr, DWORD uVersion = INDEX_FORMAT_VERSION ) + explicit IndexFiles_c ( CSphString sBase, const char* sIndex=nullptr, std::optional uVersion = std::nullopt ) : IndexFileBase_c { std::move ( sBase ) } , m_uVersion ( uVersion ) { @@ -79,13 +82,14 @@ class IndexFiles_c : public IndexFileBase_c inline const char * ErrorMsg () const { return m_sLastError.cstr(); } inline bool IsFatal() const { return m_bFatal; } - // read .sph and adopt index version from there. + // check that .sph is readable and has a supported legacy header, if applicable. bool CheckHeader ( const char * sType="" ); // read the beginning of .spk and parse killlist targets bool ReadKlistTargets ( StrVec_t & dTargets, const char * sType="" ); - DWORD GetVersion() const { return m_uVersion; } + // lazily read the version from a JSON header when CheckHeader() could not obtain it. + bool GetVersion ( DWORD & uVersion ); // simple make decorated path, like '.old' -> /path/to/index.old CSphString MakePath ( const char * szSuffix = "" ); diff --git a/src/indextool.cpp b/src/indextool.cpp index 0c2c2d5f64..8217f2acdf 100644 --- a/src/indextool.cpp +++ b/src/indextool.cpp @@ -887,12 +887,16 @@ static void ApplyKilllists ( CSphConfig & hConf ) fprintf ( stdout, "WARNING: unable to index header for table %s\n", tIndex.m_sName.cstr() ); continue; } - tIndex.m_uVersion = tIndexFiles.GetVersion(); + if ( !tIndexFiles.GetVersion ( tIndex.m_uVersion ) ) + { + fprintf ( stdout, "WARNING: unable to read header version for table %s: %s\n", tIndex.m_sName.cstr(), tIndexFiles.ErrorMsg() ); + continue; + } // no lookups prior to v.54 - if ( tIndexFiles.GetVersion() < 54 ) + if ( tIndex.m_uVersion < 54 ) { - fprintf ( stdout, "WARNING: table '%s' version: %u, min supported is 54\n", tIndex.m_sName.cstr(), tIndexFiles.GetVersion() ); + fprintf ( stdout, "WARNING: table '%s' version: %u, min supported is 54\n", tIndex.m_sName.cstr(), tIndex.m_uVersion ); continue; } From 9ec0effdafa30b39b80aa05fb6ac3f4888d9119a Mon Sep 17 00:00:00 2001 From: Sergey Nikolaev Date: Fri, 21 Aug 2026 22:27:24 +0700 Subject: [PATCH 5/7] fix: validate JSON headers before rotation --- src/gtests/gtests_json.cpp | 33 +++++++++++++++++++++++++++++++++ src/index_rotator.cpp | 13 ++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/src/gtests/gtests_json.cpp b/src/gtests/gtests_json.cpp index e087c7f2f4..368deb544f 100644 --- a/src/gtests/gtests_json.cpp +++ b/src/gtests/gtests_json.cpp @@ -16,6 +16,7 @@ #include "fileio.h" #include "indexfiles.h" +#include "index_rotator.h" #include "json/cJSON.h" #include "sphinx.h" #include "sphinxjson.h" @@ -88,6 +89,38 @@ TEST ( IndexFiles, ReadsVersionFromJsonHeader ) } +TEST ( IndexRotator, IgnoresMalformedJsonNewHeader ) +{ + CSphString sBase; + sBase.SetSprintf ( "__indexfiles_%d_rotation", GetOsProcessId() ); + CSphString sHeader; + sHeader.SetSprintf ( "%s.sph", sBase.cstr() ); + CSphString sNewHeader; + sNewHeader.SetSprintf ( "%s.new.sph", sBase.cstr() ); + + auto fnWriteHeader = [] ( const CSphString & sFile, const char * sData ) + { + CSphString sError; + CSphWriterNonThrottled tWriter; + if ( !tWriter.OpenFile ( sFile, sError ) ) + return false; + tWriter.PutBytes ( sData, strlen ( sData ) ); + tWriter.CloseFile(); + return !tWriter.IsError(); + }; + + ASSERT_TRUE ( fnWriteHeader ( sHeader, R"({"index_format_version":67})" ) ); + ASSERT_TRUE ( fnWriteHeader ( sNewHeader, "{\n" ) ); + + CheckIndexRotate_c tCheck ( sBase ); + EXPECT_FALSE ( tCheck.RotateFromNew() ); + EXPECT_TRUE ( tCheck.RotateReenable() ); + + unlink ( sHeader.cstr() ); + unlink ( sNewHeader.cstr() ); +} + + TEST_F ( JsonFileParseTest, NonJsonFormat ) { Write ( "not json" ); diff --git a/src/index_rotator.cpp b/src/index_rotator.cpp index 6644027658..de71e64af1 100644 --- a/src/index_rotator.cpp +++ b/src/index_rotator.cpp @@ -15,9 +15,20 @@ #include "detail/indexlink.h" namespace { +inline bool CheckHeader ( const CSphString & sPath, const char * sType = "" ) +{ + IndexFiles_c tFiles ( sPath ); + if ( !tFiles.CheckHeader ( sType ) ) + return false; + + DWORD uVersion; + return tFiles.GetVersion ( uVersion ); +} + + inline RotateFrom_e Check ( const CSphString& sPath ) noexcept { - switch ( ( IndexFiles_c ( sPath ).CheckHeader() ? 1 : 0 ) + ( IndexFiles_c ( sPath ).CheckHeader ( ".new" ) ? 2 : 0 ) ) + switch ( ( CheckHeader ( sPath ) ? 1 : 0 ) + ( CheckHeader ( sPath, ".new" ) ? 2 : 0 ) ) { case 0: return RotateFrom_e::NONE; case 1: return RotateFrom_e::REENABLE; From 2d416aa5a9907d15c73dd9a909073fd09e8b865e Mon Sep 17 00:00:00 2001 From: Sergey Nikolaev Date: Fri, 21 Aug 2026 22:29:29 +0700 Subject: [PATCH 6/7] test: cover applying killlists to legacy JSON indexes --- test/indextool/CMakeLists.txt | 7 +++ test/indextool/test_apply_killlists.cmake | 72 +++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 test/indextool/test_apply_killlists.cmake diff --git a/test/indextool/CMakeLists.txt b/test/indextool/CMakeLists.txt index 97bc03a13e..9326602936 100644 --- a/test/indextool/CMakeLists.txt +++ b/test/indextool/CMakeLists.txt @@ -13,6 +13,13 @@ else () -P ${CMAKE_CURRENT_SOURCE_DIR}/test.cmake WORKING_DIRECTORY "${MANTICORE_BINARY_DIR}/test" ) SET_TESTS_PROPERTIES ( Perform_indextool PROPERTIES LABELS INDEXTOOL ) + add_test ( NAME Apply_killlists_with_legacy_json_header COMMAND ${CMAKE_COMMAND} + -D INDEXER=$ + -D INDEXTOOL=$ + -D LEGACY_FIXTURE=${CMAKE_SOURCE_DIR}/test/test_406/data + -P ${CMAKE_CURRENT_SOURCE_DIR}/test_apply_killlists.cmake + WORKING_DIRECTORY "${MANTICORE_BINARY_DIR}/test" ) + SET_TESTS_PROPERTIES ( Apply_killlists_with_legacy_json_header PROPERTIES LABELS INDEXTOOL ) endif () diff --git a/test/indextool/test_apply_killlists.cmake b/test/indextool/test_apply_killlists.cmake new file mode 100644 index 0000000000..bec4375956 --- /dev/null +++ b/test/indextool/test_apply_killlists.cmake @@ -0,0 +1,72 @@ +# Verify that indextool applies a current killlist to a legacy JSON plain index. +# The fixture was written in format v65, while indexer creates delta in the current format. +cmake_minimum_required ( VERSION 3.17 ) + +set ( WORKDIR "${CMAKE_CURRENT_BINARY_DIR}/indextool-killlists" ) +execute_process ( COMMAND ${CMAKE_COMMAND} -E rm -rf "${WORKDIR}" ) +execute_process ( COMMAND ${CMAKE_COMMAND} -E make_directory "${WORKDIR}" ) + +file ( COPY "${LEGACY_FIXTURE}/" DESTINATION "${WORKDIR}/fixture" ) +file ( GLOB dLegacyFiles "${WORKDIR}/fixture/index.0.*" ) +foreach ( sFile IN LISTS dLegacyFiles ) + get_filename_component ( sName "${sFile}" NAME ) + string ( REGEX REPLACE "^index\\.0" "main" sName "${sName}" ) + file ( RENAME "${sFile}" "${WORKDIR}/${sName}" ) +endforeach () + +file ( WRITE "${WORKDIR}/delta.tsv" "1\tupdated\n" ) +file ( WRITE "${WORKDIR}/manticore.conf" " +source src_delta +{ + type = tsvpipe + tsvpipe_command = cat ${WORKDIR}/delta.tsv + tsvpipe_field = title +} + +index main +{ + type = plain + path = ${WORKDIR}/main +} + +index delta +{ + type = plain + source = src_delta + path = ${WORKDIR}/delta + killlist_target = main:id +} +" ) + +execute_process ( + COMMAND "${INDEXER}" --config "${WORKDIR}/manticore.conf" delta + RESULT_VARIABLE iIndexerResult + OUTPUT_VARIABLE sIndexerOutput + ERROR_VARIABLE sIndexerError ) +if ( iIndexerResult ) + message ( FATAL_ERROR "Failed to build current-format killer: ${sIndexerOutput}${sIndexerError}" ) +endif () + +file ( READ "${WORKDIR}/main.spm" sBefore HEX ) +execute_process ( + COMMAND "${INDEXTOOL}" --config "${WORKDIR}/manticore.conf" --apply-killlists + RESULT_VARIABLE iApplyResult + OUTPUT_VARIABLE sApplyOutput + ERROR_VARIABLE sApplyError ) +if ( iApplyResult ) + message ( FATAL_ERROR "Failed to apply killlist: ${sApplyOutput}${sApplyError}" ) +endif () +file ( READ "${WORKDIR}/main.spm" sAfter HEX ) + +if ( NOT sBefore STREQUAL "00000000" OR NOT sAfter STREQUAL "01000000" ) + message ( FATAL_ERROR "Expected kill bit change 00000000 -> 01000000, got ${sBefore} -> ${sAfter}" ) +endif () + +execute_process ( + COMMAND "${INDEXTOOL}" --config "${WORKDIR}/manticore.conf" --check main + RESULT_VARIABLE iCheckResult + OUTPUT_VARIABLE sCheckOutput + ERROR_VARIABLE sCheckError ) +if ( iCheckResult ) + message ( FATAL_ERROR "Updated legacy index failed check: ${sCheckOutput}${sCheckError}" ) +endif () From 6f3e8f53eb0c62b8794306a5b4570f43277d6f38 Mon Sep 17 00:00:00 2001 From: Sergey Nikolaev Date: Fri, 21 Aug 2026 23:13:41 +0700 Subject: [PATCH 7/7] test: clarify legacy JSON killlist coverage --- test/indextool/CMakeLists.txt | 2 +- ...y_killlists.cmake => test_apply_killlists_legacy_json.cmake} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename test/indextool/{test_apply_killlists.cmake => test_apply_killlists_legacy_json.cmake} (100%) diff --git a/test/indextool/CMakeLists.txt b/test/indextool/CMakeLists.txt index 9326602936..8f72ed909c 100644 --- a/test/indextool/CMakeLists.txt +++ b/test/indextool/CMakeLists.txt @@ -17,7 +17,7 @@ else () -D INDEXER=$ -D INDEXTOOL=$ -D LEGACY_FIXTURE=${CMAKE_SOURCE_DIR}/test/test_406/data - -P ${CMAKE_CURRENT_SOURCE_DIR}/test_apply_killlists.cmake + -P ${CMAKE_CURRENT_SOURCE_DIR}/test_apply_killlists_legacy_json.cmake WORKING_DIRECTORY "${MANTICORE_BINARY_DIR}/test" ) SET_TESTS_PROPERTIES ( Apply_killlists_with_legacy_json_header PROPERTIES LABELS INDEXTOOL ) endif () diff --git a/test/indextool/test_apply_killlists.cmake b/test/indextool/test_apply_killlists_legacy_json.cmake similarity index 100% rename from test/indextool/test_apply_killlists.cmake rename to test/indextool/test_apply_killlists_legacy_json.cmake