🐛 fix(search): align API result count with web UI

[gaborbernat]

The REST API search endpoint (/api/v1/search) reports different result counts than the web UI for identical queries. In project-less mode the API always caps resultCount at hitsPerPage * cachePages (defaulting to 125) because SearchEngine.search() returns the collected hits.length rather than Lucene's totalHits. 🔍 This is the root cause of both #3239 and #3170.

A second source of inaccuracy comes from using Short.MAX_VALUE (32,767) as the Lucene totalHitsThreshold in all collector constructions. For queries exceeding that threshold, totalHits.value becomes an approximate lower bound. The web UI already uses Integer.MAX_VALUE implicitly through the 3-arg searcher.search(query, n, sort) overload, so switching the API collectors to match eliminates the discrepancy with zero performance impact for the common case and marginal cost for very large result sets.

The endDocument field in the JSON response was also wrong, computed from the grouped result map's key count (unique file paths) instead of the actual document page size. A page of 10 documents spanning 5 files would report endDocument as startDocIndex + 4 instead of startDocIndex + 9. ✅ Now derived from the same page-size arithmetic used internally by SearchEngineWrapper.search().

Fixes #3239. Fixes #3170.

[gaborbernat]

@vladak is now ready

[vladak]

Needs rebase.

[gaborbernat]

@vladak done, thanks!

[vladak]

The webapp is still returning different total hit count on each search when performed via UI, so it is hard to evaluate whether API search is equivalent to UI search. For example with my favorite AOSP test (indexer run with -s /var/opengrok/src -d /var/opengrok/data -P --economical -c /opt/homebrew/bin/ctags -W /var/opengrok/etc/configuration.xml -U http://localhost:8080/opengrok_web_war), searching for "google" yields between 20k and 45k hits when done via UI and with your changes always the resultCount in the API search http://localhost:8080/opengrok_web_war/api/v1/search?project=AOSP&full=google&searchall=true&start=0&maxresults=1 results is always 200531 in my case.

[vladak]

The reason for the non-stable hit count in the UI is in how search totals are produced in web search (thanks Codex): SearchHelper.executeQuery() uses searcher.search(query, start + maxItems, sort) and then prints TopFieldDocs.totalHits.value. In Lucene this value can be an estimated/lower-bound total unless total-hit tracking is explicitly forced, so the displayed “of N” can drift between requests (especially with relevance sorting).

There is a way how to make the total hit count stable for the UI similarly to how this is proposed for the API:

diff --git a/opengrok-indexer/src/main/java/org/opengrok/indexer/web/SearchHelper.java b/opengrok-indexer/src/main/java/org/opengrok/indexer/web/SearchHelper.java
index f8177ef2d..ee9a7bd18 100644
--- a/opengrok-indexer/src/main/java/org/opengrok/indexer/web/SearchHelper.java
+++ b/opengrok-indexer/src/main/java/org/opengrok/indexer/web/SearchHelper.java
@@ -61,7 +61,9 @@ import org.apache.lucene.search.Sort;
 import org.apache.lucene.search.SortField;
 import org.apache.lucene.search.TermQuery;
 import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.search.TopFieldCollectorManager;
 import org.apache.lucene.search.TopFieldDocs;
+import org.apache.lucene.search.TopScoreDocCollectorManager;
 import org.apache.lucene.search.spell.DirectSpellChecker;
 import org.apache.lucene.search.spell.SuggestMode;
 import org.apache.lucene.search.spell.SuggestWord;
@@ -475,9 +477,17 @@ public class SearchHelper {
             return this;
         }
         try {
-            TopFieldDocs fdocs = searcher.search(query, start + maxItems, sort);
-            totalHits = fdocs.totalHits.value;
-            hits = fdocs.scoreDocs;
+            int resultWindow = start + maxItems;
+            TopDocs topDocs;
+            if (Sort.RELEVANCE.equals(sort)) {
+                topDocs = searcher.search(query,
+                        new TopScoreDocCollectorManager(resultWindow, Integer.MAX_VALUE));
+            } else {
+                topDocs = searcher.search(query,
+                        new TopFieldCollectorManager(sort, resultWindow, Integer.MAX_VALUE));
+            }
+            totalHits = topDocs.totalHits.value;
+            hits = topDocs.scoreDocs;
 
             /*
              * Determine if possibly a single-result redirect to xref is

For my test case that leads to stable 200483 hits in the UI, versus 200531 via the API which is pretty close.

To fix #3239 this patch would be needed.

The cons being:

• Higher query cost on large result sets because Lucene must track total hits exactly.
• Potentially increased latency and CPU usage for broad queries (especially very common terms).

Personally I find the varying result count in the UI confusing.

[gaborbernat]

Yeah, I agree, I'd consider it a bug that now is unreliable, it communicates the wrong information to the end user.

[vladak]

Also, benchmarking the UI search latency before and after the change, the original was actually slighly slower (like 15%) for some reason.

[gaborbernat]

@vladak pushed fixes, we should be good now.

[gaborbernat]

Also, when we merge this can we get a release cut? Thanks!

[vladak]

Also, when we merge this can we get a release cut? Thanks!

That's my plan. Will take another look at the changes.

[gaborbernat]

I have addressed the change requests.