Elasticsearch DSL Debugger

Start here

What it is, what it is not, and the first three minutes.

What it needs before anything works

Three things, in this order. Skipping any of them produces a tool that looks like it is working.

Three minutes

1. Open the ES Debugger tool window.
2. Choose a source: a bundled dataset, or your cluster and an index.
3. Paste a search body, or pick one of the 43 examples in the sidebar.
4. Press Debug query.
5. Press Run all stages, then read the clause badges.

Connecting a cluster

Settings holds your connections — URL plus basic auth, an API key, a bearer token, or AWS SigV4. Passwords go to the IDE password safe, never a project file.

Check cluster reports the version and whether the target resolves. Refresh fields reads the mapping, which is what makes field completion and most diagnostics possible.

The index box takes an index, an alias, a wildcard like logs-*, a comma-separated list, or remote:index for cross-cluster search. Click it for the list.

The funnel

Where the documents went, clause by clause.

Elasticsearch runs a search in two phases: a query phase on every shard, then a fetch phase on the coordinating node. The debugger breaks that into ten steps you can stop at, each reporting how many documents are still alive.

Parse · kNN · Rewrite · Match · Score · Aggregate · Sort & collect · Rescore · Reduce · Fetch

Step with Next stage and Previous stage, or jump to the end with Run all stages. Stopping matters because you get a document count after each one — that is how you find the step where they went.

Reading the badges

Step through it

Starts where you would: a query in the editor and nothing run yet. Press Next to press Debug query, then walk the ten stages the way the debugger builds them. Every count comes from the bundled E-commerce catalogue — 48 documents.

Switching clauses off

Your query in the editor is never edited. The body actually sent is in Inspector › Request, and the header counts how many clauses are switched off.

Navigating a large query

Scroll to zoom, drag to pan, Shift+scroll to pan sideways. Find a clause jumps to a field or type. Collapse all, Expand all and Fit handle the shape; Indented tree is there for queries wider than the screen.

Where the numbers come from

Which parts are Elasticsearch’s own output, which are arithmetic on real queries, and which are ours.

Elasticsearch runs a search in two phases and does not stop between them, so the ten steps are the debugger’s framing rather than something the engine exposes. Each one asks a fresh question and shows you the request it sent.

Inside a bool, the result is the intersection of the clauses. Two quantities follow from that and are measured directly, one search each:

• What a clause matches alone — that clause run by itself against the index.
• The running intersection — the first n clauses run together. removed is the difference between one of these and the next.

Against a cluster, every number in the funnel is the count of a query that was really run. The _msearch carrying them is in the Request tab, and any line of it pasted into Kibana returns the same number.

Two things worth knowing about them

removed depends on the order the clauses are walked, and that order is ours — measured counts, cheapest first, which approximates what Lucene does rather than reproducing it. matches alone does not depend on it. A clause that removes zero placed last removes zero in any order, which is why that finding is the one to act on.

The funnel checks itself. The last step is the whole conjunction, so its count must equal the count the query itself returned — two independent requests. If they disagree, the panel says so rather than showing you numbers that describe something other than your query.

Source of each number

Five of the seven are Elasticsearch’s own output, presented differently. One is arithmetic on queries that were really run. One is an ordering we chose.

That describes a run against a cluster. Offline the first three come from the plugin’s own evaluator, timings are absent rather than invented, and Text analysis is disabled — see the offline simulator.

The Inspector

Eleven tabs. Each answers a different question about the run — or, for Compare, about two of them.

Why not?

The question every bug report is phrased as: this document should be here and it isn’t.

Give an _id, or find the document by something you know — sku = SKU-0000003. Up to three fields, combined with and. Every clause is then asked about that one document separately, so a document failing three clauses reports all three — not just the first one Lucene reached. When several documents match your key they stay in a dropdown you can move between without re-running.

Compare

The rest of the Inspector answers what is happening. This answers did my edit work.

Every run past the Match stage is remembered. Change something, run again, and this shows what moved: which clauses, which documents, which buckets, and any stage whose status changed.

The finding it exists for is the one a count cannot give you. A query going from 47 hits to 47 hits looks unchanged; if three of those documents are different ones, it is not.

By default a run is measured against the one before it. Several edits into chasing a regression that compares two broken states, so Pin as baseline holds the run that was last correct.

Comparisons last for the session and are not saved: a baseline from last week would be measured against an index that has changed underneath it, so most of what you saw would be documents being written and deleted rather than anything your query did.

How far you run decides what you can compare

Stepping one stage at a time and pressing Run all stages give the same result if you stop in the same place — only where you stop matters.

Text analysis

Why a match finds nothing when the words look right.

A search succeeds when the tokens a field stored and the tokens your query produces agree. This shows both, stage by stage, with the words each filter added or removed.

• Position, length and offset for every token, so multi-word synonyms and shingles are visible
• Names which analyzer ran and what chose it — the field’s search_analyzer, an index-level default_search, or one named on the clause
• Uses search_quote_analyzer for phrase clauses, which is a common surprise
• Says plainly when a keyword field is not analysed at all

Right-click a match in the graph and choose Analyze to open it with the field, the text and any clause analyzer already filled in. This tab needs a cluster: it reads _analyze.

Performance

Where the milliseconds went, by phase, by clause, by shard, from Elasticsearch’s own profiler. Each clause is named in your DSL — wildcard sku : *00042* — rather than in Lucene’s toString().

The other six tabs

Diagnostics

38 rules for the mistakes that never raise an error.

The ones that return a result rather than a failure, so nothing tells you they happened. A term on a text field is the familiar example; most of the others are less obvious.

What sets them apart is not the list. Every finding carries its evidence — the numbers it was derived from — and a suggested fix. Several prove the fix by running both queries and comparing the results, so you see that it worked rather than being told it should.

Review and apply shows the change before it touches your editor.

The sidebar

Examples, fields, the index itself, and everything you have run.

The offline simulator

How the plugin runs a query with no cluster, and where that stops being the same thing.

Pick a bundled dataset instead of a connection and the whole tool still works: ten stages, the funnel, per-clause counts, aggregations, sorting, scoring. There is no Elasticsearch behind it. The plugin carries its own query engine and runs your query against 48 or 96 in-memory documents.

It is there so the tool can be learned, and a query understood, without a connection — on a plane, in a review, or before you have credentials for anything. The 43 worked examples all run through it.

What it does itself

Where it stops

The line matters more than the coverage, because a simulator that guesses is worse than no simulator. Where it cannot reproduce Elasticsearch it reports that it cannot, rather than producing a number:

• Pipeline aggregations it does not implement — moving_fn among them — report not simulated offline and produce no counts.
• Fetch sub-phases such as fields and docvalue_fields say they were not simulated.
• Counts that would be approximate on a real multi-shard index are labelled approximate, with the reason.
• Text analysis is disabled entirely. It reads _analyze, and answering with our own analyzer would be telling you what we do while claiming to tell you what your cluster does.
• Timings are absent rather than invented. There is no shard, so there is nothing to time.

Limitations

What the plugin caps, approximates, or cannot know. Each one is stated in the interface at the moment it applies; they are gathered here so none of them is a surprise.

Three kinds of thing are listed together, because someone who hits one cannot tell them apart: caps chosen to keep a request count sane, approximations where an exact answer is not available, and things Elasticsearch will not tell us.

The funnel

Results and documents

• The Data tab shows at most 500 documents — ten when the request has no size. The stated total is always the real total.
• Why not? takes three fields, combined with and: enough to identify a document, not a query builder.
• The per-shard listing stops at six shards — and nothing else does. That listing costs one search per shard, so on a 100-shard index it would mean a hundred extra searches against production. Timings, the clause table and the shard-skew finding are computed from every shard, because they come from the single search’s profile output.

Text analysis

• Needs a cluster, and needs the mapping loaded. The query side is resolved by analyzer name, which comes from the mapping — without it the resolution falls through to standard and the two sides get compared against the wrong chain. It refuses rather than guessing.
• One field and one text at a time. A multi_match over five fields is five checks.

Compare

• Six runs, this session only. A baseline from last week would be measured against an index that has changed underneath it.
• The document rows compare a page unless size covers every match.
• Nothing is recorded before Match, and documents only exist from Fetch.
• Timing changes under 50 ms are hidden as scheduling noise. Status changes never are.

Performance timings

• Elasticsearch’s numbers, not ours. Every figure is profile’s time_in_nanos; the plugin times nothing itself.
• Per-clause time has its children subtracted, so a bool is not credited with the work of every clause inside it.
• Shard work is not wall clock. Shards run in parallel, so the summed work exceeds took and is meant to. The panel states both.
• Profiling changes what it measures — Elasticsearch’s own caveat. Read the proportions, not the milliseconds.
• Cache state is not controlled. A second run can be much faster because filter bitsets are cached, which is why Compare hides timing changes under 50 ms.

By design, and not planned

• Elasticsearch 8 and 9. Older majors are flagged at Check cluster. OpenSearch is named correctly and its version numbers are not modelled, so the version-aware rules do not apply to it.
• The Query DSL only — not ES|QL, EQL, SQL or KQL.
• Nothing re-runs on its own. A debugger that re-ran on every keystroke would reach your cluster on every keystroke.
• No completion, deliberately.
• Two paths not yet exercised against a live server: AWS SigV4 signing, and skipping TLS verification for a self-signed development certificate. Both are implemented; neither has met the thing it exists for. If either matters to you, write and it will be confirmed or fixed.

Reference

Every request the plugin makes, and where it makes it.

What each stage sends

Outside the stage loop: Text analysis calls _analyze, Why not? calls _search once per clause for the document you named, and Refresh fields reads _mapping, _settings and _field_caps.

Keyboard

Next stage, Previous stage and Run all stages are in Find Action and bindable under Settings › Keymap.

Safety and privacy

What it can do to your cluster, and what leaves your machine.

It reads and it cannot write. Not by policy — by construction. Every request goes through an allow-list of read-only endpoints, and every mutating one is refused before it is sent: no _bulk, no _update, no _delete_by_query, no _reindex, no PUT, no DELETE.

Endpoints that would leave server-side state — _pit, _scroll, _async_search — are refused too. A point in time may be suggested to you; the plugin never opens one.

Common questions