Fix deadlock when canceling queries stuck in VDBE or busy handler#343
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses a deadlock that can occur when DBConnection.disconnect/2 closes a SQLite connection while a dirty NIF thread is blocked in VDBE execution or the busy handler, by adding a cancellable busy handler plus a new Sqlite3.cancel/1 path to wake/interrupt blocked operations before close/1.
Changes:
- Add a custom busy handler (condvar-based) + progress handler, and new NIF APIs
set_busy_timeout/2andcancel/1to support fast cancellation. - Update
Exqlite.Connection.disconnect/2to callSqlite3.cancel/1beforeclose/1, and migrate busy-timeout setting away fromPRAGMA busy_timeout. - Expand tests around busy-timeout and cancellation behavior; add build dependency tracking for NIF compilation.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/exqlite/sqlite3_test.exs | Adds baseline busy-timeout tests plus coverage for new set_busy_timeout/2 and cancel/1 behavior. |
| test/exqlite/integration_test.exs | Updates timeout test expectations given query interruption on disconnect. |
| test/exqlite/cancellation_test.exs | Adds a larger cancellation/deadlock regression suite (tagged :slow_test). |
| lib/exqlite/sqlite3_nif.ex | Adds NIF function specs for set_busy_timeout/2 and cancel/1, plus external resource tracking. |
| lib/exqlite/sqlite3.ex | Exposes set_busy_timeout/2 and cancel/1 in the public Elixir API. |
| lib/exqlite/connection.ex | Calls Sqlite3.cancel/1 during disconnect and switches busy-timeout setup to the new NIF. |
| c_src/sqlite3_nif.c | Implements cancellable busy handler, progress handler, and the new NIFs. |
| Makefile | Adds -MMD -MP and includes .d files for header dependency tracking. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
This does not fix that issue that is open. There's stuff in that issue that pertains to this, but this does not replace the Dirty NIF execution model. I'll need to check this out and understand what is going on before we merge it. |
|
Fair point on the execution model — this PR doesn't replace dirty NIFs and doesn't attempt to. The scope is narrower: fixing the specific deadlock where The What this does fix:
What this does NOT do:
|
|
I would like to stick to using the built in |
|
The reason we use Worth noting: the The Two alternatives if you'd prefer to avoid raw pthreads:
Happy to refactor to either if you have a preference. |
|
I'll need to think on this a little more. I peeked at a pthread_cond_timedwait implementation. Don't know if enif_monotonic_time would be a good choice for the clock portability issue. |
|
If you're open to the polling approach, the busy handler simplifies to roughly: static int exqlite_busy_handler(void *arg, int count) {
connection_t *conn = (connection_t *)arg;
if (conn->cancelled) return 0;
if (!enif_is_process_alive(conn->callback_env, &conn->caller_pid)) return 0;
conn->busy_elapsed_ms += 10;
if (conn->busy_elapsed_ms >= conn->busy_timeout_ms) return 0;
sqlite3_sleep(10);
return !conn->cancelled;
}The entire |
|
The polling solution is a busy wait and would eat unnecessary cpu. |
|
It's not a busy wait — In fact, this is exactly what SQLite's own built-in busy handler ( sqliteDefaultBusyCallback does) on the released version of exqlite right now — it calls We could even replicate the same delay ramp that sqliteDefaultBusyCallback uses if desired — since that's what I just liked the |
|
Does this current implementation work on Apple silicon? |
|
@warmwaffles yes; I built the condvar version on an M1 Mac originally and tested on Ryzen 9 5950X on Linux. I have not tested windows yet but I can if you wish, I just don't have elixir on my windows machine atm so it'll take a minute. The sqlite_sleep(n) approach should necessarily work since that's what exqlite was already doing, it's definitely the most portable and simplest choice. |
|
In my other failed game projects I had to use mach_timebase_info_data_t clock_timebase;
mach_timebase_info(&clock_timebase);
uint64_t mach_absolute = mach_absolute_time();
uint64_t nanos = (double)(mach_absolute * (uint64_t)clock_timebase.numer) / (double)clock_timebase.denom;
return nanos / 1.0e9; |
|
Yeah, But we don't actually need it for either approach: Condvar approach: We use Polling approach: Either way, |
|
I did a deep dive into the alternative execution models from #192 to see if any of them change what's needed here:
Every execution model ends up requiring the same SQLite-level cancellation: custom busy handler with a cancel flag + |
8936b7b to
4630766
Compare
|
sorry about all the force pushes; noticed somehow I didn't sign two of the commits. current version has the simpler sleep-based approach. |
|
I don't really mind signed or unsigned commits ¯\_(ツ)_/¯ I just haven't had time to look at this lately |
|
No worries at all, this fix works for me locally so I have no need to rush it to get merged or anything. |
Tests validate query cancellation and interrupt behavior from issue elixir-sqlite#192: - Connection can be closed/interrupted during long-running queries - cancel/interrupt work during VDBE execution - cancel/interrupt work during busy handler waits - Pool recovery after stuck queries (2 tests currently FAIL - demonstrating the bug) - close() returns quickly even with in-flight queries - Multiple simultaneous operations - Busy timeout expiration behavior - Transaction rollback during interruption 22 tests total, tagged :slow_test (take ~78s). At this commit, 20 pass and 2 fail: - FAIL: 'pool recovers after long query interrupted on disconnect' (timeout) - FAIL: 'pool recovers when query stuck in busy handler' (stuck waiting) These failures demonstrate the deadlock described in elixir-sqlite#192. Subsequent commits implement the fix that makes all 22 tests pass.
When DBConnection times out a long-running query, it calls disconnect/2 on the connection process. Previously, disconnect only called Sqlite3.close(), which blocks on conn->mutex held by the still-running dirty NIF — deadlocking the pool permanently. Now disconnect calls Sqlite3.interrupt(db) first, which sets a flag checked at every VDBE loop iteration (OP_Next, OP_Prev, etc.). The running query aborts with SQLITE_INTERRUPT, releases the mutex, and close() proceeds. This is safe because PR elixir-sqlite#342 (v0.35.0) added interrupt_mutex to the NIF layer, which prevents a use-after-free race between interrupt() and close(). The interrupt_mutex ensures interrupt reads the db pointer atomically even if close() is freeing it concurrently. Test results before this change: 149 tests, 20 pass, 2 fail (pool recovery test deadlocks at 30s timeout) Test results after this change: 160 tests, 2 failures - 21 of 22 cancellation tests now pass (busy handler test still fails as expected) - The integration test "exceeding timeout" fails with a pattern match error (expects {:ok, _, _} but gets {:error, "interrupted"} — will be fixed later) The passing cancellation tests demonstrate that interrupt successfully unblocks queries stuck in VDBE execution, fixing the primary deadlock issue. Known limitation: does not fix the case where a query is stuck in SQLite's busy handler sleep loop (waiting for a lock held by another connection). That requires a custom busy handler (separate change). Fixes the primary deadlock described in elixir-sqlite#192.
- Add timed_wait_t abstraction over pthread_cond_timedwait (POSIX) and SleepConditionVariableCS (Windows) - ~30 lines covering 100% of BEAM platforms - Extend connection_t with cancel_tw condvar, cancelled flag, busy_timeout_ms, and env+pid stashing for process-alive checks - Replace sqliteDefaultBusyCallback with exqlite_busy_handler that waits on condvar (instantly cancellable) and checks enif_is_process_alive - Add exqlite_progress_handler to interrupt long-running VDBE execution - Add exqlite_set_busy_timeout NIF (stores timeout without destroying handler, unlike PRAGMA busy_timeout which silently replaces custom handlers) - Add exqlite_cancel NIF (sets flag, signals condvar, calls sqlite3_interrupt) - Make cancel and set_busy_timeout non-dirty NIFs (must run on normal scheduler to avoid queueing behind stuck dirty NIFs) - Update destructor to signal cancel before close - Install handlers in exqlite_open, stash env+pid at 5 busy-triggerable call sites Fixes the busy handler plane of elixir-sqlite#192 - disconnect can now wake sleeping handlers instantly instead of waiting for busy_timeout to expire.
- Add NIF stubs: set_busy_timeout/2, cancel/1
- Add public wrappers: Sqlite3.set_busy_timeout/2, Sqlite3.cancel/1
- cancel/1 includes nil guard (matches close/1 and interrupt/1 convention)
- Fix bind_parameter_count/1 typespec: integer -> non_neg_integer() | {:error, reason()}
- Fix bind_parameter_index/2 typespec: integer -> non_neg_integer() | {:error, reason()}
The typespec fixes address pre-existing dialyzer warnings - both NIFs can return
{:error, :invalid_statement} when statement->statement is NULL.
- Replace Sqlite3.interrupt(db) with Sqlite3.cancel(db) in disconnect/2 (cancel is a superset that also wakes the busy handler condvar) - Replace PRAGMA busy_timeout with Sqlite3.set_busy_timeout NIF call (PRAGMA internally calls sqlite3_busy_timeout which destroys custom handlers) Together these changes complete the fix for elixir-sqlite#192 - disconnect now properly cancels all three blocking planes (VDBE execution, busy handler sleep, and mutex contention as a side effect of the first two).
Tests cover: - Default busy_timeout baseline (2000ms) - set_busy_timeout changes timeout value - set_busy_timeout to 0 disables retries - Custom handler retries on contention - Custom handler respects timeout limit - cancel breaks through busy handler sleep instantly - cancel returns ok on idle connection - cancel(nil) returns :ok - Cancelled connection can be reused (flag resets) - cancel on closed connection returns error - interrupt still works independently Also fix unused default argument warning in setup_write_conflict/2 (all call sites provide opts explicitly). All 159 tests pass (11 doctests + 148 tests).
- Add @external_resource declarations for all C files so mix compile detects changes without needing 'mix clean' first - Add -MMD -MP flags to Makefile for automatic header dependency tracking - Clean .d files in 'make clean' target
Snapshot conn->cancelled under lock in both the busy handler (after waking from condvar) and the progress handler. Previously both paths read the field outside the lock, which is a data race with exqlite_cancel writing to it on another scheduler thread. Also run clang-format to satisfy the lint check.
Only :ok and interrupted are expected results when a query races with its connection timeout. Accepting OOM would mask real regressions.
Callers assume it always returns a non_neg_integer. The | {:error, reason()}
union was incorrect and misleading.
C89-style: cancelled was declared twice in the same function scope after the data-race fix split the lock sections. Remove the second 'int' to reuse the variable declared at the top of the function.
With a 1ms checkout timeout, disconnect can race with an in-progress SQLite allocation and return SQLITE_NOMEM before the interrupt lands. This is an inherent race at that timeout granularity, not a regression.
Replace the timed_wait_t platform abstraction (pthread_cond_timedwait on POSIX, SleepConditionVariableCS on Windows) with polling using sqlite3_sleep(). This eliminates raw platform-specific threading primitives in favor of OTP and SQLite APIs. The busy handler now sleeps for short intervals (1-50ms) and checks conn->cancelled between iterations. Cancel latency is at most ~10ms (one sleep interval), which is acceptable since disconnects are measured in seconds. Changes: - Delete timed_wait_t abstraction and all platform-specific code (~124 lines) - Remove cancel_tw field from connection_t - Rewrite exqlite_busy_handler to use sqlite3_sleep() in a polling loop - Simplify exqlite_progress_handler, connection_stash_caller, exqlite_set_busy_timeout, exqlite_cancel to use volatile reads instead of locks - All 159 tests pass; linting passes - Net reduction: 166 lines (183 deleted, 17 added)
4630766 to
2cfa8e6
Compare
|
rebased to address a conflict. I could also clean up the history when you're ready to give it another look. but also no rush as I can just keep using my branch |
|
I'll be getting to this soon. I've been extremely busy lately. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
04baa4b to
1dc222a
Compare
| int authorizer_deny[AUTHORIZER_DENY_SIZE]; | ||
|
|
||
| // Custom busy handler state | ||
| volatile int cancelled; // guarded by interrupt_mutex |
There was a problem hiding this comment.
I am probably going to merge this and work on it some in another branch.
What is the benefit of throwing volatile here versus just having it be a basic int? I know that volatile is a signal to ensure the compiler doesn't re-arrange order of execution, but what I want to know, is this actually a risk?
There was a problem hiding this comment.
That was an oversight from when I backed out the initial proposed handler. No real benefit now that all reads/writes are guarded by interrupt_mutex. volatile does not make this thread-safe, and the mutex is what provides the synchronization here, so I removed it.
|
@mjc can you update the PR description with the most accurate information you can? I'm going to proof it and then use it for the commit message. |
|
Updated! Happy to help as much as I can, also. I've been running the previous version (with the condvar) since Feb 1 in a bunch of environments with none of the previous issues and no segfaults. I still get busy timeouts but that is largely due to not yet quite having exactly the right concurrency model in that specific distributed app - the timeouts make sense and are traced to obvious things I know I need to fix. down like 90% from before though |
I still get this as well, it's been a minor irritant because it backs an MCP server I am running. |
|
If you figure it out, don't hesitate to open another PR. |
Partially addresses elixir-sqlite/exqlite#192.
Summary
This PR fixes a deadlock where
DBConnection.disconnect/2could block forever while closing a connection whose dirty NIF thread was still inside SQLite. The dirty NIF execution model is unchanged; the fix is to make the blocked SQLite operation cancellable so it can releaseconn->mutexand allow close to finish.The deadlock can happen in two important cases: a long-running statement inside SQLite VDBE execution, or a statement sleeping in SQLite's busy handler while waiting for a lock. In both cases the NIF call holds
conn->mutex. If the pool times out and disconnect tries to close the connection, close also needsconn->mutex, so it waits behind the stuck operation.What changed
This adds
Exqlite.Sqlite3.cancel/1, which sets a connection cancellation flag and callssqlite3_interrupt().Exqlite.Connection.disconnect/2now callscancel/1beforeclose/1, so teardown can interrupt a running statement or cause a busy wait to stop retrying.This replaces SQLite's default busy handler with an Exqlite busy handler. The handler preserves SQLite's default retry delay schedule, stores the timeout in
conn->busy_timeout_ms, sleeps withsqlite3_sleep(), and checks the cancellation flag between sleeps. This avoids the non-cancellable behavior of SQLite's default busy handler while keeping the implementation portable.This adds
Exqlite.Sqlite3.set_busy_timeout/2and changes connection setup to use it instead ofPRAGMA busy_timeout.PRAGMA busy_timeoutcallssqlite3_busy_timeout(), which installs SQLite's default busy handler and replaces any custom handler. The new API updates Exqlite's stored timeout without destroying the cancellable busy handler.This adds a configurable progress handler interval through
Exqlite.Sqlite3.set_progress_handler_steps/2and the:progress_handler_stepsconnection option. The default is1000VDBE steps. Values less than1disable the progress handler for callers that want to avoid that overhead.The shared cancellation flag is protected by
interrupt_mutex. The busy handler, progress handler,cancel/1, and close/destructor paths read or write that flag under the same lock. The busy-timeout and progress-handler-step fields are also updated underinterrupt_mutex.User-facing behavior
High-level users who configure
:busy_timeoutonExqlite.start_link/1or an Ecto SQLite repo should not need to change anything. The connection setup path now applies that timeout throughExqlite.Sqlite3.set_busy_timeout/2.Low-level users who execute
PRAGMA busy_timeout = ...directly should migrate toExqlite.Sqlite3.set_busy_timeout/2if they want to keep cancellation of busy waits. Executing the pragma still works as SQLite behavior, but it replaces Exqlite's custom busy handler with SQLite's default handler, socancel/1can no longer stop the busy-handler retry loop before SQLite's busy timeout expires.Exqlite.Sqlite3.interrupt/1remains available for SQLite interruption.Exqlite.Sqlite3.cancel/1is the stronger teardown-oriented API because it sets Exqlite's cancellation flag for the busy/progress handlers and also callssqlite3_interrupt().Implementation notes
The busy handler uses SQLite's default delay schedule:
1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50ms, then50ms for later retries. Cancellation is observed between sleep intervals, so a busy wait exits after the current sleep finishes, with later sleeps capped at 50 ms.The NIF stores the calling environment and pid while SQLite operations are running so the busy handler can detect when the caller process has died. If the caller is gone, the busy handler sets the cancellation flag and stops retrying.
cancel/1andinterrupt/1avoid takingconn->mutexwhile a query may be running. A running SQLite call holds that mutex, so taking it in the cancellation path would block behind the operation we are trying to cancel. The timeout/progress configuration setters still takeconn->mutexbecause they are not used to break a currently running SQLite call.The NIF resource destructor and close path coordinate through
conn->mutexandinterrupt_mutexso connection teardown does not race with cancellation state updates or a concurrent interrupt.The Mix project now tracks the C source files as external resources, and the Makefile emits dependency files for C headers, so C/NIF changes are rebuilt more reliably.
Tests
This PR adds regression coverage for busy-timeout behavior, low-level
set_busy_timeout/2, configurable progress-handler steps,cancel/1, connection reuse after cancellation, pool recovery after timeouts, cancellation of busy-handler waits, and close/cancel races.It also adds sanitizer-tagged stress tests for the cancellation and busy-timeout update paths. Those tests are excluded from the default suite and can be run explicitly with
--include sanitizer.