diff --git a/docs/collections/powersync-collection.md b/docs/collections/powersync-collection.md index c028f22ed..c8ddbabbb 100644 --- a/docs/collections/powersync-collection.md +++ b/docs/collections/powersync-collection.md @@ -12,6 +12,7 @@ The `@tanstack/powersync-db-collection` package allows you to create collections - Automatically mirror the state of an underlying PowerSync SQLite database - Reactively update when PowerSync records change +- Support query-driven sync, loading only the data relevant to active live queries - Support optimistic mutations with rollback on error - Provide persistence handlers to keep PowerSync in sync with TanStack DB transactions - Use PowerSync's efficient SQLite-based storage engine @@ -301,6 +302,9 @@ When connected to a PowerSync backend, changes are automatically synchronized in - Queue management for offline changes - Automatic retries on connection loss +### Query-Driven Sync +Collections support an optional `on-demand` sync mode that loads only the rows your active live queries need, rather than syncing the full table into memory upfront. Useful for large datasets where most rows won't be accessed at a given time. + ### Working with Rich JavaScript Types PowerSync collections support rich JavaScript types like `Date`, `Boolean`, and custom objects while maintaining SQLite compatibility. The collection handles serialization and deserialization automatically: @@ -577,7 +581,7 @@ This approach allows you to: - Wait for persistence confirmation - Handle complex transaction scenarios -## On-demand Sync Mode (Query Driven Sync) +## On-demand Sync Mode (Query-Driven Sync) By default the PowerSync collection uses the `eager` sync mode, but you can opt to use `on-demand` sync mode instead. In on-demand mode, the collection is query-driven - it only loads data from SQLite that satisfies the predicates of your active live queries, rather than syncing everything and filtering afterwards. @@ -606,47 +610,34 @@ In examples 3 and 4, Sync Stream subscription parameters are hardcoded. Examples In `eager` mode, all data from the SQLite database is synced into the TanStack DB collection. The live query then filters this full dataset to produce the final result. This means the collection holds all rows regardless of whether they match any active query. -```mermaid -block-beta - columns 5 - - block:SQL - columns 2 - sql_title["SQLite"]:2 - SQL1["id: 1\nlist_id: list_1\ncompleted: 0"] - SQL2["id: 2\nlist_id: list_1\ncompleted: 1"] - SQL3["id: 3\nlist_id: list_2\ncompleted: 0"] - SQL4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:COL - columns 2 - col_title["Collection"]:2 - COL1["id: 1\nlist_id: list_1\ncompleted: 0"] - COL2["id: 2\nlist_id: list_1\ncompleted: 1"] - COL3["id: 3\nlist_id: list_2\ncompleted: 0"] - COL4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:RES - columns 2 - res_title["Result"]:2 - RES1["id: 1\nlist_id: list_1\ncompleted: 0"] - RES2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - SQL -- "Eager mode syncs all SQLite data\nto TanStack DB Collection" --> COL - COL -- "TanStack DB Query filters\nfor list_id = 'list_1'" --> RES - - style sql_title fill:none,stroke:none - style col_title fill:none,stroke:none - style res_title fill:none,stroke:none -``` +**SQLite** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*Eager mode syncs all SQLite data to TanStack DB Collection* + +**Collection** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*TanStack DB Query filters for list_id = 'list_1'* + +**Result** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | ```typescript const collection = createCollection( @@ -675,46 +666,32 @@ const liveQuery = createLiveQueryCollection({ In `on-demand` mode, only data from the SQLite database that satisfies the predicates of active TanStack DB live queries is synced into the TanStack DB collection. This reduces the amount of data held in memory, since the collection only contains rows that are relevant to the queries currently registered against it. -```mermaid -block-beta - columns 5 - - block:SQL - columns 2 - sql_title["SQLite"]:2 - SQL1["id: 1\nlist_id: list_1\ncompleted: 0"] - SQL2["id: 2\nlist_id: list_1\ncompleted: 1"] - SQL3["id: 3\nlist_id: list_2\ncompleted: 0"] - SQL4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:COL - columns 2 - col_title["Collection"]:2 - COL1["id: 1\nlist_id: list_1\ncompleted: 0"] - COL2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - space - - block:RES - columns 2 - res_title["Result"]:2 - RES1["id: 1\nlist_id: list_1\ncompleted: 0"] - RES2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - SQL -- "On-demand mode only syncs \n data from SQLite to the\n TanStack DB collection if they\nsatisfy any of the live queries:\nlist_id = 'list_1'" --> COL - COL -- "TanStack DB Query filters\nfor list_id = 'list_1'" --> RES - - style sql_title fill:none,stroke:none - style col_title fill:none,stroke:none - style res_title fill:none,stroke:none -``` +**SQLite** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*On-demand mode only syncs data from SQLite to the TanStack DB collection if they satisfy any of the live queries: list_id = 'list_1'* + +**Collection** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | + +*TanStack DB Query filters for list_id = 'list_1'* + +**Result** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | ```typescript const collection = createCollection( @@ -763,58 +740,40 @@ The hook can optionally return a cleanup function where you can unsubscribe from This example starts with 4 todos in the PowerSync Service, only 2 of which get synced via the Sync Stream to the SQLite database. Because it's eager mode, both get synced from the SQLite database to the collection. Finally the TanStack DB query only returns the single todo that matches the live query predicate. -```mermaid -block-beta - columns 7 - - block:PS - columns 2 - ps_title["PS Service"]:2 - PS1["id: 1\nlist_id: list_1\ncompleted: 0"] - PS2["id: 2\nlist_id: list_1\ncompleted: 1"] - PS3["id: 3\nlist_id: list_2\ncompleted: 0"] - PS4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:SQL - columns 2 - sql_title["SQLite"]:2 - SQL1["id: 1\nlist_id: list_1\ncompleted: 0"] - SQL2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - space - - block:COL - columns 2 - col_title["Collection"]:2 - COL1["id: 1\nlist_id: list_1\ncompleted: 0"] - COL2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - space - - block:RES - columns 2 - res_title["Result"]:2 - RES1["id: 2\nlist_id: list_1\ncompleted: 1"] - space - space space - end - - PS -- "Subscribe to Sync Stream,\nonly sync where list_id = 'list_1'" --> SQL - SQL -- "Eager mode syncs all SQLite data\nto TanStack DB Collection" --> COL - COL -- "TanStack DB Query filters\nfor completed = 1" --> RES - - style ps_title fill:none,stroke:none - style sql_title fill:none,stroke:none - style col_title fill:none,stroke:none - style res_title fill:none,stroke:none -``` +**PS Service** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*Subscribe to Sync Stream, only sync where list_id = 'list_1'* + +**SQLite** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | + +*Eager mode syncs all SQLite data to TanStack DB Collection* + +**Collection** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | + +*TanStack DB Query filters for completed = 1* + +**Result** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 2 | list_1 | 1 | ```typescript const collection = createCollection( @@ -858,58 +817,39 @@ The hook can optionally return a cleanup function where you can unsubscribe from This example starts with 4 todos in the PowerSync Service, only 2 of which get synced via the Sync Stream to the SQLite database. Because it's on-demand mode, only 1 todo matches gets synced from the SQLite database to the collection. Finally the TanStack DB query only returns the single todo that matches the live query predicate. -```mermaid -block-beta - columns 7 - - block:PS - columns 2 - ps_title["PS Service"]:2 - PS1["id: 1\nlist_id: list_1\ncompleted: 0"] - PS2["id: 2\nlist_id: list_1\ncompleted: 1"] - PS3["id: 3\nlist_id: list_2\ncompleted: 0"] - PS4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:SQL - columns 2 - sql_title["SQLite"]:2 - SQL1["id: 1\nlist_id: list_1\ncompleted: 0"] - SQL2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - space - - block:COL - columns 2 - col_title["Collection"]:2 - COL1["id: 2\nlist_id: list_1\ncompleted: 1"] - space - space space - end - - space - - block:RES - columns 2 - res_title["Result"]:2 - RES1["id: 2\nlist_id: list_1\ncompleted: 1"] - space - space space - end - - PS -- "Subscribe to Sync Stream,\nonly sync where list_id = 'list_1'" --> SQL - SQL -- "On-demand mode only syncs \n data from SQLite to the\n TanStack DB collection if they\nsatisfy any of the live queries:\ncompleted = 1" --> COL - COL -- "TanStack DB Query filters\nfor completed = 1" --> RES - - style ps_title fill:none,stroke:none - style sql_title fill:none,stroke:none - style col_title fill:none,stroke:none - style res_title fill:none,stroke:none -``` +**PS Service** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*Subscribe to Sync Stream, only sync where list_id = 'list_1'* + +**SQLite** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | + +*On-demand mode only syncs data from SQLite to the TanStack DB collection if they satisfy any of the live queries: completed = 1* + +**Collection** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 2 | list_1 | 1 | + +*TanStack DB Query filters for completed = 1* + +**Result** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 2 | list_1 | 1 | ```typescript const collection = createCollection( @@ -963,58 +903,41 @@ Note: This example slightly differs from the previous two as it aims to illustra This example starts with 4 todos in the PowerSync Service, the Sync Stream subscription criteria (`list_id = "list_1"`) is derived from the live query registered against the collection. Only 2 todos get synced via the Sync Stream to the SQLite database. Two todos get synced from the SQLite database to the collection. Finally the TanStack DB query returns both todos as they both match `eq(todo.list_id, 'list_id')`. -```mermaid -block-beta - columns 7 - - block:PS - columns 2 - ps_title["PS Service"]:2 - PS1["id: 1\nlist_id: list_1\ncompleted: 0"] - PS2["id: 2\nlist_id: list_1\ncompleted: 1"] - PS3["id: 3\nlist_id: list_2\ncompleted: 0"] - PS4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:SQL - columns 2 - sql_title["SQLite"]:2 - SQL1["id: 1\nlist_id: list_1\ncompleted: 0"] - SQL2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - space - - block:COL - columns 2 - col_title["Collection"]:2 - COL1["id: 1\nlist_id: list_1\ncompleted: 0"] - COL2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - space - - block:RES - columns 2 - res_title["Result"]:2 - RES1["id: 1\nlist_id: list_1\ncompleted: 0"] - RES2["id: 2\nlist_id: list_1\ncompleted: 1"] - space space - end - - PS -- "Derive the list_id criteria from \n live query, subscribe to sync\n stream, only sync \n where list = 'list_1'" --> SQL - SQL -- "On-demand mode only syncs\n data from SQLite to the\n TanStack DB collection if they\nsatisfy any of the live queries:\nlist_id = 'list_1'" --> COL - COL -- "TanStack DB Query filters\nfor list_id = 'list_1'" --> RES - - style ps_title fill:none,stroke:none - style sql_title fill:none,stroke:none - style col_title fill:none,stroke:none - style res_title fill:none,stroke:none -``` +**PS Service** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*Derive the list_id criteria from live query, subscribe to Sync Stream, only sync where list = 'list_1'* + +**SQLite** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | + +*On-demand mode only syncs data from SQLite to the TanStack DB collection if they satisfy any of the live queries: list_id = 'list_1'* + +**Collection** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | + +*TanStack DB Query filters for list_id = 'list_1'* + +**Result** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | ```typescript const collection = createCollection( @@ -1082,58 +1005,38 @@ The `list` parameter name is kept as-is (consistent with most examples), but mus This example starts with 4 todos in the PowerSync Service, the Sync Stream subscription criteria (`list_id = "list_1" and completed = 1`) is derived from the live query registered against the collection. Only 1 todo gets synced via the Sync Stream to the SQLite database. One todo gets synced from the SQLite database to the collection. Finally the TanStack DB query returns 1 todo that matches `eq(todo.list_id, 'list_id') and eq(todo.completed, 1)`. -```mermaid -block-beta - columns 7 - - block:PS - columns 2 - ps_title["PS Service"]:2 - PS1["id: 1\nlist_id: list_1\ncompleted: 0"] - PS2["id: 2\nlist_id: list_1\ncompleted: 1"] - PS3["id: 3\nlist_id: list_2\ncompleted: 0"] - PS4["id: 4\nlist_id: list_2\ncompleted: 1"] - end - - space - - block:SQL - columns 2 - sql_title["SQLite"]:2 - SQL1["id: 2\nlist_id: list_1\ncompleted: 1"] - space - space space - end - - space - - block:COL - columns 2 - col_title["Collection"]:2 - COL1["id: 2\nlist_id: list_1\ncompleted: 1"] - space - space space - end - - space - - block:RES - columns 2 - res_title["Result"]:2 - RES1["id: 2\nlist_id: list_1\ncompleted: 1"] - space - space space - end - - PS -- "Derive the list_id criteria and \n completed from live query,\n subscribe to Sync Stream\nwhere\n list = 'list_1' AND completed = 1" --> SQL - SQL -- "On-demand mode only syncs\n data from SQLite to the TanStack DB\n collection if they\nsatisfy any of the live queries:\nlist = 'list_1' AND completed = 1" --> COL - COL -- "TanStack DB Query filters for\nlist = 'list_1' AND completed = 1" --> RES - - style ps_title fill:none,stroke:none - style sql_title fill:none,stroke:none - style col_title fill:none,stroke:none - style res_title fill:none,stroke:none -``` +**PS Service** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 1 | list_1 | 0 | +| 2 | list_1 | 1 | +| 3 | list_2 | 0 | +| 4 | list_2 | 1 | + +*Derive the list_id and completed criteria from live query, subscribe to Sync Stream where list = 'list_1' AND completed = 1* + +**SQLite** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 2 | list_1 | 1 | + +*On-demand mode only syncs data from SQLite to the TanStack DB collection if they satisfy any of the live queries: list = 'list_1' AND completed = 1* + +**Collection** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 2 | list_1 | 1 | + +*TanStack DB Query filters for list = 'list_1' AND completed = 1* + +**Result** + +| id | list_id | completed | +|:--:|:-------:|:---------:| +| 2 | list_1 | 1 | ```typescript const collection = createCollection(