Skip to content

Use Automerge doc for user state#976

Merged
epatters merged 76 commits intomainfrom
kb/automerge-user-state
Mar 27, 2026
Merged

Use Automerge doc for user state#976
epatters merged 76 commits intomainfrom
kb/automerge-user-state

Conversation

@kasbah
Copy link
Copy Markdown
Member

@kasbah kasbah commented Jan 30, 2026

This is ready for review but I suggest we wait for "Performance regression on main" (automerge/automerge#1159) to be resolved before deploying to production. As it stands it takes 100ms to create a new user state when the user first visits the site which manifest in a delay of seeing any documents if cold visiting /documents.

Closes the "server side state" part of #881 by implementing Automerge docs, derived from the db data, for every user state. Mutations are still handled through regular RPC calls. The UserState mainly holds the document lists and some user information and is put into a Context on the frontend.

I couldn't resist slightly improving the UI of "My documents" and "Trash" since I had to replace them anyway. The overall feeling of reactivity of the document titles and the user names is very nice.

image

I am not sure how useful it is, but inspired by Benjamin Pierce's Colloquium talk, I made a commutative diagram of the paths I used property based testing on.

image
$$t_1$$ write_user_state_to_db Writes a UserState to the database (testing only).
$$f_1$$ read_user_state_from_db Reads a UserState from the database.
$$f_2$$ run_user_state_subscription Listens for Postgres notifications related to user state changes.
$$g$$ user_state_to_automerge Converts a UserState into an Automerge document.
$$t_2$$ automerge_to_user_state Converts an Automerge document to a UserState (testing only).

Note that run_user_state_subscription will be fine in normal usage but if you e.g. use ALTER TABLE by loading a dump or something the automerge doc will be inconsistent with the DB. read_user_state_from_db is run on request at the start so re-starting the server will fix the inconsistencies.

Here is a very imperfect schema for the UserState. (Imperfect since we can't really represent Vec<T> or Option<T> where T is an entity in the schema on CatColab -- also papering over how things are referred to, directly or via ID.)

https://catcolab.org/model/019cae54-2940-7971-9060-dcd2b8657ff5/analysis/019cae83-2b0a-7062-85a8-ceb4d1665c46

UserState - ERD(4)

@kasbah kasbah force-pushed the kb/automerge-user-state branch from df280da to 00d279a Compare January 30, 2026 19:24
@epatters epatters added enhancement New feature or request backend Backend, including web server and database labels Jan 30, 2026
@kasbah kasbah force-pushed the kb/automerge-user-state branch 3 times, most recently from 27fe54c to ad099f8 Compare January 31, 2026 00:39
@epatters epatters changed the title Use automerge for user state Use Automerge doc for user state Jan 31, 2026
@kasbah kasbah force-pushed the kb/automerge-user-state branch 3 times, most recently from 9dbbd8c to 4ff0d67 Compare February 9, 2026 15:56
@kasbah kasbah force-pushed the kb/automerge-user-state branch 20 times, most recently from 0364555 to b3122c7 Compare February 26, 2026 18:21
@kasbah
Copy link
Copy Markdown
Member Author

kasbah commented Mar 25, 2026

In happy news Automerge, Autosurgeon and Samod have all gotten a release with the performance improvements. 🚀 🎉 I've upgraded them here.

@kasbah kasbah force-pushed the kb/automerge-user-state branch from ec09d5b to bb429fe Compare March 25, 2026 18:46
@kasbah kasbah force-pushed the kb/automerge-user-state branch from bb429fe to 04d5bb1 Compare March 25, 2026 18:54
Copy link
Copy Markdown
Member

@epatters epatters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kaspar. There's a lot to process here. I've left some questions/comments focusing on the backend piece. I'll try to make another pass soon.

My overall impression is that I'm glad the complexity has shifted from Postgres queries/notification log to Rust code, because I think the Rust code will be easier to maintain and involves less magic. That said, the code footprint on the backend is quite large and going forward the complexity of adding new features to the backend will increase. So I'm really hoping that this is worth it on the frontend side, both in terms of dev experience and user experience.

Comment thread packages/backend/src/user_state.rs Outdated
Comment thread packages/backend/src/user_state.rs Outdated
Comment thread packages/backend/src/user_state.rs Outdated
Comment thread packages/backend/src/user_state.rs Outdated
Comment thread packages/backend/src/user_state.rs
Comment thread packages/backend/src/user_state_updates.rs Outdated
Comment thread packages/ui-components/src/document_type_icon.tsx Outdated
Comment thread packages/frontend/src/util/virtual_list.ts
@kasbah kasbah force-pushed the kb/automerge-user-state branch from 34cb699 to 0aa7eda Compare March 26, 2026 15:03
@jmoggr
Copy link
Copy Markdown
Collaborator

jmoggr commented Mar 26, 2026

Are the performance concerns mentioned here:

Since we are not using the public docs at the moment I have removed them for now and will leave performance optimization for another PR.

Addressed by this?:

In happy news Automerge, Autosurgeon and Samod have all gotten a release with the performance improvements. 🚀 🎉 I've upgraded them here.

I think benefits of the user-state-doc decrease if we need a parallel system for public docs. I'd be happy to help out with that or explore feasibility.

Comment thread packages/backend/src/auth.rs
Comment thread packages/backend/src/user_state.rs Outdated
Comment thread packages/frontend/src/util/immutable_string.ts Outdated
Comment thread packages/frontend/tsconfig.json
@kasbah kasbah force-pushed the kb/automerge-user-state branch from 7e14f8b to dca18e9 Compare March 27, 2026 12:27
@kasbah kasbah force-pushed the kb/automerge-user-state branch from dca18e9 to ef29ea4 Compare March 27, 2026 12:34
@kasbah
Copy link
Copy Markdown
Member Author

kasbah commented Mar 27, 2026

@jmoggr

Are the performance concerns mentioned here:

Since we are not using the public docs at the moment I have removed them for now and will leave performance optimization for another PR.

Addressed by this?:

No, they are not.

I think benefits of the user-state-doc decrease if we need a parallel system for public docs. I'd be happy to help out with that or explore feasibility.

It's not really public vs private it's just the number of docs I believe. We might need different ways of determining "these are the docs the user cares about right now", though with that we are sort of back to something akin to paginating I guess.

@kasbah kasbah force-pushed the kb/automerge-user-state branch 2 times, most recently from 6ef5e45 to 2919b19 Compare March 27, 2026 13:10
@kasbah kasbah force-pushed the kb/automerge-user-state branch from 2919b19 to 7aeba2f Compare March 27, 2026 13:16
@epatters epatters merged commit 8016ea7 into main Mar 27, 2026
17 checks passed
@epatters epatters deleted the kb/automerge-user-state branch March 27, 2026 18:10
tim-at-topos pushed a commit that referenced this pull request Mar 31, 2026
* ENH: User state as an Automerge doc in backend

* REFACTOR: Public lib for backend

* ENH: Use Automerge UserState from frontend

* REFACTOR: Move get_or_create user state function

* REFACTOR: DbPermission

* REFACTOR: Use same automerge doc creation as in test

* ENH: Add parents to UserState DocInfo

* ENH: Add children to DocInfo

* ENH: Switch sidebar to use UserState

* CLEANUP: Remove RefStub related code

* CLEANUP: Remove unecessary refetch on delete doc

* ENH: Update .sqlx queries

* REFACTOR: Use fix in autosurgeon for Option<Text>

* REFACTOR: Rename UserState UserSummary to UserInfo

* ENH: Use String for user id in UserState

* ENH: Sort side-bar sub-docs by createdAt

* ENH: Show icons in documents and trash

* ENH: Show "Analysis of" and "Diagram in" in document lists

* ENH: Properly handle middle/Ctrl click for document lists

* REFACTOR: DocumentList component

* ENH: Add a link to Analysis of ...

* REFACTOR: Use `A` from solid for document links

* ENH: Show "Untitled" for empty doc title in list

* ENH: Tweak user state doc init

* TEST: Make sure doc is read in user state frontend test

* REFACTOR: User state DB migration

* TEST: Fix frontend user_state test isolation

* ENH: Make user name changes notify user state docs

* ENH: Add profile to UserState

* ENH: Don't create user state in response to DB only

* ENH: Handle profile updates more efficiently

* ENH: Tweak reactivity around owners in document list

* ENH: Tweak document list style and click behaviour

* REFACTOR: Rename users field to known_users

* ENH: Show docs with unknown parent as orphaned

* ENH: Re-generate .sqlx

* CLEANUP: Remove Paginated struct

* TEST: Reduce wait times in user state tests

* ENH: Switch to automerge/autosurgeon repo as crate source

* ENH: Better types for type_name

* TEST: Test user state url persistence

* ENH: Persist user state urls in db

* ENH: Clear persisted user states on restart

* TEST: Split catlog and backend tests in CI

* REFACTOR: ImmutabelString handling on user state permissions

* REFACTOR: Add autosurgeon_datetime.rs

* ENH: Handle unknown doc type in UserState DocInfo

* ENH: Switch to depends_on and used_by in UserState

* FIX: Move systemd "ready" to just before loop

* DOC: Add comment about arbitrary DocInfo relations

* ENH: Put user state doc id on users table

* REFACTOR: Say "state doc ID" instead of "state doc URL"

* ENH: Include public docs in UserState.documents

* REFACTOR: Extract filterDocuments helper for document list filtering

* REFACTOR: Extract reconcile_user_state helper for automerge reconciliation

* ENH: Update user state from autosave listener instead of DB trigger

Extract snapshot relations and update user state docs directly in the
Rust autosave listener, eliminating the pg_notify round trip that
re-read the snapshot from the database.

* BUILD: Fix copying of api bindings for user_state

* REFACTOR: Rename user_state_subscription migration

* WIP: Use automerge batch-insertion branch

* BUILD: Turn off debug-assertions for dev (automerge)

* BUILD: Disable debug-assertions for rust deps

* ENH: Bypass the DB for user state updates (#1151)

* Revert "ENH: Include public docs in UserState.documents"

This reverts commit c9eeb1c.

* BUILD: Switch to release versions of automerge, autosurgeon and samod

* TEST: Fix frontend test isolation by disallowing parallel runs

* ENH: Simplify automerge user state updates (#1163)

* DOC: Improve virtual_list doc comment

* REFACTOR: Use isImmutableString from Automerge

* REFACTOR: Define a DocumentType in notebook-types

* REFACTOR: Move reconcile_into to a method on UserState

* REFACTOR: Remove DbPermission intermediate type

* CLEANUP: Remove profiling code from user_state.rs

* FIX: UserState context needs to re-init on logout/login

---------

Co-authored-by: Jason Moggridge <git@jmoggr.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Backend, including web server and database enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants