-
Notifications
You must be signed in to change notification settings - Fork 16
Update ENSDb schema definitions #1752
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
6acbea0
eed4800
16c5584
92afddf
b5bd5ca
3f7f0e8
c1964a2
b4fb135
6b6f805
9c05883
f13b6af
1791217
03fecc1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "@ensnode/ensnode-schema": minor | ||
| --- | ||
|
|
||
| Split database schemas into Ponder Schema, ENSIndexer Schema, and ENSNode Schema. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "ensapi": minor | ||
| --- | ||
|
|
||
| Updated ENSDb connections to be always read-only. | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,11 +1,14 @@ | ||||||
| import config from "@/config"; | ||||||
|
|
||||||
| import * as schema from "@ensnode/ensnode-schema"; | ||||||
| import * as ensIndexerSchema from "@ensnode/ensnode-schema/ensindexer"; | ||||||
|
|
||||||
| import { makeDrizzle } from "@/lib/handlers/drizzle"; | ||||||
| import { makeReadOnlyDrizzle } from "@/lib/handlers/drizzle"; | ||||||
|
|
||||||
| export const db = makeDrizzle({ | ||||||
| /** | ||||||
| * Read-only Drizzle instance for ENSDb queries to ENSIndexer Schema | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How should we think about which db schemas ENSApi reads from? My understanding is it should read from both the ENSNode and ENSIndexer schemas. But here it's only reading from ENSIndexer. Why is that? Assuming ENSApi really does read from multiple schemas than that would suggest it's wrong to call this variable something generic such as Please invest more effort into naming and communicating ideas. |
||||||
| */ | ||||||
| export const db = makeReadOnlyDrizzle({ | ||||||
| databaseUrl: config.databaseUrl, | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please see my other related comment about renaming things in a follow-up issue. I'd also like to see Eager to make all our terminology use be 100% precise, consistent, and aligned everywhere.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Issue created here: |
||||||
| databaseSchema: config.databaseSchemaName, | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please create a follow-up issue for us to rename all the environment variable names / other variable names to be more precise and consistent in how they relate to our improved architecture. For example: We should stop using the generic Suggest to create a follow-up issue for this and action it in a separate PR.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Issue created here: |
||||||
| schema, | ||||||
| schema: ensIndexerSchema, | ||||||
| }); | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| import { setDatabaseSchema } from "@ponder/client"; | ||
| import { drizzle } from "drizzle-orm/node-postgres"; | ||
| import { parseIntoClientConfig } from "pg-connection-string"; | ||
|
|
||
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| import { makeLogger } from "@/lib/logger"; | ||
|
|
||
|
|
@@ -8,21 +9,32 @@ type Schema = { [name: string]: unknown }; | |
| const logger = makeLogger("drizzle"); | ||
|
|
||
| /** | ||
| * Makes a Drizzle DB object. | ||
| * Makes a read-only Drizzle DB object. | ||
| */ | ||
| export const makeDrizzle = <SCHEMA extends Schema>({ | ||
| export const makeReadOnlyDrizzle = <SCHEMA extends Schema>({ | ||
| schema, | ||
| databaseUrl, | ||
| databaseSchema, | ||
| }: { | ||
| schema: SCHEMA; | ||
| databaseUrl: string; | ||
| databaseSchema: string; | ||
| databaseSchema?: string; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This code is confusing. Why is there one variable named We really need you to put more effort into this!!! |
||
| }) => { | ||
| // monkeypatch schema onto tables | ||
| setDatabaseSchema(schema, databaseSchema); | ||
| // monkeypatch schema onto tables if databaseSchema is provided | ||
| if (databaseSchema) { | ||
| setDatabaseSchema(schema, databaseSchema); | ||
| } | ||
|
|
||
| return drizzle(databaseUrl, { | ||
| const parsedConfig = parseIntoClientConfig(databaseUrl); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Naming here is bad. |
||
| const existingOptions = parsedConfig.options || ""; | ||
| const readOnlyOption = "-c default_transaction_read_only=on"; | ||
|
|
||
| return drizzle({ | ||
| connection: { | ||
| ...parsedConfig, | ||
| // Combine existing options from URL with read-only requirement | ||
| options: existingOptions ? `${existingOptions} ${readOnlyOption}` : readOnlyOption, | ||
| }, | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| schema, | ||
| casing: "snake_case", | ||
| logger: { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,6 @@ | |
| // We currently duplicate the makeDrizzle function, as we don't have | ||
| // a shared package for backend code yet. When we do, we can move | ||
| // this function to the shared package and import it in both places. | ||
| import { setDatabaseSchema } from "@ponder/client"; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we remove the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, |
||
| import { drizzle } from "drizzle-orm/node-postgres"; | ||
|
|
||
| type Schema = { [name: string]: unknown }; | ||
|
|
@@ -13,14 +12,9 @@ type Schema = { [name: string]: unknown }; | |
| export const makeDrizzle = <SCHEMA extends Schema>({ | ||
| schema, | ||
| databaseUrl, | ||
| databaseSchema, | ||
| }: { | ||
| schema: SCHEMA; | ||
| databaseUrl: string; | ||
| databaseSchema: string; | ||
| }) => { | ||
| // monkeypatch schema onto tables | ||
| setDatabaseSchema(schema, databaseSchema); | ||
|
|
||
| return drizzle(databaseUrl, { schema, casing: "snake_case" }); | ||
| }; | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,10 @@ export const databaseUrl = "postgres://user:pass@localhost:5432/ensdb"; | |
|
|
||
| export const databaseSchemaName = "public"; | ||
|
|
||
| // This is the same as the default value of config.databaseSchemaName, | ||
| // which is used as the ensIndexerRef for multi-tenancy in ENSDb. | ||
| export const ensIndexerRef = databaseSchemaName; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fix the bad naming. |
||
|
|
||
| export const publicConfig = { | ||
| databaseSchemaName, | ||
| ensRainbowPublicConfig: { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| import type { NodePgDatabase } from "drizzle-orm/node-postgres"; | ||
| import { eq, sql } from "drizzle-orm/sql"; | ||
| import { and, eq, sql } from "drizzle-orm/sql"; | ||
tk-o marked this conversation as resolved.
Dismissed
Show dismissed
Hide dismissed
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| import { ensNodeMetadata } from "@ensnode/ensnode-schema"; | ||
| import * as ensNodeSchema from "@ensnode/ensnode-schema/ensnode"; | ||
| import { | ||
| type CrossChainIndexingStatusSnapshot, | ||
| deserializeCrossChainIndexingStatusSnapshot, | ||
|
|
@@ -20,21 +20,12 @@ import { | |
|
|
||
| import { makeDrizzle } from "./drizzle"; | ||
|
|
||
| /** | ||
| * ENSDb Client Schema | ||
| * | ||
| * Includes schema definitions for {@link EnsDbClient} queries and mutations. | ||
| */ | ||
| const schema = { | ||
| ensNodeMetadata, | ||
| }; | ||
|
|
||
| /** | ||
| * Drizzle database | ||
| * | ||
| * Allows interacting with Postgres database for ENSDb, using Drizzle ORM. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I find it confusing exactly what this class has responsibility for. Is it for everything in an ENSDb, or is it only for the ENSNode schema in an ENSDb? Please make all the naming and use of language 100% precise and clear. |
||
| */ | ||
| interface DrizzleDb extends NodePgDatabase<typeof schema> {} | ||
| interface DrizzleDb extends NodePgDatabase<typeof ensNodeSchema> {} | ||
|
|
||
| /** | ||
| * ENSDb Client | ||
|
|
@@ -53,16 +44,22 @@ export class EnsDbClient implements EnsDbClientQuery, EnsDbClientMutation { | |
| */ | ||
| private db: DrizzleDb; | ||
|
|
||
| /** | ||
| * ENSIndexer reference string for multi-tenancy in ENSDb. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Be more precise. Multi-tenancy of what exactly? Always write for an assumed audience that isn't familiar with all the context yet. |
||
| */ | ||
| private ensIndexerRef: string; | ||
|
|
||
| /** | ||
| * @param databaseUrl connection string for ENSDb Postgres database | ||
| * @param databaseSchemaName Postgres schema name for ENSDb tables | ||
| * @param ensIndexerRef reference string for ENSIndexer instance (used for multi-tenancy in ENSDb) | ||
| */ | ||
| constructor(databaseUrl: string, databaseSchemaName: string) { | ||
| constructor(databaseUrl: string, ensIndexerRef: string) { | ||
| this.db = makeDrizzle({ | ||
| databaseSchema: databaseSchemaName, | ||
| databaseUrl, | ||
| schema, | ||
| schema: ensNodeSchema, | ||
| }); | ||
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+56
to
60
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ship the This client now hard-depends on the Also applies to: 182-190 🤖 Prompt for AI Agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
🧠 Learnings used |
||
|
|
||
| this.ensIndexerRef = ensIndexerRef; | ||
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
vercel[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
|
|
||
| /** | ||
|
|
@@ -147,15 +144,21 @@ export class EnsDbClient implements EnsDbClientQuery, EnsDbClientMutation { | |
| * | ||
| * @returns selected record in ENSDb. | ||
| * @throws when more than one matching metadata record is found | ||
| * (should be impossible given the PK constraint on 'key') | ||
| * (should be impossible given the composite PK constraint on | ||
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * 'ensIndexerRef' and 'key'). | ||
| */ | ||
| private async getEnsNodeMetadata<EnsNodeMetadataType extends SerializedEnsNodeMetadata>( | ||
| metadata: Pick<EnsNodeMetadataType, "key">, | ||
| ): Promise<EnsNodeMetadataType["value"] | undefined> { | ||
| const result = await this.db | ||
| .select() | ||
| .from(ensNodeMetadata) | ||
| .where(eq(ensNodeMetadata.key, metadata.key)); | ||
| .from(ensNodeSchema.ensNodeMetadata) | ||
| .where( | ||
tk-o marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| and( | ||
| eq(ensNodeSchema.ensNodeMetadata.ensIndexerRef, this.ensIndexerRef), | ||
| eq(ensNodeSchema.ensNodeMetadata.key, metadata.key), | ||
| ), | ||
| ); | ||
|
|
||
| if (result.length === 0) { | ||
| return undefined; | ||
|
|
@@ -176,25 +179,16 @@ export class EnsDbClient implements EnsDbClientQuery, EnsDbClientMutation { | |
| private async upsertEnsNodeMetadata< | ||
| EnsNodeMetadataType extends SerializedEnsNodeMetadata = SerializedEnsNodeMetadata, | ||
| >(metadata: EnsNodeMetadataType): Promise<void> { | ||
| await this.db.transaction(async (tx) => { | ||
| // Ponder live-query triggers insert into live_query_tables. | ||
| // Because this worker writes outside the Ponder runtime connection pool, | ||
| // the temp table must be ensured to exist on this connection. Without this, | ||
| // the upsert would fail with "relation 'live_query_tables' does not exist" error. | ||
| await tx.execute( | ||
| sql`CREATE TEMP TABLE IF NOT EXISTS live_query_tables (table_name TEXT PRIMARY KEY)`, | ||
| ); | ||
|
|
||
| await tx | ||
| .insert(ensNodeMetadata) | ||
| .values({ | ||
| key: metadata.key, | ||
| value: metadata.value, | ||
| }) | ||
| .onConflictDoUpdate({ | ||
| target: ensNodeMetadata.key, | ||
| set: { value: metadata.value }, | ||
| }); | ||
| }); | ||
| await this.db | ||
| .insert(ensNodeSchema.ensNodeMetadata) | ||
| .values({ | ||
| ensIndexerRef: this.ensIndexerRef, | ||
| key: metadata.key, | ||
| value: metadata.value, | ||
| }) | ||
| .onConflictDoUpdate({ | ||
| target: [ensNodeSchema.ensNodeMetadata.ensIndexerRef, ensNodeSchema.ensNodeMetadata.key], | ||
| set: { value: metadata.value }, | ||
| }); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,11 @@ import config from "@/config"; | |
|
|
||
| import { EnsDbClient } from "./ensdb-client"; | ||
|
|
||
| // config.databaseSchemaName is unique per ENSIndexer instance and is used as the ensIndexerRef | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's big naming problems here. I should not have the responsibility to catch and fix these. You should notice how there's 3 different terms being used for exactly the same idea here and fix it before sending to me for review.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. After reading feedback you shared, I see more clearly how this could just be |
||
| // tenant key in the shared ENSNode schema (ensnode.*). | ||
| const ensIndexerRef = config.databaseSchemaName; | ||
|
|
||
| /** | ||
| * Singleton instance of {@link EnsDbClient} for use in ENSIndexer. | ||
| */ | ||
| export const ensDbClient = new EnsDbClient(config.databaseUrl, config.databaseSchemaName); | ||
| export const ensDbClient = new EnsDbClient(config.databaseUrl, ensIndexerRef); | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.