Allow ENSIndexer to run ENSDb migrations and populate ENSNode Metadata without waiting for ENSRainbow

[tk-o]

Currently, an ENSIndexer instance has a dependency on the related ENSRainbow instace, such that until the ENSRainbow instance is ready to serve HTTP traffic, the ENSIndexer instance:

1. Cannot start indexing onchain data.
2. Cannot determine the EnsIndexerPublicConfig object, which prevents ENSDb Writer Worker from writing EnsIndexerPublicConfig record to ENSDb, and as a result, crashes the ENSIndexer instance.

In a cold-start scenario, it might take more than 30 minutes for the ENSRainbow instance to reach the readiness and open its HTTP server.

While the 1st issue remains unadressed, we can and should address the 2nd isssue.

We could make the ensRainbowPublicConfig field on EnsIndexerPublicConfig data model optional. By doing so, the ENSIndexer instance will be able to determine the EnsIndexerPublicConfig object even during the ENSRainbow instance cold-start period. Having EnsIndexerPublicConfig object, ENSDb Writer Worker will be able to populate ENSDb with ENSNode Meetadata (including ENSIndexer Public Config, and Indexing Status where all chains in queued status).

Key points to reflect on while designing the solution for this issue:

1. The deeper goal here is we want to make it so that the startup dependencies across our services are softened to improve the experience of operators launching services.
2. Fetching the EnsRainbowPublicConfig object will require multiple retries. Once the config object was fetched successfully, it should remain cached indefinitely.
3. When EnsRainbowPublicConfig object becomes available in the cache, the ENSDb Writer Worker should update the ENSIndexer Public Config record in ENSNode Metadata table to and update its ensRainbowPublicConfig column from undefined to the cached EnsRainbowPublicConfig object.
4. Changing the type of EnsIndexerPublicConfig['ensRainbowPublicConfig'] field from EnsIndexerPublicConfig to EnsIndexerPublicConfig | underfined will create side effects on EnsIndexerPublicConfig data model consumers, for example, ENSApi, ENSAdmin. Please make sure to take that into consideration while designing the updates for this issue.

[coderabbitai]

🔗 Related PRs

#1613 - feat(ensindexer): introduce Indexing Status Builder module [merged]
#1702 - ENSDb Writer Worker for ENSIndexer [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[lightwalker-eth]

@tk-o Thanks for logging this!

Another downstream consequence to consider here is how it will no longer be appropriate for ENSApi to assume that the ENSIndexerPublicConfig it loads from ENSDb is immutable. This is ok and I see how we can solve this in a good way. Happy to discuss when you're ready. Just flagging how this is also a factor to consider.

[tk-o]

@lightwalker-eth yes, there's a nice way to handle the situation where ENSApi needs to keep reading ENSIndexer Public Config from ENSDb until it's the "full" config with ensRainbowPublicConfig field defined.

Suggested approach can be found in this Slack post, point 3.

[tk-o]

Related to #1417

[tk-o]

There was an idea discussed on Slack to simplify the public config data model for all services and to introduce "stack info" data model:

These "Stack Info" objects is where we can define invariants about the relationships between the "Public Config" objects in a broader "stack" of services.

Examples

EnsRainbowStackInfo

{
  ensRainbow: {
    publicConfig: EnsRainbowPublicConfig;
  };
}

EnsIndexerStackInfo

{
  ensIndexer: {
    publicConfig: EnsIndexerPublicConfig;
  };

  ensRainbow?: {
    publicConfig: EnsRainbowPublicConfig;
  };
}

EnsApiStackInfo

{
  ensApi: {
    publicConfig: EnsApiPublicConfig;
  };

  ensDb: {
    publicConfig: EnsDbPublicConfig;
  };

  ensIndexer: {
    publicConfig: EnsIndexerPublicConfig;
  };

  ensRainbow?: {
    publicConfig: EnsRainbowPublicConfig;
  };
}