Skip to content

feat: framework for parallelizable statistic computation#77

Open
laggycomputer wants to merge 35 commits intomainfrom
parallelizable_stats
Open

feat: framework for parallelizable statistic computation#77
laggycomputer wants to merge 35 commits intomainfrom
parallelizable_stats

Conversation

@laggycomputer
Copy link
Copy Markdown
Collaborator

@laggycomputer laggycomputer commented Jan 28, 2026

  • Introduce the SiteComposable trait, guaranteeing that a GlobalStatistic can be computed from its parts indepedently, concurrently, and potentially out-of-order.
  • Use this trait to create generic code capable of parallelizing any such statistic. There is an example instead.

Comment thread src/stats.rs Outdated
@laggycomputer laggycomputer force-pushed the parallelizable_stats branch from 15aae01 to 63569fc Compare April 2, 2026 18:45
@laggycomputer laggycomputer marked this pull request as ready for review April 14, 2026 20:46
@laggycomputer laggycomputer requested a review from molpopgen April 14, 2026 20:47
@molpopgen
Copy link
Copy Markdown
Member

It seems that we need some tests? A computation done in parallel should be "close enough" to the serial version?

Copy link
Copy Markdown
Collaborator Author

@laggycomputer laggycomputer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We definitely need tests, and we should at least prepare, if not also provide, a sketch demonstrating the use of indexed queries to do parallelism without the bottleneck of writing and cloning a records on a single thread

is this test still meaningful?
@laggycomputer
Copy link
Copy Markdown
Collaborator Author

As discussed today:

  • Let's sanity check at least one other stat.
  • We should combine components in an order besides the conventional left-to-right fold, to verify that this works as intended.

@molpopgen
Copy link
Copy Markdown
Member

@laggycomputer -- is this ready for more look/review?

@laggycomputer
Copy link
Copy Markdown
Collaborator Author

We're still missing one other stat, but feel free to look at the test for pi to see if it makes sense

@molpopgen
Copy link
Copy Markdown
Member

I think that the strategy looks sound. Boy, it would be handy to abstract this out for use for other stats. (Can definitely not do that for this PR though!)

Copy link
Copy Markdown
Member

@molpopgen molpopgen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Only issue (which we can address later) is if we want anything "parallel" to be feature-gated? I'm agnostic on that at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants