Allow worker nodes to accept work from any active build, not just 'lock' to a single build at a time

[baronfel]

Summary

Now that we have the safety guarantees that the Multithreaded APIs (and marker interfaces) provide, we can consider an exciting feature - allowing worker nodes to do work across multiple active builds, taking work from whichever builds are active instead of being constrained to a single logical build for the duration of that build.

Before these APIs, this was too risky because Tasks couldn't be trusted to be stateless - but with the APIs (and some additional documentation clarifying how stateful constructs like C# statics should be used safely) we can experiment with this possibility, which should help prevent the thundering-herd problem of multiple concurrent builds spawning NUMPROCS nodes that we have today.

Background and Motivation

Fill in later

Proposed Feature

Fill in later

Alternative Designs

No response

[KalleOlaviNiemitalo]

Will this interfere with KillTree? I mean if the parent process of the node no longer indicates which build it is working on.

[baronfel]

@KalleOlaviNiemitalo for worker nodes that are kept alive via node reuse, the worker nodes already lose the parent-PID relationship. They are re-parented to PID 1 on Unix after the build that led to their initial creation is finished.

[jeroenjanssen-cpp]

will this also work correctly on azure-devops-pipeline agents if there are multiple agents running on a windows buildserver?
(both the node-reuse and killing the relevant msbuild nodes once an agent finishes with its build job)