🩹 [Patch]: Consolidate test setup/teardown tasks

[MariusStorhaug]

Describe the change

Now that Process-PSModule supports BeforeAll and AfterAll scripts, that run once before and after all the parallel tests, we should move some of the setup/teardown tasks to these. This will hopefully speed up the testing process, and result in fewer failures due to ratelimiting.

Problem

The primary bottleneck is secondary rate limits on repository creation. Multiple test files independently create their own repositories per auth case, resulting in a large burst of New-GitHubRepository calls that trigger GitHub's secondary rate limits.

Current repo creation per auth case (5 active cases)

Test File	Repos Created	Other Resources
Repositories.Tests	7	5 teams (org auth), forks, templates
Secrets.Tests	3	Org secrets, repo secrets, environment + env secrets
Variables.Tests	3	Org variables, repo variables, environment + env variables
Releases.Tests	1	5 releases, 6+ release assets
Environments.Tests	0 (uses shared)	Environments
Organizations.Tests	0	1 org (enterprise only), app installations, invitations
Teams.Tests	0	6 teams (org only)
Total per auth case	~14
Total across 5 cases	~70

On top of that, BeforeAll.ps1 creates 1 repo per case (5 total). So a full test run attempts ~75 repository creations plus cleanup.

Goal

Reduce repository creations from ~75 to ~40 by pre-provisioning shared repos in BeforeAll.ps1 and having test files consume them via Get-GitHubRepository instead of New-GitHubRepository. Test files that inherently test repo CRUD (Repositories.Tests) keep their own creation logic.

Target test file pattern

BeforeAll {
    $testName = 'TestName'
    $os = $env:RUNNER_OS
    $id = $env:GITHUB_RUN_ID
}

Describe 'TestName' {
    $authCases = . "$PSScriptRoot/Data/AuthCases.ps1"

    Context 'As <Type> using <Case> on <Target>' -ForEach $authCases {
        BeforeAll {
            $context = Connect-GitHubAccount @connectParams -PassThru -Silent
            if ($AuthType -eq 'APP') {
                $context = Connect-GitHubApp @connectAppParams -PassThru -Default -Silent
            }
            # Reference the shared repo (NOT New-GitHubRepository)
            $repoPrefix = "Test-$os-$TokenType"
            $repoName = "$repoPrefix-$id"
            $repo = Get-GitHubRepository -Owner $Owner -Name $repoName
        }

        AfterAll {
            Get-GitHubContext -ListAvailable | Disconnect-GitHubAccount -Silent
        }

        It 'Should do something' -Skip:($OwnerType -in ('repository', 'enterprise')) {
            # Test logic using $repo, $Owner, $repoName
        }
    }
}

Key conventions:

• $id = $env:GITHUB_RUN_ID — not [guid]::NewGuid() or Get-Random
• Get-GitHubRepository — test files fetch the shared repo, they do not create repos
• -Skip:($OwnerType -in ('repository', 'enterprise')) — standard skip condition for repo-dependent tests
• Test-specific ephemeral resources (releases, secrets, variables, environments, teams) are still created/cleaned within each test file

Shared infrastructure design

tests/BeforeAll.ps1

For each auth case (except GITHUB_TOKEN):

1. Clean up stale resources from previous failed runs (repos matching Test-$os-$TokenType-*)
2. Create shared repositories:
   • 1 base repo per case per OS: Test-{OS}-{TokenType}-{RunID} — used by Environments, Releases, general tests
   • 3 additional repos per case: Test-{OS}-{TokenType}-{RunID}-1, -2, -3 — for Secrets/Variables selected-repository testing
3. Create shared teams (org auth cases only): admin, maintain, push, triage, pull — for Repositories.Tests Permissions context
4. Create shared environments on the base repo — for Secrets/Variables environment-scope testing

tests/AfterAll.ps1

For each auth case (except GITHUB_TOKEN):

1. Remove org-level secrets and variables matching test prefixes
2. Remove teams matching test prefixes
3. Remove all test repositories matching Test-{OS}-{TokenType}-* prefix
4. Remove test organizations (enterprise auth only)

Special patterns

Fork and template repos (Repositories.Tests)

• Uses PSModule/Template-Action and PSModule/Template-Docs as template sources (external, not created)
• Uses psmodule-test/fork-{OS} as fork source
• Fork and template repo creation tests inherently need New-GitHubRepository — cannot be consolidated

Selected-repository testing (Secrets.Tests, Variables.Tests)

• Both need 3 repos to test Add/Remove/Set-GitHub{Secret,Variable}SelectedRepository
• These 3 repos are shared between Secrets and Variables tests via BeforeAll

Organization lifecycle (Organizations.Tests)

• Creates orgs under the msx enterprise, installs apps, sends invitations
• Inherently a lifecycle test — cannot pre-create the orgs

---

Migration checklist

Migration is done in PR #541

Shared infrastructure (prerequisite for heavy migrations)

• BeforeAll.ps1 — Create 1 base repo per auth case per OS
• AfterAll.ps1 — Clean up repos matching prefix
• BeforeAll.ps1 — Create 3 additional repos per case for Secrets/Variables ($repoName-1, -2, -3)
• BeforeAll.ps1 — Create shared environments on base repo
• BeforeAll.ps1 — Create 5 shared teams per org auth case
• AfterAll.ps1 — Clean up org-level secrets/variables matching test prefixes
• AfterAll.ps1 — Clean up teams matching test prefixes

Lightweight — no repo changes, add top-level BeforeAll, move from tmp/ to tests/

• GitHubFormatter.Tests — Pure unit tests, no API calls. Just move back
• Emojis.Tests — Add top-level BeforeAll
• Enterprise.Tests — Add top-level BeforeAll
• GitHub.Tests — Add top-level BeforeAll
• Permissions.Tests — Add top-level BeforeAll
• Apps.Tests — Add top-level BeforeAll
• Artifacts.Tests — Add top-level BeforeAll (uses $env:GITHUB_REPOSITORY_* for current repo — unique pattern, keep as-is)
• Users.Tests — Add top-level BeforeAll; convert inline if ($OwnerType -eq 'user') to -Skip:() pattern

Medium — align naming, use shared repos where applicable

• Environments.Tests — Migrated. Uses Get-GitHubRepository, $id = $env:GITHUB_RUN_ID
• Teams.Tests — Change $guid → $id = $env:GITHUB_RUN_ID; remove prefix cleanup from BeforeAll; keep team cleanup in AfterAll
• Organizations.Tests — Change $number = Get-Random → $id = $env:GITHUB_RUN_ID; enterprise org lifecycle is inherent — keep own setup
• Releases.Tests — Change $guid → $id; replace New-GitHubRepository with Get-GitHubRepository; remove repo cleanup; shared repo must have at least 1 commit (README)

Heavy — restructuring + depends on shared infrastructure expansion

• Secrets.Tests — Change $guid → $id; replace 3× New-GitHubRepository with Get-GitHubRepository; remove repo cleanup; keep secret cleanup in AfterAll. Needs 3 shared repos + 1 shared environment
• Variables.Tests — Nearly identical to Secrets; change $guid → $id; replace 3× New-GitHubRepository with Get-GitHubRepository; remove repo cleanup; keep variable cleanup. Needs 3 shared repos + 1 shared environment
• Repositories.Tests — Change $guid → $id; special case — repo CRUD IS the test, so New-/Set-/Remove-GitHubRepository calls must stay for mutation tests; use shared repo for Get- read-only tests; remove prefix cleanup from BeforeAll; keep AfterAll cleanup. Needs 5 shared teams (org only)

---

Progress summary

• 4 of 22 tasks completed (BeforeAll base, AfterAll base, Environments.Tests, shared infra foundation)
• 14 test files remaining in tmp/ pending migration
• 8 lightweight, 3 medium, 3 heavy
• 5 shared infrastructure expansion tasks needed before heavy migrations