Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a39f9a0
Updates for spack-stack: gsibec, fms, pfunit
mathomp4 Feb 5, 2026
fe1f147
Add gsibec 1.4.1
mathomp4 Feb 5, 2026
63f8354
NAS updates
mathomp4 Feb 5, 2026
336eb89
Update gsibec in many files
mathomp4 Feb 5, 2026
95e9a3f
Updates for ifx stack
mathomp4 Feb 9, 2026
ecded67
Preliminary NAS TOSS5 setup
mathomp4 Feb 9, 2026
6f6f456
Fix for concretize oddity
mathomp4 Feb 12, 2026
decf0d4
Clean up packages.yamls
mathomp4 Feb 12, 2026
33df5cc
Merge remote-tracking branch 'jcsda/develop' into mathomp4-updates
mathomp4 Feb 12, 2026
00bbb6b
Fix fms
mathomp4 Feb 12, 2026
bcbc12a
Fix for Toss5
mathomp4 Feb 13, 2026
1e8ab3c
Update readme
mathomp4 Feb 13, 2026
a6e4e75
Fix for gnu
mathomp4 Feb 13, 2026
0c7883e
Fix gnu
mathomp4 Feb 13, 2026
24bf8aa
Fix for openmpi
mathomp4 Feb 13, 2026
f27d54b
Update to spack-packages@37965a7fc583e94c50cb0748199cf2d9b346be2d
mathomp4 Feb 14, 2026
e5f3bab
Move to CMake 3 for ecflow issue
mathomp4 Feb 17, 2026
5079134
Build p4est with -lm on gnu
mathomp4 Feb 17, 2026
0b58543
Fix bad commit
mathomp4 Feb 17, 2026
9a7319b
Updates for GCC 14 with hdf-eos2
mathomp4 Feb 17, 2026
05da1fe
More GNU fixes
mathomp4 Feb 18, 2026
f1ba211
Update readmes
mathomp4 Feb 18, 2026
bbc5959
Merge remote-tracking branch 'jcsda/develop' into mathomp4-updates
mathomp4 Feb 19, 2026
2f910c5
Update repos/builtin hash
mathomp4 Feb 19, 2026
c4f2c3a
Update to gsibec@1.4.2
mathomp4 Feb 19, 2026
bff28c1
Merge remote-tracking branch 'jcsda/develop' into mathomp4-updates
mathomp4 Feb 19, 2026
b69b426
Update repos/builtin hash to 340a3da5d2
mathomp4 Feb 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions configs/common/packages.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,12 @@ packages:
- '@2.6.4'
fms:
require:
- '@2024.02'
- '@2025.03'
- precision=32,64
- +quad_precision
- +openmp
- +pic
- build_type=Release
- +deprecated_io
- any_of:
- +gfs_phys constants=GFS
- ~gfs_phys constants=GEOS
Expand Down Expand Up @@ -199,7 +198,7 @@ packages:
- '@1.4.0'
gsibec:
require:
- '@1.2.1'
- '@1.4.2'
gsi-ncdiag:
require:
- '@1.1.2'
Expand Down Expand Up @@ -349,6 +348,10 @@ packages:
pflogger:
require:
- +mpi
pfunit:
require:
- +mpi
- +fhamcrest
pixman:
require:
- +pic
Expand Down Expand Up @@ -480,10 +483,6 @@ packages:
snappy:
require:
- +shared
sp:
require:
- '@2.5.0'
- precision=4,d,8
udunits:
require:
- '@2.2.28'
Expand Down
2 changes: 1 addition & 1 deletion configs/containers/specs/jedi-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
fms@2024.02,
g2@3.5.1,
g2tmpl@1.17.0,
gsibec@1.2.1,
gsibec@1.4.2,
hdf@4.2.15,
hdf5@1.14.5,
ip@5.4.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/sites/tier1/acorn/packages_intel-19.1.3.304.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ packages:
go:
require: ['%gcc']
gsibec:
require: ['@1.2.1']
require: ['@1.4.2']
harfbuzz:
require: ['%gcc']
libffi:
Expand Down
2 changes: 1 addition & 1 deletion configs/sites/tier1/acorn/packages_oneapi-2024.2.1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ packages:
glib:
require: ['%gcc']
gsibec:
require: ['@1.2.1']
require: ['@1.4.2']
libtiff:
require: ['build_system=cmake']
nco:
Expand Down
2 changes: 1 addition & 1 deletion configs/sites/tier1/aws-pcluster/packages_intel.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ packages:
- '@1.2.0 ~mkl +fftw'
gsibec:
require::
- '@1.2.1 ~mkl'
- '@1.4.2 ~mkl'
py-numpy:
require::
- '^openblas'
Expand Down
2 changes: 1 addition & 1 deletion configs/sites/tier1/hera/packages_intel.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ packages:
- '@1.2.0 ~mkl +fftw'
gsibec:
require::
- '@1.2.1 ~mkl'
- '@1.4.2 ~mkl'
py-numpy:
require::
- '^openblas'
Expand Down
2 changes: 1 addition & 1 deletion configs/sites/tier1/hera/packages_oneapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ packages:
- '@1.2.0 ~mkl +fftw'
gsibec:
require::
- '@1.2.1 ~mkl'
- '@1.4.2 ~mkl'
py-numpy:
require::
- '@1.26'
Expand Down
246 changes: 246 additions & 0 deletions configs/sites/tier1/nas-toss5/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,246 @@
# How to Build **spack-stack** at NAS on TOSS5

This guide documents how to build **spack-stack** on NASA NAS TOSS5 systems. The previous install process used three stages across compute and login nodes, but recent testing shows installation can now be done directly on a compute node.

---

## Table of Contents

- [Overview](#overview)
- [Machines Required](#machines-required)
- [Clone spack-stack](#clone-spack-stack)
- [Obtain an Interactive Compute Node](#obtain-an-interactive-compute-node)
- [Setup spack-stack](#setup-spack-stack)
- [Create Environments](#create-environments)
- [oneAPI - ifx Environment](#oneapi---ifx-environment)
- [oneAPI - ifort Environment](#oneapi---ifort-environment)
- [GCC Environment](#gcc-environment)
- [Activate the Environment](#activate-the-environment)
- [Concretize the Environment](#concretize-the-environment)
- [Create Source Cache (LOGIN NODE ONLY)](#create-source-cache-login-node-only)
- [Pre-Fetch Cargo Dependencies (LOGIN NODE ONLY)](#pre-fetch-cargo-dependencies-login-node-only)
- [Install Packages (COMPUTE NODE)](#install-packages-compute-node)
- [Update Module Files](#update-module-files)
- [Deactivate the Environment](#deactivate-the-environment)
- [Debugging Package Builds](#debugging-package-builds)
- [Deprecated: Legacy Three-Step Install](#deprecated-legacy-three-step-install)

---

## Overview

NAS login nodes and compute nodes have different constraints:

- **Login nodes**:
- Have internet access
- Limited to **2 processes**

- **Compute nodes** (Turin):
- No internet access
- Allow parallel builds

Use login nodes for setup steps that require internet access (e.g., concretization, mirroring sources, pre-fetching Cargo dependencies), then run installation on a compute node.

The install step itself is now a single compute-node command.

---

## Machines Required

You will need:

- **A login node**
Used for setup steps that require internet access.

- **A Turin compute node**
Used for package installation with higher parallelism.

---

## Clone spack-stack

Use the appropriate branch or tag:

```bash
git clone --recurse-submodules https://github.com/JCSDA/spack-stack.git -b spack-stack-2.1.0 spack-stack-2.1.0
```

---

## Obtain an Interactive Compute Node

NAS login nodes allow only **2 processes**, so use:

```bash
qsub -I -V -X -l select=1:ncpus=128:mpiprocs=128:model=tur_ath -l walltime=12:00:00 -W group_list=s1873 -m b -N Interactive
```

This gives a **Turin** compute node for up to 12 hours.

---

## Setup spack-stack

Run on a **login node with internet**:

```bash
cd spack-stack-2.1.0
. setup.sh
```

---

## Create Environments

You only need to create each environment once.

### oneAPI - ifx Environment

```bash
spack stack create env --name ue-oneapi-2025.3.0 --template unified-dev --site nas-toss5 --compiler=oneapi-2025.3.0
cd envs/ue-oneapi-2025.3.0
```

### oneAPI - ifort Environment

```bash
spack stack create env --name ue-oneapi-2024.2.0 --template unified-dev --site nas-toss5 --compiler=oneapi-2024.2.0
cd envs/ue-oneapi-2024.2.0
```

### GCC Environment

```bash
spack stack create env --name ue-gcc-14.2.1 --template unified-dev --site nas-toss5 --compiler=gcc-14.2.1
cd envs/ue-gcc-14.2.1
```

---

## Activate the Environment

```bash
spack env activate .
```

> **Important:** Run this in *every* terminal where you plan to run Spack commands.

---

## Concretize the Environment

Run on a **login node** (internet required for bootstrapping Clingo and other tools):

```bash
spack concretize 2>&1 | tee log.concretize ; bell
```

### Optional `bell` helper

```bash
bell() { tput bel ; printf "\nFinished at: " ; date; }
```

---

## Create Source Cache (LOGIN NODE ONLY)

This downloads all source tarballs for your environment:

```bash
spack mirror create -a -d /swbuild/gmao_SIteam/spack-stack/source-cache
```

> ⚠️ **Do not run this outside an activated environment.**
> Otherwise Spack will attempt to mirror **every** known package/version.

---

## Pre-Fetch Cargo Dependencies (LOGIN NODE ONLY)

Rust packages frequently require network access during build. Pre-fetch their dependencies:

```bash
export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache
../../util/fetch_cargo_deps.py
```

> ⚠️ **Set `CARGO_HOME` on compute nodes** before running `spack install`.

---

## Install Packages (COMPUTE NODE)

Run installation on a **compute node**:

```bash
export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache
spack install -j 16 --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install ; bell
```

This replaces the former compute → login → compute install sequence.

> **Note:** You may need to re-run this command multiple times. Some builds fail intermittently but succeed on retry.

---

## Update Module Files

After installation completes, run:

```bash
spack module tcl refresh -y --delete-tree ; bell
spack stack setup-meta-modules
```

---

## Deactivate the Environment

```bash
spack env deactivate
```

---

## Debugging Package Builds

```bash
spack clean
spack stage <package>
spack build-env <package> -- bash --norc --noprofile
```

This drops you into a clean build environment with the package’s full compiler/runtime environment loaded.

---

## Deprecated: Legacy Three-Step Install

> ⚠️ **Deprecated:** Keep this only as historical reference. Prefer the single compute-node install above.

Older workflows used three stages:

1. Compute node: build dependencies for Rust-related Python packages and ecFlow.
2. `athfe` login node: build `py-cryptography`, `py-maturin`, `py-rpds-py`, and `ecflow` with `-j 2`.
3. Compute node: run full `spack install` to finish remaining packages.

Typical commands were:

```bash
# Step 1 (compute node)
export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache
spack install -j 16 --verbose --fail-fast --show-log-on-error --no-check-signature \
--only dependencies py-cryptography py-maturin py-rpds-py ecflow 2>&1 | tee log.install.deps-for-rust-and-ecflow ; bell

# Step 2 (athfe login node)
export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache
spack install -j 2 -p 1 --verbose --fail-fast --show-log-on-error --no-check-signature \
py-cryptography py-maturin py-rpds-py ecflow 2>&1 | tee log.install.rust-and-ecflow ; bell

# Step 3 (compute node)
export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache
spack install -j 16 --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install.after-cargo ; bell
```

---
9 changes: 9 additions & 0 deletions configs/sites/tier1/nas-toss5/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
config:
build_jobs: 6

# Overrides for spack build and staging areas to speed up builds
# and avoid errors with Lustre file locking and xattr issues
build_stage: /swbuild/gmao_SIteam/spack-stack/cache/build_stage
test_stage: /swbuild/gmao_SIteam/spack-stack/cache/test_stage
source_cache: /swbuild/gmao_SIteam/spack-stack/cache/source_cache
misc_cache: /swbuild/gmao_SIteam/spack-stack/cache/misc_cache
8 changes: 8 additions & 0 deletions configs/sites/tier1/nas-toss5/mirrors.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
mirrors:
local-source:
url: file:///swbuild/gmao_SIteam/spack-stack/source-cache
binary: false
local-binary:
url: file:///swbuild/gmao_SIteam/spack-stack/build-cache
binary: true

8 changes: 8 additions & 0 deletions configs/sites/tier1/nas-toss5/modules.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
modules:
default:
enable::
- tcl
tcl:
include:
# List of packages for which we need modules that are blacklisted by default
- python
Loading