Tool to run CABLE configurations without payu

[Whyborn]

Why

We would like a light-weight runner that runs CABLE configurations designed for payu. We want a tool other than payu for 2 reasons:

1. payu only operates on Gadi, and that's unlikely to change anytime soon. We want to be able to run on any machine with a standard python installation.
2. payu is fairly heavy. We want to make something very light that people can fire off without thinking.

What

What should the tool actually be able to do?

1. Run a configuration in the same way as payu would (at the surface level- in that it reads the config.yaml, creates the same input and output directories) e.g. with run-config from the configuration directory.
2. Add metadata about the configuration being run to the output files of the run. The metadata should consist of:

• The configuration repository
• The configuration commit hash (with a -dirty for a modified config?)
• The date of the run
• Brief record of the inputs used

What it doesn't need to do:

1. Run multi-stage configurations- each configuration consists of a single call to cable.
2. Proper provenance tracking of inputs, past recording of file names. While this is a nice to have, I don't think is feasible while still being lightweight and portable. Minimal proper provenance would require checksum calculations, which can be a significant task for inputs spread over many files as atmosphere forcing often is. Some machines may have official catalogues with DOIs and such, but relying this is certainly not portable.

On recording the commit hash

Anyone wanting to run in a different location will inevitably have to modify the configuration, to point to the correct inputs on that machine. This will lead to a "dirty" repository. Is this desirable? Do we need to provide information about the diff? If we don't, then it's not possible to determine whether a dirty represents a change in the input paths, or a change in the science configuration. I would be in favour of preventing the config.yaml modifying the namelist altogether, and only configure the input locations and job submission. Then, if the repository is "dirty", we can provide a diff excluding the config.yaml? Or perhaps exclude the config.yaml from the "dirty" check?

My choice would be to ignore config.yaml changes in the "dirty" check, assuming we exclude namelist changes from the config.yaml. This way the "dirty" strictly relates to science choices.

On tracking inputs

As stated above, doing checksums on inputs I think is outside the remit of this tool, as it can easily get expensive. What we can do very cheaply is provide "date last modified" information about each of the inputs. This way, if configuration results change unexpectedly, we can quickly look at the "date last modified" for the inputs and determine whether this was a source of change.

Summary

The tool will allow someone to call config-run from a payu configuration directory, to run the configuration like payu run would. The tool will run on any machine that has a standard installation of python. The model outputs would be the same as it would be with payu, with the additional metadata applied to each of the outputs:

• `"configuration_repo": <repo_URL>"
• "configuration_hash": <commit_hash with optional "-dirty">
• "date_created": <ISO 8601 format date>
• "inputs": "<path_to_input> <date_last_modified ISO8601 format>\n
  <path_to_input> <date_last_modified ISO8601 format>\n"

@gabsun @SeanBryan51 How does this sound? Does this meet the specifications we outlined in our meeting? My only remaining question is how to distribute this- should it be a standalone package installable via pip or conda? Should it ship with CABLE directly?

[SeanBryan51]

Thanks for putting this together, I think it's a good summary of the discussions. Just a couple things which caught my eye:

Anyone wanting to run in a different location will inevitably have to modify the configuration, to point to the correct inputs on that machine. This will lead to a "dirty" repository. Is this desirable? Do we need to provide information about the diff? If we don't, then it's not possible to determine whether a dirty represents a change in the input paths, or a change in the science configuration. I would be in favour of preventing the config.yaml modifying the namelist altogether, and only configure the input locations and job submission. Then, if the repository is "dirty", we can provide a diff excluding the config.yaml? Or perhaps exclude the config.yaml from the "dirty" check?

My choice would be to ignore config.yaml changes in the "dirty" check, assuming we exclude namelist changes from the config.yaml. This way the "dirty" strictly relates to science choices.

I find the ignoring of specific files a bit complex for what it's worth. It would be nice if all the relevant input files for a configuration were version controlled and accounted for in the commit/version identifier.

To get around the issue of having files on various machines, you could use an environment variable which points to the root directory of the input file location which is machine specific, and have the rest of the input file path be relative to that environment variable. This would keep the version controlled configuration clean. ILAMB has functionality for installing a fixed directory tree of datasets, perhaps we could follow a similar approach for installing CABLE input files?

2. Proper provenance tracking of inputs, past recording of file names. While this is a nice to have, I don't think is feasible while still being lightweight and portable. Minimal proper provenance would require checksum calculations, which can be a significant task for inputs spread over many files as atmosphere forcing often is. Some machines may have official catalogues with DOIs and such, but relying this is certainly not portable.

How does checksum functionality hurt portability? I personally think this would be a nice to have, but not absolutely necessary as you say.

[Whyborn]

I find the ignoring of specific files a bit complex for what it's worth. It would be nice if all the relevant input files for a configuration were version controlled and accounted for in the commit/version identifier.

It's trivial to exclude a file from a diff. I think the goal of all relevant input files being version controlled is a truly impossible task for the full user base. We can leave that full provenance to payu, where we do have control over the machine. How can you possibly account for all the things people may do to an input? They'll never be stored on a versioning service, the best you can do is inspect the metadata for DOIs and things, and this is no guarantee that the data actual is the same as the original. The only way to get truly full provenance is to download the data from a remote source, but even then the host can modify the data.

To get around the issue of having files on various machines, you use an environment variable which points to the root directory of the input file location which is machine specific, and have the rest of the input file path be relative to that environment variable.

I can't see that ever being consistent enough to be used across machines. We can't control how people will download and store their inputs- for those working on their own computers, they can format the directories however they want, but other machines will publish the datasets to specific places, and won't want people duplicating them across the machine. They could symlink the files into a recognised structure? It seems like a lot of work for no gain, as this doesn't promise that the files are the same.

How does checksum functionality hurt portability? I personally think this would be a nice to have, but not absolutely necessary as you say.

It wouldn't hurt portability, but it would hurt the lightweight-ness. When I was doing the TRENDY runs with payu, computing the checksums took over an hour as each meteorological variable was split into years. This problem is unlikely to go away as we go to higher frequency meteorology.

[Whyborn]

After more thinking, I'm more firmly on the side of not using environment variables to set locations for inputs. If we want to be able to use the same configurations as payu, then making changes to the config.yaml to run on other machines is truly unavoidable- payu will never use environment variables in their configs to specify and inputs, and nor should they I think.

As long as we provide both a list of the input files used and the date they were last modified in the output metadata, and information about any other changes made to the configuration, then I think that should be sufficient provenance to determine what differs between configuration runs.

[SeanBryan51]

I personally don't mind the use of environment variables. Environment variables are the primary way for tools to search for libraries, or binaries on a given system, I don't see why a similar approach couldn't be done for input files. Take for example CMake with CMAKE_PREFIX_PATH or pkg-config with PKG_CONFIG_PATH.

So I guess I don't agree with payu's initial design of embedding an absolute path in the configuration file, I'd rather specify input file names and have payu search for these input files using a PATH like variable. Different way of doing things I suppose 🙃

[aidanheerdegen]

What are your use cases?

payu only operates on Gadi, and that's unlikely to change anytime soon

Not the case. We will shortly be working on porting to Pawsey, which uses slurm. A hacked version ofpayu currently works on Pawsey, so it does work, but we're planning on using HPCpy to provide cross-platform queue support.

[aidanheerdegen]

The model outputs would be the same as it would be with payu, with the additional metadata applied to each of the outputs:

• `"configuration_repo": <repo_URL>"
• "configuration_hash": <commit_hash with optional "-dirty">
• "date_created": <ISO 8601 format date>
• "inputs": "<path_to_input> <date_last_modified ISO8601 format>\n
  <path_to_input> <date_last_modified ISO8601 format>\n"

This could easily be accommodated with the addmeta tool that has been developed to support the ACCESS Output Data Specification.

You can see an example of how it works here:

https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/test-addmeta/addmeta

It is trivial to add global metadata to netCDF files in this way. Two of the fields mentioned above area already part of the standard.

[SeanBryan51]

Given CABLE is computationally inexpensive to run, the main use case would be for CABLE users to run CABLE payu configurations locally in addition to running on various HPC clusters.

It would be great if I could simply run the following on my laptop (assuming input files are setup correctly):

git clone git@github.com:CABLE-LSM/crujra_accessN96_1h.git; cd crujra_accessN96_1h
pip install payu
payu run

[SeanBryan51]

Given CABLE is computationally inexpensive to run, the main use case would be for CABLE users to run CABLE payu configurations locally in addition to running on various HPC clusters.

Some of the blockers to this are listed here, e.g. system dependency requirements (environment modules, PBS, MPI), hard coded paths in configurations (payu-org/payu#586)

[aidanheerdegen]

Given CABLE is computationally inexpensive to run, the main use case would be for CABLE users to run CABLE payu configurations locally in addition to running on various HPC clusters.

Would local use be via queue submission or directly? Noting that both are supported: payu run, payu-run

Some of the blockers to this are listed here, e.g. system dependency requirements (environment modules, PBS, MPI), hard coded paths in configurations (payu-org/payu#586)

All of which we'd like to fix/change.

Some of the issues are related to HPC queue configuration. If your local use doesn't use a queue that is a non-problem. If you're not using manifests the hard-coded paths therein are also not an issue.

Where do you expect input files to live? Given your use-case above, it seems like they'll be in the repo?

[SeanBryan51]

Where do you expect input files to live? Given your use-case above, it seems like they'll be in the repo?

That would probably depend on the configuration: forcing files for our flux site simulations are small enough that they could be committed to the configuration repo (we store this flux site input file in the CABLE repository as a simple test case for users to run). Gridded forcing data sets like the ones used here would have to live outside the configuration repo, in that case, users would have to pull the input data sets somewhere onto their machine and update config.yaml.

For the latter case, Lachlan and I were discussing if there was a way to avoid the configuration state becoming "dirty" when the user manually updates the input paths on a given machine. Could we specify paths in config.yaml to be relative (or just even file names), and have payu search for these inputs using a PATH-like environment variable?

[aidanheerdegen]

That would probably depend on the configuration: forcing files for our flux site simulations are small enough that they could be committed to the configuration repo (we store this flux site input file in the CABLE repository as a simple test case for users to run)

As an example of how this works you can check out the test bowl1 MOM5 ocean configuration being updated in this PR
.

Gridded forcing data sets like the ones used here would have to live outside the configuration repo, in that case, users would have to pull the input data sets somewhere onto their machine and update config.yaml.

For the latter case, Lachlan and I were discussing if there was a way to avoid the configuration state becoming "dirty" when the user manually updates the input paths on a given machine. Could we specify paths in config.yaml to be relative (or just even file names), and have payu search for these inputs using a PATH-like environment variable?

Similar to above you could have that data live in a sub-directory in the control directory and have a payu init userscript that downloads the data (or just a script you ask people to run).

So have the paths pre-baked into the config.yaml and even the input manifest, but not committed to the repo. That way you can require reproducibility on the input manifest to provide a check that the data is as expected.

That is the cleanest way to do it. payu already supports search files in an input directory in the laboratory. So you can put them there and it will find them, or we could fairly easily add support for additional search paths to payu. Another option is to make a link in the control dir to where the input files are located. I think that works.

Basically, I think there are a number of different ways to support this.

[SeanBryan51]

Thanks @aidanheerdegen super useful stuff!

[Whyborn]

This could easily be accommodated with the addmeta tool that has been developed to support the ACCESS Output Data Specification.

I was thinking about whether we could leverage addmeta, but I wasn't sure how to support the inclusion of the input files and their date last modified.

Some of the blockers to this are listed here, e.g. system dependency requirements (environment modules, PBS, MPI), hard coded paths in configurations (payu-org/payu#586)

All of which we'd like to fix/change.

I know these are on the list of things to change, but realistically, what is the timeline for actually getting them fixed? It seems like a low priority, and as usual, we're pretty (people) resource limited.

Some of the issues are related to HPC queue configuration. If your local use doesn't use a queue that is a non-problem. If you're not using manifests the hard-coded paths therein are also not an issue.

Where do you expect input files to live? Given your use-case above, it seems like they'll be in the repo?

It will be both- people will be running on their local machines and a range of HPC machines. I don't think the data will live with the repository- the example dataset @SeanBryan51 mentioned is the exception (or at least I think it should be). There'll definitely be configurations that have many GB of inputs, we definitely won't be porting them with the repo.

Similar to above you could have that data live in a sub-directory in the control directory and have a payu init userscript that downloads the data (or just a script you ask people to run).

The idea of having a having a payu init script is an interesting one, but how would a user specify that they already have the data available to use? We definitely don't want to download the data every time the configuration is run.

Basically, regardless of what methods we use, I don't see how the specifying of inputs doesn't result in a "dirty" repository when running on any machine other than the NCI machine (or maybe Pawsey if we ever end up writing configurations specifically for there). I'm pretty strongly opposed to using environment variables, my experience is that they're not very visible and lead to problems later.

[Whyborn]

@aidanheerdegen Has anyone tried to run payu locally? I don't find it obvious precisely which bits of payu are active when calling payu run vs payu-run.

If I was to update hpcpy to run agnostically on local machines as well as Slurm and PBS, would payu be ready to go on other machines? I'm assuming not yet?