Consider including `securesystemslib[crypto]` as a dependency in TUF

[rroctavian]

Issue Description

The purpose of this issue is to kindly ask whether listing securesystemslib[crypto] as a dependency would improve the out-of-the-box experience with python-tuf.

I noticed that pip3 install tuf did not upgrade the pre-existing, old cryptography==3.4.8 present in a standard python installation. (The exact situation is an AWS Ubuntu 22.04 machine where python3 comes pre-installed with cryptography==3.4.8.) A more recent version like cryptography>=37.0.0 is required to perform a tuf.ngclient.Updater.download_target operation.

cryptography>=37.0.0 is listed as a dependency of the custom install securesystemslib[crypto] of securesystemslib, but not for the base install. Unfortunately, tuf==3.1.0 only mentions securesystemslib[crypto] in requirements/main.txt, but not as a dependency in pyproject.toml.

Reproduce issue

The issue becomes evident during signature verification processes, where the older cryptography library cannot correctly handle the signatures. Here are relevant snippets from the logs:

# create dirs  
mkdir -p ~/.tuf_import_error_issue/metadata ~/.tuf_import_error_issue/tmp  
# get root.json  
curl -o ~/.tuf_import_error_issue/metadata/root.json https://raw.githubusercontent.com/sigstore/root-signing/main/ceremony/2022-10-18/repository/5.root.json  

Please find attached the python file which generates the error and its logs below. You should be able to run the python script from anywhere as it has the paths indicated above hard-coded for this example.
tuf_import_error_issue.py.txt

Logs

DEBUG - tuf/ngclient/_internal/trusted_metadata_set.py:98 - Updating initial trusted root  
  
INFO - securesystemslib/signer/_key.py:429 - Key xyz...123 failed to verify sig: 'pyca/cryptography' library required  
  
INFO - tuf/api/metadata.py:744 - Key xyz...123 failed to verify root  
...  
tuf.api.exceptions.UnsignedMetadataError: root was signed by 0/3 keys  

Summary

• python-tuf lists securesystemslib>=0.26.0 as a dependency but does not specify that it should include the [crypto] extras.
• When an outdated version of the cryptography library is already installed, installing python-tuf does not prompt an upgrade to meet securesystemslib[crypto]'s requirements, leading to potential signature verification issues.
• Kindly consider including securesystemslib[crypto] as a direct dependency for python-tuf. This change would ensure that the necessary cryptography version is installed or upgraded during python-tuf's installation, mitigating issues related to outdated dependencies and improving the out-of-the-box security and reliability of python-tuf, especially in environments where dependency management is crucial.

I appreciate that managing dependencies is a delicate balance and I am curious to hear your thoughts. Please let me know if there is any further information I could help with.

[jku]

The suggestion has a lot of merit... The issue is that

• python-tuf should also be usable in apps that do not want cryptography
• python packaging, as I understand it, does not support the idea of "default but optional" dependencies. There's plenty of discussion in the python ecosystem but likely no near future improvements: https://discuss.python.org/t/adding-a-default-extra-require-environment/4898

For background, the "should be usable without cryptography" requirement comes from the idea that python-tuf wants to be potentially vendorable in pip (and cryptography as natively built package is not acceptable there)... It could be acceptable that python-tuf had a default cryptography dependency but we just expect the vendoring process to break that dependency (and make sure it's easy to do). The issue here would be, how do we ensure python-tuf remains easily "usable without cryptography" in the future: I suppose we can somehow make sure there are still tests that run without cryptography

[rroctavian]

@jku, thank you for getting back.

Given pyca/cryptography's page says

Our goal is for it (i.e. pyca/cryptography) to be your "cryptographic standard library".

I was wondering if pyca/cryptography might have a timeline of their own to become vendorable-friendly?

• python-tuf should also be usable in apps that do not want cryptography

... python-tuf wants to be potentially vendorable in pip (and cryptography as natively built package is not acceptable there)...

Please let me know if I am getting the point here. The way I see it, python-tuf would either:

• Extract the pyca/cryptography bits that are needed.
  • complicated by the fact that it goes via securesystemslib
  • is it obvious whether that would include any Rust code, which I am guessing is what makes cryptography un-vendorable?
• Choose another cryptographic package to rely on, a pure-Python one.
  • are there any viable alternatives here?
• Reimplement cryptographic logic, tailoring it to python-tuf needs.
  • is this feasible at all?

Are there, by any chance, any issues/conversations with active initiatives on this topic? I'd be happy to have a read and fill in my gaps.

I am slightly concerned that this issue might be off-putting to new users and have an impact on the adoption of python-tuf in favour of e.g. go-tuf CLI, thus resulting in fewer eyes here. The reason is that the problem arises in the first few minutes of using the package and it is might not be trivial to explain initially. One would either need to perform break-point debugging to find the import error or would need to have the logging properly configured. It does not help that securesystemslib logger has to be grabbed and forced to output at DEBUG level, otherwise the issue goes unnoticed in the first read of the logs. That is also not obvious.

The way I see it, we either live with the current inconvenience while working on becoming pip-vendorable, or patch it to help new users and continue thinking about the best solution.

Would it be acceptable at all to list securesystemslib[crypto] as a dependency in the meantime? I am curious to hear thoughts on this.

[jku]

I think this might be a good idea, and I'm even thinking python-tuf might not have to do anything here except add the dependency: With the current state of the code I'm fairly confident python-tuf will not accidentally start requiring cryptography in code in the future assuming securesystemslib does not do so -- so we'd just want to ensure that securesystemslib has tests that ensure it works without cryptography (it might already, I did not check). This should ensure that vendoring both (for ed25519 verificiations) is possible.

@lukpueh opinions on this?

That said, I'll point out a detail about this bit:

I am slightly concerned that this issue might be off-putting to new users and have an impact on the adoption of python-tuf in favour of e.g. go-tuf CLI

There is a reason for python-tuf being "harder to install" than a CLI app: python-tuf is not an application but a library. We don't expect end-users to ever install it, we expect:

1. applications using python-tuf to manage the dependencies for their users (See e.g. tuf-on-ci-signer: if you install that from pypi you also get securesystemslib[gcpkms,hsm,sigstore]). The application is the only one who knows what is going to be needed, python-tuf can never do that 100%
2. developers who manually install python-tuf to also manually choose the dependencies they actually will need

Now, developer experience does matter even for libraries and it is true that e.g. any tutorial would have to start with "first remember to install securesystemslib[cryptography]"... This is not ideal so I think your idea is good.

[lukpueh]

I guess it's ok. If someone wants to vendor tuf for a pure-python environment it should be easy for them to just ignore that dependency, as long as our code supports it (securesystemslib does have tests for this).

It will become harder though to use tuf in a pure-python environment without vendoring dependencies. But I don't know if that use case exists. If it does, it should be rare enough to justify improving DX for everyone else.

If we do this, we need to update the installation docs here: https://theupdateframework.readthedocs.io/en/latest/INSTALLATION.html

[h4l0gen]

@jku @lukpueh what's the conclusion, are we still okay on adding this? with above comments, It looks that we were agreed on adding it. If yes, I would like to work.