[Docs] (WIP) Add a new page to help for first time contributors

[v01dXYZ]

Issue: No issue

Brief Summary

My first contribution was a little bit harder than expected. So I wrote down some pieces of advice for new contributors.

It is a WIP and you're welcome to edit it!

[hughperkins]

I don't agree with building LLVM oneself. It is very heavy and difficult. I never have to do this.

Please can you look at how our CI runs the build, and identify what is different between your current build process and the CI build process?

[hughperkins]

(meaning how Quadrants runs the build I mean)

[v01dXYZ]

Sorry to disagree on this but I don't think it is really hard to build LLVM as it is mainly cmake <option_to_use_ninja_or_make> -S llvm -B build -D<option_to_enable_targets or shared libs> && cmake --build build -jN. It takes some time then but it works well.

Why is it fine to document it? Sometimes, there could be some bugs in LLVM and it is great to have a version that could output the intermediate passes. or a debug version (that one is the really hard one as LLVM debug symbols are HUGE).

The CI build process uses download_llvm.py.

[hughperkins]

Sorry to disagree on this but I don't think it is really hard to build LLVM as it is mainly cmake <option_to_use_ninja_or_make> -S llvm -B build -D<option_to_enable_targets or shared libs> && cmake --build build -jN. It takes some time then but it works well.

Yes, it takes a long time to build.

Why is it fine to document it? Sometimes, there could be some bugs in LLVM and it is great to have a version that could output the intermediate passes. or a debug version (that one is the really hard one as LLVM debug symbols are HUGE).

It's documented by the CI at https://github.com/Genesis-Embodied-AI/quadrants-sdk-builds/blob/main/.github/workflows/llvm-ci.yml I feel.

I'm fine with linking to this in some "advanced, here be dragons" section. But not as a 'this is the standard build process' doc.

The CI build process uses download_llvm.py.

Yes. It should just download and work. I'd like a diagnosis into why this doesn't work for you first please, before considering any documentation beyond an 'advanced, here be dragons' type section, please.

[v01dXYZ]

:D I didn't say it doesn't work. I went straight to that as I wanted to understand why Instruction Selection failed.

But yes, it is for advanced users that want to find out why LLVM panics or if some passes could be better implemented.

Thinking again you're right, let's cut it off.

[hughperkins]

Oh, I see, you are covering multiple approaches:

• OS install of LLVM
• download_llvm.py
• build oneself

I actually missed that you were covering two other approaches, on initial skimming, to be honest.

Note that the build process as far as llvm and clang is a little bit confused currently, and could probably be cleaned up a little. We actually use three somewhat orthogonal things:

• some C/C++ compiler, e.g. 'msvc': for building C++ files
• clang/clang++ compiler: used to build the LLVM runtime bitcode
• LLVM library: linked with, at build time

The C/C++ compiler is pretty much always using a platform-native compiler, e.g. gcc, or msvc, currently. Which seems reasonable I feel. This is defined here

quadrants/.github/workflows/scripts/ti_build/entry.py

Line 43 in a346d2d

setup_clang(as_compiler=False)

The clang/clang++ compiler currently uses an OS-installed clang/clang++ on all platforms, I believe though I was pushing to use the downloaded clang/clang++ for a while. I saw advantages/disadvantages as:

good points of using packaged/downloaded clang/clang++:

• same version of clang/clang++ everywhere

downsides of using packaged/downloaded clang/clang++:

• inflates the size of the llvm library download (and the llvm library build)
• hard to get it to work with permissions on Mac

Lastly we have the LLVM libraries themselves, which are always downloaded, by default. They are linked with, when building Quadrants. This is the only bit which must be a specific version, and that is tightly coupled with the code, AFAIK.

Anyway... I'm not averse to having a section on building clang and/or llvm yourself, if you feel this is useful, but should clearly be marked as 'advanced here be dragons' (or similar) I feel.

It would probably be worth documenting clearly the difference between the three things above, somehow, somewhere?

[v01dXYZ]

My only intent is to debug the LLVM code that is called by the library. But what you said is worth mentioning about what is compiled.

To reformulate, quadrants being a compiler for running kernels, there are at least three pieces:

• quadrants cpp code: compiled with OS compiler
• LLVM used by quadrants: provided by download_llvm.py
• runtime bitcode (code that will be loaded in GPU memory to setup the kernel): compiled with clang as the compiler for that needs to target CUDA/AMDGPU and gcc/mvcc does not support them.

[hughperkins]

Are you sure this is needed? I never set this myself.

I simply do:

./build.py --shell
python setup.py develop

[v01dXYZ]

What does --shell? Does it activate a venv with all the deps?

[hughperkins]

it sets a bunch of env vars

there's also a version for AI. I don't pay close attention to what AI does, but it seems to be doing somethign like:

./build.py -w env.sh
source env.sh
python setup.py develop

[hughperkins]

Lets keep just the second sentence, to keep as concise as posible "There are three pieces that need to be compiled or are themselves part
of a compiler project.". Actually we can shorten further: "Quadrants comprises three parts:".

But actually, what about the python code that is part of Quadrants? One can argue it is included in the 'host runtime' part. But on the other hand, below it says that the 'host runtime' is compiled, usng a C/C++ compiler. Thoughts?

[hughperkins]

(I am wary of saying that Quadrants 'comprises 3 parts' to be honest. There are also .pyi stub files from pybind11, for example. Thoughts?)

[hughperkins]

Just to check, you are still addressing the comments right? Not expecting anything from me on this currently I think?

[v01dXYZ]

Thanks for asking. Yes, I addressed your review. You can take a look if you want to point to place where the style could be improved (maybe I should make it more formal).

I also opened an issue about ditch ./setup.py invocation with pip/uv. It is low priority as it currently works but should serve as a reminder for later.

[hughperkins]

there is also the AI way, where it outputs a script to set the env vars.

[hughperkins]

this seems not to have been addressed yet?

[v01dXYZ]

No, I've got to take a look at it before answering something relevant.

[hughperkins]

do you mean, when we run ./build.py --shell?

[v01dXYZ]

it is appended by setup.py before invoking cmake.

To me, it doesn't seem related to build.py --shell.

[hughperkins]

I've never run cmake directly to be honest. I've always done ./build.py --shell. I have no objection to making the build more and more standard, but I would like some explanation please of:

• why are you not simply using ./build.py --shell and python setup.py develop?
• how does calling cmake direclty relate to the above?

[v01dXYZ]

1. I didn't know build.py existed and I used at first pip as it is the way. But now I use build.py.
2. I don't call cmake directly as I have then to copy manually the shared object. But using the cmake options is required to disable some unwanted backends. It's not necessary though.

[v01dXYZ]

There is an env var that is read by setup.py to add cmake CLI args.

[hughperkins]

ther'es an env var that is read by build.py too I believe.

[v01dXYZ]

Yes, it is called CMakeArgsManager.

[hughperkins]

They're used by a lot more than final lowering. They're used throughout the entire c++ side, that receives the python side AST, via pybind11 api calls, transforms them to CHIIR, optimizes them, then gradually lowers them to LLVM IR.

[v01dXYZ]

I'll take a look at that to be more accurate.

[hughperkins]

This hasnt been addressed yet right?

[v01dXYZ]

Not yet.

[v01dXYZ]

sorry, but I don't understand what you said. You talk about a Chi-IR which is an intermediate representation. Could you talk a bit more about this, b/c there is no mention of this in the source code?

Once it is lowered into LLVM IR, do you have custom passes or do you rely only on the ones provided by LLVM?

[hughperkins]

custom passes.

Please consider asking Opus 4.6 to walk you trhough how the passes work, what passes there are, etc.

Personally I like Cursor CLI interface https://cursor.com/docs/cli/overview

Most people prefer Claude Code for reasons unclear to me.

[v01dXYZ]

Sorry to state it as plainly but I don't have the money to use Opus 4.6.

I grep-ed the code base to look at LLVM pass and I hardly found ones. I'm talking about LLVM passes not the ones related to higher IRs.

[hughperkins]

Ah, yes, I believe our custom passes are for our custom IR, right.

[v01dXYZ]

yes, indeed, it's in transforms.

[hughperkins]

Just to check, have you addressed all comments, and would like a fresh review?

Alternatively, once you believe you have addressed all comments, and would like a fresh review, please add a comment stating that.

[v01dXYZ]

I was trying to understand the PR about f32 atomics support with Vulkan/Metal, so I left this PR aside a little bit. I plan to take a look before the end of the day.