[FEATURE] Add public API to Scene for drawing cameras frustum and trajectories.

[Mehak261124]

Description

Added two new debug visualization methods to scene.py:

• draw_debug_frustum(camera): Visualizes the frustum of any camera including sensor cameras, by reusing the existing create_camera_frustum() from utils/mesh.py. Previously this was only available for the interactive viewer camera via show_cameras=True in vis_options.

• draw_debug_trajectory(positions): Draws a trajectory as a series of connected lines by connecting a list of 3D positions. Useful for visualizing robot paths and motion history.

Both methods follow the same pattern as existing draw_debug_* methods and render as markers — visible in the interactive viewer but invisible to robot cameras (depth, segmentation, RGB renders are unaffected).

Related Issue

Resolves #1049

Motivation and Context

draw_debug_frustum reuses the existing create_camera_frustum() from utils/mesh.py which was previously only used internally for the viewer camera. This change makes frustum visualization available for any camera including sensor cameras, without duplicating code.

draw_debug_trajectory was missing entirely from the debug visualization API and is a commonly needed tool for visualizing robot paths during development and debugging.

How Has This Been / Can This Be Tested?

Tested manually on macOS Apple Silicon (M2), Python 3.11.

Test draw_debug_frustum:

import genesis as gs

gs.init(backend=gs.cpu)
scene = gs.Scene(show_viewer=True)
scene.add_entity(gs.morphs.Plane())
cam = scene.add_camera(res=(640, 480), pos=(3.5, 0.0, 2.5), lookat=(0, 0, 0.5), fov=30, GUI=False)
scene.build()
scene.draw_debug_frustum(cam, color=(0.0, 1.0, 0.0, 0.3))
while scene.viewer.is_alive():
    scene.step()

Test draw_debug_trajectory:

import genesis as gs
import numpy as np

gs.init(backend=gs.cpu)
scene = gs.Scene(show_viewer=True)
scene.add_entity(gs.morphs.Plane())
scene.build()
t = np.linspace(0, 2 * np.pi, 50)
positions = np.column_stack([np.cos(t), np.sin(t), np.ones_like(t) * 0.5])
scene.draw_debug_trajectory(positions, radius=0.005, color=(1.0, 0.5, 0.0, 1.0))
while scene.viewer.is_alive():
    scene.step()

Screenshots:

Checklist:

• I read the CONTRIBUTING document
• I followed the Submitting Code Changes section
• I tagged the title correctly
• I updated the documentation accordingly
• I tested my changes and added instructions on how to test it for reviewers

[duburcqa]

All PRs adding features or fixing bugs must come with accompanying unit tests. In this case you have no other choice than resorting on screenshot-based validation.

[Mehak261124]

Hey @duburcqa, I have addressed your feedback and added the screenshot-based unit tests for draw_debug_frustum and draw_debug_trajectory. Could you take a look?

[duburcqa]

Hey @duburcqa, I have addressed your feedback and added the screenshot-based unit tests for draw_debug_frustum and draw_debug_trajectory. Could you take a look?

Look at tests/test_render.py. The corresponding fixture is png_snapshot. One of the most relevant example is probably test_sensors_draw_debug. See snippet:

    pyrender_viewer = scene.visualizer.viewer._pyrender_viewer
    assert pyrender_viewer.is_active
    rgb_arr, *_ = pyrender_viewer.render_offscreen(
        pyrender_viewer._camera_node,
        pyrender_viewer._renderer,
        rgb=True,
        depth=False,
        seg=False,
        normal=False,
    )

    if sys.platform == "darwin":
        glinfo = pyrender_viewer.context.get_info()
        renderer = glinfo.get_renderer()
        if renderer == "Apple Software Renderer":
            pytest.xfail("Tile ground colors are altered on Apple Software Renderer.")

    assert rgb_array_to_png_bytes(rgb_arr) == png_snapshot

[duburcqa]

The procedure to quite painful unfortunately, but there is no way around if you are planning to working on rendering features for quite some time. It is based on a thin custom wrapper around pytest-snapshot to support pixel-matching validation:

• Add assertion in your unit test: assert rgb_array_to_png_bytes(rgb_arr) == png_snapshot
• Run your unit test in isolation, with distributed framework disabled, and snapshot update enabled: pytest -xsv -n 0 --snapshot-update tests/test_render.py::test_rasterizer_env_separate
• Fork our internal HuggingFace-hosted Git repository snapshots that is used to manage snapshots.
• Copy the newly created snapshots that are stored under tests/__snapshots__ in your local clone of snapshots repository, commit them and push a branch on your fork.
• Open a Pull Request on HuggingFace and ping me, on this PR so that I can review and merge the HuggingFace. Note that I will only merge once every comments in Genesis' PR is approved for merging.
• Update HUGGINGFACE_SNAPSHOT_REVISION hash that is hardcoded in tests/utils.py to update the commit being used for downloading snapshots.
• If Genesis' CI pass we are good to merge.

[duburcqa]

By the way, the title of the PRs must focus on what the end user should be aware of. Not implementation details. In this case, add scene helper methods for drawing cameras frustum and trajectories.

[github-actions]

🔴 Benchmark Regression Detected ➡️ Report

[duburcqa]

Ignore performance regression report. They are a bit unstable at the moment unfortunately.

[Mehak261124]

Hi @duburcqa, I've uploaded the snapshot to my HuggingFace fork.

Here's the commit: https://huggingface.co/datasets/Mehak261124/snapshots/commit/c947da0356cb88838242a484d8763086bf50a35c

Please review and merge into Genesis-Intelligence/snapshots when ready.
I'll update HUGGINGFACE_SNAPSHOT_REVISION in tests/utils.py once it's merged.

[duburcqa]

This snapshot is not right. The camera frustum is not visible because the camera is misplaced. It may be necessary to do at least one simulation step to avoid camera artefacts.

You should already update the PR with corresponding png_snapshot.

[Mehak261124]

Hey @duburcqa, I've addressed all your feedback:

• Fixed camera placement so the frustum is now clearly visible in the snapshot
• Added scene.step() to avoid camera artifacts
• Merged the duplicate test into a single one using the png_snapshot fixture
• Updated the snapshot on my HuggingFace fork: https://huggingface.co/datasets/Mehak261124/snapshots/tree/add-debug-visualization-snapshot
• Updated HUGGINGFACE_SNAPSHOT_REVISION in tests/utils.py

Could you please review and merge the HuggingFace snapshot PR?

[duburcqa]

By the way, do not forget to remove doc update.

[duburcqa]

There seems to be something wrong... Here is the image diff:

I'm afraid it would take some time to investigate this issue and I don't have any bandwidth for now :/

[Mehak261124]

Based on the latest CI diff it looks like the issue is due to llvmpipe rendering on Linux (the scene appears mostly black), so it's not something the blur change can fix.

Since this seems platform-specific, I’m planning to mark this test as xfail on llvmpipe (similar to the Apple Software Renderer case) so CI can pass for now.

Let me know if that works.

[duburcqa]

Based on the latest CI diff it looks like the issue is due to llvmpipe rendering on Linux (the scene appears mostly black), so it's not something the blur change can fix.

This is the image diff. Black means there is no difference.

[duburcqa]

Let me know if that works.

No this is not ok. llvmpipe is reliable on linux. I don't think it is the issue. Moreover, it is failing on production CI, which is using Nvidia RTX6000 Blackwell Datacenter Edition.

[Mehak261124]

Thanks for clarifying! I understand now - the issue is that the stored PNG snapshot doesn't match what's actually rendered on Linux/production CI. The snapshot was generated on a different platform and the visual difference is too large to be covered by blurring alone.
Could you point me to how snapshots are supposed to be generated in this repo? I'd like to regenerate them correctly on Linux so they match CI. Alternatively, if there's a way to run the snapshot update command locally or via CI, I can do that.

[duburcqa]

Could you point me to how snapshots are supposed to be generated in this repo? I'd like to regenerate them correctly on Linux so they match CI. Alternatively, if there's a way to run the snapshot update command locally or via CI, I can do that.

I don't think it will help. We are using one single snapshot for all platform. If I generate it on Linux, it will pass on Linux but fail on Mac presumably.

[duburcqa]

Could you first try to disable run_in_thread? Just to make sure it is a pure rendering issue?

[Mehak261124]

Hey @duburcqa, I've pushed the change with run_in_thread=False.

[duburcqa]

working on this issue. it is due to a bug in Genesis.

[duburcqa]

Fixed! The CI should pass now \o/

[duburcqa]

Updated the unit test. Hopefully it is finally ready now.

[duburcqa]

Ok it is still failing... I will have a look tomorrow.

[duburcqa]

Found yet another bug... I will fix it.

[duburcqa]

It should pass this time!

[Mehak261124]

Thankyou so much @duburcqa !