Major Merge of Linfeng's project on "Seeing is Believing"

[lf-zhao]

Change logs

This PR includes the major work for RSS 2025 submission "Seeing is Believing: Belief-Space Planning with Foundation Models as Uncertainty Estimators".

• Add belief-space VLM-based predicates and operators
• Add new VLM-based belief state update perceiver for Spot
• Add synthetic/mock Spot robot environment (real image based synthetic environment, "RISE")
• Add lots of visualization and tests for the synthetic tasks
• Include a few real-image synthetic tasks for experiments
• Update VLM/LLM-based planners and lots of interface
• Include running scripts for experiments
• Many other changes

Note for BKLVA Approach

BKLVA (Belief-space Planning with K-fluents and Language-based Goal Grounding) is an approach for integrated perception and belief-space planning using large vision-language models (VLMs) as state estimation modules. It extends task planning and perception pipeline to a symbolic belief space using belief-space predicates and operators.

Key Features

• VLM Perception: Uses vision-language models to evaluate visual predicates from images
• Task Planner: PDDL-based symbolic planner that generates plans in belief space
• Belief Space Planning: Support for tasks involving uncertainty about object properties
• Execution Monitoring: Detects unexpected outcomes and triggers replanning when needed

Quick Start

Running Experiments

1. Directly run:

# Example on two-cup pick-place synthetic task
python predicators/main.py --env mock_spot_pick_place_two_cup \
  --approach oracle --seed 0 --perceiver mock_spot_perceiver \
  --mock_env_vlm_eval_predicate True --num_train_tasks 0 \
  --num_test_tasks 1 --bilevel_plan_without_sim True \
  --horizon 20

3. Using mock_experiments.py:

# Run all planners on a single environment
python scripts/mock_experiments.py --env mock_spot_drawer_cleaning

# Run specific planner
python scripts/mock_experiments.py --env mock_spot_drawer_cleaning --planner vlm_closed_loop

4.. Using run_local_experiments.sh (recommended for systematic evaluation):

# Run experiments with multiple seeds
./predicators_deploy/run_local_experiments.sh 5 mock_spot_drawer_cleaning mock_spot_sort_weight

Available Planners

1. BKLVA (Oracle): Basic bilevel planning with VLM perception
2. BKLVA with Execution Monitoring: Oracle with replanning on unexpected outcomes
3. BKLVA Open Loop: Oracle without replanning
4. LLM Closed Loop: LLM-based planning with execution monitoring
5. VLM Closed Loop: VLM-based planning with execution monitoring
6. VLM Captioning: Scene captioning for state estimation
7. VLM Captioning Open Loop: Scene captioning without replanning

Available Environments

1. Pick and Place Tasks:

   • mock_spot_pick_place: Simple pick and place
   • mock_spot_pick_place_two_cup: Two-cup manipulation

2. Belief-Space Tasks:

   • mock_spot_cup_emptiness: Cup content detection
   • mock_spot_drawer_cleaning: Drawer manipulation
   • mock_spot_sort_weight: Weight-based sorting

Environment Setup

export PYTHONPATH="${PYTHONPATH:-$PWD}"
export PYTHONUNBUFFERED=1
export PYTHONHASHSEED=0

Results Organization

• results_deploy/<timestamp>_<env>_<planner>/: Experiment results
• runlogs/run_<env>_planner_<planner>_seed_<N>.txt: Detailed logs

Documentation

For detailed documentation, see:

• BKLVA Approach Documentation
• Planning with Mock Spot Environment
• Creating New Mock Robot Tasks
• Mock Spot Perceiver