(lib) Add support for vLLM-backed text embedder

[charlesbluca]

Description

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• If adjusting docker-compose.yaml environment variables have you ensured those are mimicked in the Helm values.yaml file.

[greptile-apps]

Greptile Summary

This PR adds vLLM-backed local text embedding as an alternative to the existing HuggingFace path, introducing LlamaNemotronEmbed1BV2Embedder (vLLM), LlamaNemotronEmbed1BV2HFEmbedder (lazy-loaded HF), and LlamaNemotronEmbedVL1BV2VLLMEmbedder (vLLM VL), along with factory routing in create_local_embedder and create_local_query_embedder.

• P1 — processor.py HF backend crash: maybe_inject_local_hf_embedder builds a _embed closure that passes prefix=\"\" to embedder_instance.embed(), but LlamaNemotronEmbed1BV2HFEmbedder.embed() has no prefix parameter — any user setting local_ingest_backend=\"hf\" hits an immediate TypeError.
• P1 — gpu_operator.py device silently dropped (noted in previous outside-diff comment, still present): create_local_embedder call omits the device kwarg, so user-configured device is ignored for the HF backend.

Confidence Score: 3/5

Not safe to merge as-is: the HF ingest backend crashes with a TypeError, and multiple lifecycle and device-placement issues remain unresolved.

Two P1 defects are present: processor.py crashes when local_ingest_backend="hf" due to an unexpected prefix="" kwarg, and gpu_operator.py silently drops the user-configured device for the HF path. Multiple P2 issues (eager loading in the HF VL embedder, missing unload methods) remain from earlier review rounds. Score is pulled below the P1 ceiling of 4 because the processor crash affects a documented configuration path.

text_embed/processor.py (HF backend crash), text_embed/gpu_operator.py (missing device forwarding), model/local/llama_nemotron_embed_vl_1b_v2_embedder.py (eager HF model load).

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/text_embed/vllm.py	New module for vLLM batched embedding inference — clean, well-structured, with proper batching, empty-slot handling, and multimodal support.
nemo_retriever/src/nemo_retriever/text_embed/processor.py	Injects a local embedder into task_config; passes prefix="" kwarg to LlamaNemotronEmbed1BV2HFEmbedder.embed() which doesn't accept it — crashes the HF ingest backend.
nemo_retriever/src/nemo_retriever/model/init.py	Factory functions updated to route VL and non-VL models to vLLM or HF backends; create_local_query_embedder correctly defaults to "hf" for queries.
nemo_retriever/src/nemo_retriever/model/local/llama_nemotron_embed_1b_v2_embedder.py	vLLM-backed embedder with _finalize_vectors padding; still loads eagerly in __post_init__ (known issue from prior review rounds).
nemo_retriever/src/nemo_retriever/model/local/llama_nemotron_embed_1b_v2_hf_embedder.py	New HF-only embedder with correct lazy loading via _ensure_loaded(); prefix logic handled internally without a prefix kwarg.
nemo_retriever/src/nemo_retriever/model/local/llama_nemotron_embed_vl_1b_v2_embedder.py	Adds LlamaNemotronEmbedVL1BV2VLLMEmbedder; existing HF VL embedder still loads model eagerly in __post_init__, violating model-lazy-loading.
nemo_retriever/src/nemo_retriever/text_embed/gpu_operator.py	Refactored to use create_local_embedder; device kwarg is not forwarded, silently ignoring user-configured device for the HF backend.
nemo_retriever/tests/test_vllm_embed.py	Thorough unit tests for embed_with_vllm_llm, embed_multimodal_with_vllm_llm, and the VL vLLM embedder; covers batching, None embedding slots, and L2 normalization.
nemo_retriever/tests/test_create_local_embedder.py	Factory routing tests cover all four embedder classes including the new VL vLLM class; fixture correctly stubs both LlamaNemotronEmbedVL1BV2Embedder and LlamaNemotronEmbedVL1BV2VLLMEmbedder.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[embed request] --> B{has endpoint?}
    B -- yes --> C[Remote NIM HTTP]
    B -- no --> D[maybe_inject_local_hf_embedder]
    D --> E{local_ingest_backend}
    E -- vllm --> F[create_local_embedder backend=vllm]
    E -- hf --> G[create_local_embedder backend=hf]
    F --> H{is_vl_embed_model?}
    G --> I{is_vl_embed_model?}
    H -- yes --> J[LlamaNemotronEmbedVL1BV2VLLMEmbedder]
    H -- no --> K[LlamaNemotronEmbed1BV2Embedder]
    I -- yes --> L[LlamaNemotronEmbedVL1BV2Embedder]
    I -- no --> M[LlamaNemotronEmbed1BV2HFEmbedder]
    K --> N["_embed closure: embed(prefixed, prefix='')"]
    M --> O["_embed closure: embed(prefixed, prefix='') ❌ TypeError"]
    J --> P["_embed closure: embed(texts) ✓"]
    L --> Q["_embed closure: embed(texts) ✓"]

Loading

Prompt To Fix All With AI

This is a comment left during a code review.
Path: nemo_retriever/src/nemo_retriever/text_embed/processor.py
Line: 124-129

Comment:
**`prefix=""` kwarg crashes HF non-VL ingest backend**

When `local_ingest_backend="hf"`, `create_local_embedder` returns a `LlamaNemotronEmbed1BV2HFEmbedder`. The `_embed` closure then calls `embedder_instance.embed(prefixed, batch_size=local_batch_size, prefix="")`, but `LlamaNemotronEmbed1BV2HFEmbedder.embed()` is declared as `def embed(self, texts, *, batch_size=64)` with no `prefix` parameter. Python raises `TypeError: embed() got an unexpected keyword argument 'prefix'` immediately, crashing the ingest pipeline for any user who sets `local_ingest_backend="hf"`.

The `LlamaNemotronEmbed1BV2HFEmbedder.embed()` already prepends the `"passage: "` prefix internally, so the texts should be passed unprefixed and without `prefix=`:

```python
def _embed(texts):
    if ingest_backend == "hf":
        # HF embedder manages its own prefix internally
        vecs = embedder_instance.embed(texts, batch_size=local_batch_size)
    else:
        prefixed = [prefix + t for t in texts] if prefix else texts
        vecs = embedder_instance.embed(prefixed, batch_size=local_batch_size, prefix="")
    return vecs.tolist()
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: nemo_retriever/src/nemo_retriever/model/local/llama_nemotron_embed_vl_1b_v2_embedder.py
Line: 51-86

Comment:
**Eager model load in `__post_init__` violates `model-lazy-loading` rule**

`LlamaNemotronEmbedVL1BV2Embedder.__post_init__` calls `AutoModel.from_pretrained()` (plus a `.to(dev)` CUDA transfer) unconditionally at construction time. The `model-lazy-loading` rule requires models to load on first use, not at construction. Any code path that constructs this class — e.g., `create_local_embedder(..., backend="hf")` for a VL model inside `gpu_operator.py` — will immediately trigger a full HF download and GPU transfer before any embedding call is made.

Apply the same `_ensure_loaded()` pattern used by `LlamaNemotronEmbed1BV2HFEmbedder`: set `self._model = None` in `__post_init__`, and defer the `AutoModel.from_pretrained` call to a private `_ensure_loaded()` helper invoked in `embed()`, `embed_queries()`, `embed_images()`, and `embed_text_image()`.

**Rule Used:** Models must be loaded lazily (on first use or expl... ([source](https://app.greptile.com/review/custom-context?memory=model-lazy-loading))

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (38): Last reviewed commit: "Merge branch 'main' into retriever-vllm-..." | Re-trigger Greptile}

[greptile-apps]

prefix="" kwarg crashes HF non-VL ingest backend

When local_ingest_backend="hf", create_local_embedder returns a LlamaNemotronEmbed1BV2HFEmbedder. The _embed closure then calls embedder_instance.embed(prefixed, batch_size=local_batch_size, prefix=""), but LlamaNemotronEmbed1BV2HFEmbedder.embed() is declared as def embed(self, texts, *, batch_size=64) with no prefix parameter. Python raises TypeError: embed() got an unexpected keyword argument 'prefix' immediately, crashing the ingest pipeline for any user who sets local_ingest_backend="hf".

The LlamaNemotronEmbed1BV2HFEmbedder.embed() already prepends the "passage: " prefix internally, so the texts should be passed unprefixed and without prefix=:

def _embed(texts):
    if ingest_backend == "hf":
        # HF embedder manages its own prefix internally
        vecs = embedder_instance.embed(texts, batch_size=local_batch_size)
    else:
        prefixed = [prefix + t for t in texts] if prefix else texts
        vecs = embedder_instance.embed(prefixed, batch_size=local_batch_size, prefix="")
    return vecs.tolist()

Prompt To Fix With AI

This is a comment left during a code review.
Path: nemo_retriever/src/nemo_retriever/text_embed/processor.py
Line: 124-129

Comment:
**`prefix=""` kwarg crashes HF non-VL ingest backend**

When `local_ingest_backend="hf"`, `create_local_embedder` returns a `LlamaNemotronEmbed1BV2HFEmbedder`. The `_embed` closure then calls `embedder_instance.embed(prefixed, batch_size=local_batch_size, prefix="")`, but `LlamaNemotronEmbed1BV2HFEmbedder.embed()` is declared as `def embed(self, texts, *, batch_size=64)` with no `prefix` parameter. Python raises `TypeError: embed() got an unexpected keyword argument 'prefix'` immediately, crashing the ingest pipeline for any user who sets `local_ingest_backend="hf"`.

The `LlamaNemotronEmbed1BV2HFEmbedder.embed()` already prepends the `"passage: "` prefix internally, so the texts should be passed unprefixed and without `prefix=`:

```python
def _embed(texts):
    if ingest_backend == "hf":
        # HF embedder manages its own prefix internally
        vecs = embedder_instance.embed(texts, batch_size=local_batch_size)
    else:
        prefixed = [prefix + t for t in texts] if prefix else texts
        vecs = embedder_instance.embed(prefixed, batch_size=local_batch_size, prefix="")
    return vecs.tolist()
```

How can I resolve this? If you propose a fix, please make it concise.