feat:Add WebRTC and real-time multimedia schemas to OpenAPI

[HavenDV]

Summary by CodeRabbit

• New Features
  • Added real-time conversation capabilities with live audio and video streaming support
  • Introduced event-driven architecture for continuous monitoring and updates during multimedia conversations
  • Implemented comprehensive configuration options for voice settings, audio/video processing, and interactive session management with real-time transcript handling

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2781eb27-7d86-46ac-88dd-06b74adb5287

📥 Commits

Reviewing files that changed from the base of the PR and between 5d6b034 and fbd87ea.

⛔ Files ignored due to path filters (5)

• src/libs/tryAGI.OpenAI/Generated/tryAGI.OpenAI.BatchClient.CreateBatch.g.cs is excluded by !**/generated/**
• src/libs/tryAGI.OpenAI/Generated/tryAGI.OpenAI.IBatchClient.CreateBatch.g.cs is excluded by !**/generated/**
• src/libs/tryAGI.OpenAI/Generated/tryAGI.OpenAI.Models.CreateBatchRequest.g.cs is excluded by !**/generated/**
• src/libs/tryAGI.OpenAI/Generated/tryAGI.OpenAI.Models.CreateBatchRequestEndpoint.g.cs is excluded by !**/generated/**
• src/libs/tryAGI.OpenAI/Generated/tryAGI.OpenAI.Models.FunctionToolParam.g.cs is excluded by !**/generated/**

📒 Files selected for processing (1)

• src/libs/tryAGI.OpenAI/openapi.yaml

---

Walkthrough

This PR extends the OpenAPI specification to support real-time multimedia conversation capabilities, adding comprehensive schemas for WebRTC sessions, event-driven streaming, audio/video processing, and bidirectional client-server interactions.

Changes

Cohort / File(s)

Summary

Real-time API Specification src/libs/tryAGI.OpenAI/openapi.yaml

Added extensive schema definitions for real-time multimedia conversations including WebRTC session management (RealtimeSession, RealtimeSessionCreateRequest), event-driven architecture with server-sent events (ResponseCreatedEvent, ResponseCompletedEvent, etc.), streaming content deltas for audio/text/transcripts, client-initiated events, and configuration schemas for voice, audio formats, video processing, and turn detection (VAD). Also added schemas for conversation items, input/output content parts, and error handling.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Server
    participant AudioProcessor
    participant Transcription
    
    Client->>Server: RealtimeClientEvent (audio input)
    activate Server
    Server->>AudioProcessor: Process audio stream
    activate AudioProcessor
    AudioProcessor->>AudioProcessor: Generate audio delta
    AudioProcessor-->>Server: ResponseAudioDelta
    deactivate AudioProcessor
    
    Server->>Transcription: Transcribe audio
    activate Transcription
    Transcription-->>Server: InputAudioTranscriptionDelta
    deactivate Transcription
    
    Server->>Server: Generate response
    Server->>Server: Create ResponseTextDelta
    Server-->>Client: ResponseTextDeltaEvent
    Server-->>Client: ResponseTextDoneEvent
    
    Server-->>Client: ResponseCompletedEvent
    deactivate Server

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat:Refactor OpenAPI schema for streaming and multi-modal support #254: Modifies the same OpenAPI schema for streaming and multi-modal response events like ResponseStreamEvent, audio/text deltas, and completion schemas.
• feat:Expand OpenAI OpenAPI schema with streaming and multi-modal support #253: Adds extensive streaming, multi-modal, and real-time event schemas to the same OpenAPI spec, including ResponseTextDelta, ResponseAudioDelta, and RealtimeSession definitions.

Poem

🐰 Hop, hop, huzzah! Real-time talks now flow,
Audio streams dance, transcripts glow,
WebRTC whispers, events take flight,
Video visions and schemas so bright,
Multimedia magic—what a delight! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title Check	✅ Passed	Title check skipped as CodeRabbit has written the PR title.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch bot/update-openapi_202603140427

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}