Skip to content

ROADMAP.md

Latest commit

 

History

History
137 lines (125 loc) · 9.97 KB

ROADMAP.md

File metadata and controls

137 lines (125 loc) · 9.97 KB

Devloop Project Roadmap

This document outlines the high-level roadmap for the devloop project.

Phase 1: Core Functionality (Complete)

  • Configuration Loading: Define and parse .devloop.yaml files, resolving relative paths.
  • File Watching: Monitor file systems for changes using fsnotify.
  • Glob Matching: Match file changes against user-defined glob patterns (both relative and absolute).
    • Migrated to doublestar library for conventional ** glob behavior
    • Pattern src/**/*.go now matches both src/main.go and src/pkg/utils.go
  • Command Execution: Run shell commands in response to file changes.
  • Process Management: Gracefully terminate and restart long-running processes.
  • Debouncing: Prevent command storms by debouncing file change events.
  • CLI Arguments: Basic command-line interface for specifying configuration files.
  • Graceful Shutdown: Handle SIGINT and SIGTERM to shut down cleanly.
  • gRPC/HTTP API: Implement a unified API layer using gRPC and a gRPC-Gateway for monitoring and interaction.
  • Test Suite: Comprehensive test coverage with all tests passing.
  • Code Organization: Restructured into agent/, gateway/, and utils/ directories.

Phase 2: Multi-Instance Architecture & Usability (Complete)

  • Event-Driven LRO Architecture (2025-08-20):
    • Dual-Execution System: Implemented separate execution paths for long-running (LRO) vs short-running operations
    • Scheduler Component: Created event-driven routing system using TriggerEvents for clean separation of concerns
    • LROManager: Dedicated process lifecycle manager with proper process replacement and graceful termination
    • WorkerPool Integration: Connected existing WorkerPool for semaphore-controlled short-running job execution
    • Status Callback Architecture: Unified status management via RuleRunner callbacks from execution engines
    • Configuration Enhancement: Added lro: true/false field to protobuf and YAML with proper parsing
    • Comprehensive Testing: 57% test coverage with dedicated LRO and integration tests
    • Process Safety: Proper SIGTERM → SIGKILL termination with port/socket cleanup verification
  • Architecture Refactoring:
    • Separated concerns: Orchestrator (file watching) and RuleRunner (command execution)
    • Simplified to single orchestrator implementation with Agent Service integration
    • Created comprehensive testing infrastructure
    • Rule-specific configuration (debounce delay, verbose logging)
  • Process Management Improvements:
    • Fixed zombie process issues with proper signal handling
    • Implemented sequential command execution with failure propagation
    • Platform-specific process management (Linux, Darwin, Windows)
  • Testing Infrastructure:
    • Simplified testing infrastructure for single orchestrator implementation
    • Comprehensive test coverage for all core functionality
    • Cross-platform testing support
  • Port Configuration & Auto-Discovery:
    • Updated default ports to avoid common conflicts (HTTP: 8080→19080, gRPC: 50051→5555)
    • Implemented automatic port discovery with --auto-ports flag
    • Fast TCP bind/close port availability checking
    • Fallback port search with configurable range limits
  • Gateway Mode (grpcrouter-based):
    • Research and integrate grpcrouter library for automatic gateway/proxy functionality
    • Implement simplified gateway mode using grpcrouter instead of custom implementation
    • Add agent connection logic for grpcrouter-based gateway
    • Add comprehensive tests for distributed mode operations with new architecture
    • Agent Service integration completed (provides foundation for gateway)
  • Streaming Logs Overhaul (Completed 2026-03-05):
    • Design doc: docs/streaming-logs-design.md
    • Single-poller-per-rule with fan-out broadcaster (fixes finishedRules bug)
    • Configurable truncate vs append on re-run (append_on_restarts per-rule config)
    • Structured LogEvent messages in StreamLogsResponse (RUN_STARTED, RUN_COMPLETED, RUN_FAILED, TIMEOUT)
  • Disabled Rules (Issue #3, 2026-03-06):
    • Added disabled: true per-rule config field to skip rules without removing config
    • Disabled rules get no RuleRunner, no file watcher, zero resource overhead
    • devloop status shows disabled rules as "Disabled"
    • devloop trigger on a disabled rule returns a clear error
    • Pre-push git hook runs make test before pushing
  • Shell Files Env Capture & Cascade (2026-03-14):
    • Shell file env vars captured once before command loop (not re-executed per command)
    • Env vars cascade between sequential commands by default
    • reset_env config flag (global + per-rule override) to disable cascade
    • Shell files still sourced per-command for functions/aliases
    • 20 new tests (unit + integration exercising full executeNow flow)
  • Enhanced Logging:
    • Structured logging (e.g., JSON) for machine-readability.
    • Log filtering and searching capabilities via the API.
  • Improved CLI (Completed 2025-07-07):
    • Cobra Framework Integration: Complete CLI restructuring with modern subcommand architecture
    • Client Commands: devloop config, devloop status, devloop trigger, devloop paths
    • Server Command: devloop server with all original functionality
    • Default Behavior: devloop starts server, subcommands act as gRPC client
    • Signal Handling: Fixed Ctrl-C interrupt handling for proper server shutdown
    • Code Organization: Modular structure with cmd/, server/, client/ packages
    • Professional UX: Help system, usage examples, and consistent flag handling
    • Project Initialization (Completed 2025-08-07): Added devloop init command with embedded profile templates
      • Profile-Based Bootstrapping: Pre-configured templates for Go, TypeScript, and Python Flask projects
      • Multiple Profile Support: Single command multi-service setup (devloop init go ts py)
      • Extensible Architecture: Easy addition of new profiles via cmd/profiles/ YAML files
  • Cycle Detection & Prevention (Completed 2025-07-08):
    • Static Validation: Startup detection of self-referential patterns in rule configurations
    • Dynamic Rate Limiting: TriggerTracker with frequency monitoring and exponential backoff
    • Cross-Rule Cycle Detection: Trigger chain tracking with configurable max depth limits
    • File Thrashing Detection: Sliding window frequency analysis for rapid file modifications
    • Emergency Cycle Breaking: Rule disabling and cycle resolution suggestions
    • Configuration Parser Fix: Fixed YAML-to-protobuf parsing for cycle_detection settings
    • Comprehensive Configuration: Full cycle_detection settings block with all parameters
  • Startup Resilience & Retry Logic (Completed 2025-07-16):
    • Exponential Backoff Retries: Configurable retry logic for failed rule startup with exponential backoff
    • Graceful Failure Handling: Rules fail independently without stopping devloop unless explicitly configured
    • Configurable Exit Behavior: exit_on_failed_init flag for critical rules that must succeed
    • Retry Configuration: max_init_retries and init_retry_backoff_base for fine-tuned retry behavior
    • Comprehensive Logging: Detailed retry attempt logging with next retry time and success notifications
    • Backward Compatibility: Default behavior allows devloop to continue running despite startup failures
    • Non-Blocking Implementation: Fixed blocking issue by moving retry logic to background execution
  • Web-based UI:
    • A simple web interface, built on the Agent Service HTTP API, to visualize rule status, logs, and trigger commands manually.

Phase 3: Ecosystem & Integration (Future)

  • Enhanced MCP Integration:

    • Add streaming log support to MCP tools for real-time monitoring
    • Implement additional MCP resources for project files and configurations
    • Explore MCP prompts for common development workflows

  • IDE Integration:

    • Plugins for popular IDEs (e.g., VS Code, GoLand) to provide a seamless development experience.

  • Cloud-Native Development:

    • Integration with tools like Docker and Kubernetes for containerized development workflows.

  • Performance Optimization:

    • Optimize file watching and command execution for large projects.

  • Community & Documentation:

    • Build a community around devloop and create comprehensive documentation and tutorials for the new architecture.

Phase 2.5: LRO Architecture (Completed 2025-08-20)

  • ✅ Event-Driven LRO Architecture Implementation:
    • ✅ Dual-Execution System: Implemented separate execution paths for long-running (LRO) vs short-running operations
    • ✅ Scheduler Component: Created event-driven routing using TriggerEvents for clean separation of concerns
    • ✅ LROManager: Dedicated process lifecycle manager with proper process replacement and graceful termination
    • ✅ WorkerPool Integration: Connected existing WorkerPool for semaphore-controlled short-running job execution
    • ✅ Status Callback Architecture: Unified status management via RuleRunner callbacks from execution engines
    • ✅ Configuration Support: Added lro: true/false field to protobuf and YAML with proper parsing
    • ✅ Comprehensive Testing: 57% test coverage with LRO lifecycle and integration tests
    • ✅ Process Safety: Proper SIGTERM → SIGKILL termination with port/socket cleanup verification
    • ✅ Developer Experience: Added Makefile coverage targets with HTML output in organized reports/ folder
    • ✅ Backward Compatibility: Default lro: false maintains existing behavior for all rules