Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 35 additions & 5 deletions .github/workflows/build.yml
Comment thread
wdconinc marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,28 @@ jobs:
os: [ubuntu-latest] # TODO re-enable: , macos-latest]
python-version: ['3.10', '3.11', '3.12', '3.13']

env:
PGHOST: localhost
PGPORT: 5432
PGDATABASE: test
PGUSER: test
PGPASSWORD: test

services:
postgres:
image: postgres:16
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

steps:
- name: Checkout repository
uses: actions/checkout@v5
Expand Down Expand Up @@ -62,7 +84,7 @@ jobs:
rm apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb

sudo apt-get update
sudo apt-get install -y --no-install-recommends libpq-dev libarrow-dev liblz4-dev libre2-dev
sudo apt-get install -y --no-install-recommends libpq-dev libarrow-dev liblz4-dev libre2-dev pybind11-dev

# Set GCC 14 as the default compiler
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100
Expand Down Expand Up @@ -113,15 +135,23 @@ jobs:
# Install testing tools
pip install pytest wheel

- name: Build and run C++ tests
run: |
mkdir -p build
cd build
cmake .. -DBUILD_TESTING=ON
cmake --build . --target test_database
ctest --output-on-failure -V
env:
CC: ${{ runner.os == 'Linux' && 'gcc-14' || 'gcc' }}
CXX: ${{ runner.os == 'Linux' && 'g++-14' || 'g++' }}
- name: Build and install Python package
run: |
pip install . --verbose
env:
CC: ${{ runner.os == 'Linux' && 'gcc-14' || 'gcc' }}
CXX: ${{ runner.os == 'Linux' && 'g++-14' || 'g++' }}

- name: Run tests
- name: Run Python tests
run: |
# This is a placeholder. Create a 'tests' directory with pytest files.
# For example: pytest tests/
echo "Tests would run here."
pytest tests/python/ -v
83 changes: 76 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,20 +77,52 @@ cmake ..
make
```

### 5.2. Git Submodules
**Building with Tests:**
```bash
mkdir build && cd build
cmake .. -DBUILD_TESTING=ON
cmake --build .
ctest --output-on-failure
```

### 5.2. Testing

The project includes comprehensive test suites for both C++ and Python.

**Test Frameworks:**
- **C++ Tests**: Google Test (gtest) v1.15.2, integrated via CMake FetchContent
- **Python Tests**: pytest, already listed in `pyproject.toml` test dependencies

**Running Tests:**
```bash
# Python tests
pytest tests/python/ -v

# C++ tests
cd build
ctest --output-on-failure
```

**Test Coverage:**
- C++ tests validate core Database class functionality
- Python tests verify bindings and module integration
- Integration tests ensure C++/Python data interchange works correctly
- All tests run automatically in CI on every PR

### 5.3. Git Submodules
The project uses git submodules for dependencies like sqlpp23. Always ensure submodules are initialized:
```bash
git submodule update --init --recursive
```

### 5.3. Code Style
### 5.4. Code Style
- Follow the existing code style in each file
- C++ code uses C++23 standard (as specified in CMakeLists.txt)
- Python code should follow PEP 8 guidelines
- Keep the C++ core focused on database operations
- Keep Python bindings thin and delegate to C++ core

### 5.4. Data Flow
### 5.5. Data Flow
1. C++ functions query the database using sqlpp23
2. Results are converted to Apache Arrow `Table` objects
3. Arrow tables are passed to Python with zero-copy
Expand All @@ -110,10 +142,12 @@ git submodule update --init --recursive
- Apache Arrow C++ library
- PostgreSQL client library (libpq)
- pybind11
- Google Test (for testing, fetched automatically via CMake)

### 6.2. Python Dependencies
- scikit-build-core (build)
- pybind11 (build)
- pytest (testing)
- pyarrow (runtime - for Arrow table interaction)
- pandas (optional - for DataFrame conversion in user code and examples)

Expand Down Expand Up @@ -168,9 +202,11 @@ The documentation is automatically deployed to GitHub Pages via the `.github/wor
1. Implement the C++ function in `src/Database.cpp`
2. Use sqlpp23 for database interaction
3. Return results as Arrow `Table` objects
4. Expose the function in `python/bindings.cpp` using pybind11
5. Update Python package `__init__.py` if needed
6. Update documentation in `docs/README.md` with usage examples
4. Add C++ tests in `tests/cpp/test_database.cpp`
5. Expose the function in `python/bindings.cpp` using pybind11
6. Add Python tests in `tests/python/test_database.py`
7. Update Python package `__init__.py` if needed
8. Update documentation in `docs/README.md` with usage examples

### 9.2. Modifying Build Configuration
- C++ build: Edit `CMakeLists.txt`
Expand All @@ -182,10 +218,43 @@ The documentation is automatically deployed to GitHub Pages via the `.github/wor
- Pass table pointers through pybind11
- Python receives Arrow tables that can be converted to Pandas DataFrames

### 9.4. Running Tests

The project includes comprehensive test suites for both C++ and Python components.

**Python Tests (pytest):**
```bash
pytest tests/python/ -v
```

**C++ Tests (Google Test via CTest):**
```bash
mkdir build && cd build
cmake .. -DBUILD_TESTING=ON
cmake --build .
ctest --output-on-failure
```

**Test Structure:**
- `tests/cpp/` - C++ tests using Google Test
- `test_database.cpp` - Database class tests
- `tests/python/` - Python tests using pytest
- `test_basic.py` - Import and initialization tests
- `test_database.py` - Database class tests
- `test_integration.py` - Integration tests
- `conftest.py` - pytest fixtures and configuration

**Writing Tests:**
When adding new features or fixing bugs:
1. Add C++ tests for core functionality in `tests/cpp/`
2. Add Python tests for bindings and integration in `tests/python/`
3. Ensure tests pass locally before submitting PR
4. CI will automatically run all tests on pull requests

## 10. Important Notes

- **Minimal Changes**: Make the smallest possible changes to achieve goals
- **Testing**: Run existing tests before and after changes
- **Testing**: Run existing tests before and after changes. Add tests for new functionality
- **Dependencies**: Avoid adding new dependencies unless absolutely necessary
- **Platform Independence**: Keep code portable (Linux, macOS, Windows)
- **Performance**: The C++ core is designed for high performance; maintain this in all changes
Expand Down
22 changes: 22 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,28 @@ set(BUILD_SQLCIPHER_CONNECTOR OFF CACHE BOOL "Build SQLCipher Connector" FORCE)
set(BUILD_TESTING OFF CACHE BOOL "Build tests" FORCE)
add_subdirectory(thirdparty/sqlpp23)

# --- Testing ---
# Enable testing for this project (only when mollerdb is the top-level project)
option(BUILD_TESTING "Build mollerdb tests" ON)

if(CMAKE_PROJECT_NAME STREQUAL PROJECT_NAME AND BUILD_TESTING)
enable_testing()

# Fetch Google Test
include(FetchContent)
FetchContent_Declare(
googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG v1.15.2
)
# For Windows: Prevent overriding the parent project's compiler/linker settings
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
FetchContent_MakeAvailable(googletest)

# Add tests subdirectory
add_subdirectory(tests/cpp)
endif()

# --- Core C++ Library ---
add_library(mollerdb_core SHARED
src/Database.cpp
Expand Down
69 changes: 69 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,75 @@ Compile with:
g++ -std=c++23 -o my_program my_program.cpp -lmollerdb -larrow
```

## Testing

The mollerdb project includes comprehensive test suites for both C++ and Python components.

### Running Tests

#### Python Tests

The Python tests use pytest and can be run after installing the package:

```bash
# Install the package with test dependencies
pip install -e .[test]

# Run all Python tests
pytest tests/python/ -v

# Run specific test file
pytest tests/python/test_database.py -v

# Run with coverage
pytest tests/python/ --cov=mollerdb
```

#### C++ Tests

The C++ tests use Google Test and are built with CMake:

```bash
# Build with tests enabled
mkdir build && cd build
cmake .. -DBUILD_TESTING=ON
cmake --build .

# Run all C++ tests
ctest --output-on-failure

# Run tests with verbose output
ctest -V

# Run specific test
./tests/cpp/test_database
```

### Test Structure

```
tests/
├── cpp/
│ ├── CMakeLists.txt
│ └── test_database.cpp # C++ Database class tests
└── python/
├── __init__.py
├── conftest.py # pytest fixtures
├── test_basic.py # Import and initialization tests
├── test_database.py # Database class tests
└── test_integration.py # Integration tests
```

### Writing Tests

When contributing new features, please include tests:

- **C++ Tests**: Add test cases to existing files in `tests/cpp/` or create new test files
- **Python Tests**: Add test functions to files in `tests/python/` following pytest conventions
- **Integration Tests**: Add end-to-end tests to `tests/python/test_integration.py`

All tests are automatically run in CI on every pull request.

## API Reference

### Database Class
Expand Down
Loading
Loading