Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ wheels/
*.egg-info/
.installed.cfg
*.egg
projects/*/dist/
packages/*/dist/

# Virtual environments
.venv
Expand Down
26 changes: 7 additions & 19 deletions .github/Dockerfile.ci
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Dockerfile for CI/CD with Poetry and Python pre-installed
FROM python:3.12-slim
# Dockerfile for CI/CD with uv and Python pre-installed
FROM ghcr.io/astral-sh/uv:python3.12-trixie

LABEL org.opencontainers.image.source="https://github.com/OpenSecFlow/netdriver"
LABEL org.opencontainers.image.description="CI/CD image with Python 3.12 and Poetry"
LABEL org.opencontainers.image.description="CI/CD image with Python 3.12 and uv"
LABEL org.opencontainers.image.licenses="Apache-2.0"

# Install system dependencies
Expand All @@ -11,22 +11,10 @@ RUN apt-get update && apt-get install -y \
curl \
&& rm -rf /var/lib/apt/lists/*

# Install Poetry
ENV POETRY_VERSION=1.8.3
ENV POETRY_HOME=/opt/poetry
ENV POETRY_NO_INTERACTION=1
ENV POETRY_VIRTUALENVS_IN_PROJECT=false
ENV POETRY_VIRTUALENVS_CREATE=true

RUN curl -sSL https://install.python-poetry.org | python3 - \
&& ln -s /opt/poetry/bin/poetry /usr/local/bin/poetry

# Install Poetry plugins
RUN poetry self add poetry-multiproject-plugin \
&& poetry self add poetry-polylith-plugin

# Verify installation
RUN poetry --version && poetry self show plugins
# Set uv environment variables
ENV UV_PYTHON=python3.12
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy

WORKDIR /workspace

Expand Down
25 changes: 13 additions & 12 deletions .github/workflows/QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ The `publish-pypi.yml` workflow uses a pre-built Docker image for faster executi
**Or build locally:**

```bash
docker build -t ghcr.io/opensecflow/netdriver/python-poetry:3.12 -f .github/Dockerfile.ci .
docker push ghcr.io/opensecflow/netdriver/python-poetry:3.12
docker build -t ghcr.io/opensecflow/netdriver/python-uv:3.12 -f .github/Dockerfile.ci .
docker push ghcr.io/opensecflow/netdriver/python-uv:3.12
```

**Note**: This only needs to be done once. The image will be cached and reused.
Expand Down Expand Up @@ -103,11 +103,11 @@ Before publishing to production PyPI, test with TestPyPI:

```bash
# 1. Update version numbers
poetry version -P projects/agent 0.3.1
poetry version -P projects/simunet 0.3.1
sed -i 's/^version = ".*"/version = "0.3.1"/' packages/agent/pyproject.toml
sed -i 's/^version = ".*"/version = "0.3.1"/' packages/simunet/pyproject.toml

# 2. Commit changes
git add projects/*/pyproject.toml
git add packages/*/pyproject.toml
git commit -m "chore: bump version to 0.3.1"
git push

Expand Down Expand Up @@ -156,27 +156,28 @@ The `release.yml` workflow will automatically:
**Solution:** Version already exists on PyPI. Bump the version:

```bash
poetry version -P projects/agent patch
poetry version -P projects/simunet patch
# Manually update version in pyproject.toml files
sed -i 's/^version = ".*"/version = "0.3.2"/' packages/agent/pyproject.toml
sed -i 's/^version = ".*"/version = "0.3.2"/' packages/simunet/pyproject.toml
# Then rebuild and publish
```

### Workflow fails with "Poetry not found" or image pull error
### Workflow fails with "uv not found" or image pull error

**Solution:** Build the CI Docker image first

```bash
# Go to Actions → "Build CI Image" → Run workflow
```

Or check the image name matches: `ghcr.io/opensecflow/netdriver/python-poetry:3.12`
Or check the image name matches: `ghcr.io/opensecflow/netdriver/python-uv:3.12`

### Package shows as "0 B" or malformed

**Solution:** Check build output - Polylith path warnings are normal, verify wheel contents:

```bash
unzip -l projects/agent/dist/netdriver_agent-*.whl
unzip -l packages/agent/dist/netdriver_agent-*.whl
```

## Best Practices
Expand All @@ -185,7 +186,7 @@ unzip -l projects/agent/dist/netdriver_agent-*.whl

✅ **DO:**

- Keep version numbers in sync across `projects/agent/pyproject.toml` and `projects/simunet/pyproject.toml`
- Keep version numbers in sync across `packages/agent/pyproject.toml` and `packages/simunet/pyproject.toml`
- Use semantic versioning: `MAJOR.MINOR.PATCH`
- Test on TestPyPI before production

Expand All @@ -198,7 +199,7 @@ unzip -l projects/agent/dist/netdriver_agent-*.whl
### Release Process

1. Develop features on branches
2. Test locally: `poetry build-project -C projects/agent`
2. Test locally: `uv build --directory packages/agent`
3. Create PR and verify build test passes
4. Merge to master
5. Test on TestPyPI
Expand Down
54 changes: 26 additions & 28 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ This directory contains GitHub Actions workflows for automated building, testing

**What it does**:

- Uses pre-built Docker container with Poetry installed
- Uses pre-built Docker container with uv installed
- Builds wheel packages for selected projects
- Publishes to PyPI or TestPyPI
- Uploads build artifacts
Expand Down Expand Up @@ -68,10 +68,10 @@ The project supports independent release workflows for agent and simunet:

```bash
# Update agent version in pyproject.toml (optional, will be updated by workflow)
poetry version 1.0.0 -C projects/agent
sed -i 's/^version = ".*"/version = "1.0.0"/' packages/agent/pyproject.toml

# Commit version changes (optional)
git add projects/agent/pyproject.toml
git add packages/agent/pyproject.toml
git commit -m "chore: bump agent version to 1.0.0"

# Create and push agent tag
Expand All @@ -98,10 +98,10 @@ git push origin agent-1.0.0

```bash
# Update simunet version in pyproject.toml (optional, will be updated by workflow)
poetry version 2.5.0 -C projects/simunet
sed -i 's/^version = ".*"/version = "2.5.0"/' packages/simunet/pyproject.toml

# Commit version changes (optional)
git add projects/simunet/pyproject.toml
git add packages/simunet/pyproject.toml
git commit -m "chore: bump simunet version to 2.5.0"

# Create and push simunet tag
Expand Down Expand Up @@ -165,13 +165,13 @@ Use this when you only need to release the agent:
1. **Update version number** (optional):

```bash
poetry version 1.0.0 -C projects/agent
sed -i 's/^version = ".*"/version = "1.0.0"/' packages/agent/pyproject.toml
```

2. **Commit changes** (optional):

```bash
git add projects/agent/pyproject.toml
git add packages/agent/pyproject.toml
git commit -m "chore: bump agent version to 1.0.0"
git push origin master
```
Expand All @@ -197,13 +197,13 @@ Use this when you need to release simunet:
1. **Update version number** (optional):

```bash
poetry version 2.5.0 -C projects/simunet
sed -i 's/^version = ".*"/version = "2.5.0"/' packages/simunet/pyproject.toml
```

2. **Commit changes** (optional):

```bash
git add projects/simunet/pyproject.toml
git add packages/simunet/pyproject.toml
git commit -m "chore: bump simunet version to 2.5.0"
git push origin master
```
Expand Down Expand Up @@ -235,8 +235,8 @@ To test publishing before official release:
2. **Or use CLI**:

```bash
poetry publish -P projects/agent -r testpypi
poetry publish -P projects/simunet -r testpypi
uv publish --directory packages/agent --publish-url https://test.pypi.org/legacy/ --token $TESTPYPI_TOKEN
uv publish --directory packages/simunet --publish-url https://test.pypi.org/legacy/ --token $TESTPYPI_TOKEN
```

3. **Verify on TestPyPI**:
Expand Down Expand Up @@ -382,12 +382,12 @@ The project uses prefixed tag patterns for independent releases:

### 5. Build CI Image (`build-ci-image.yml`)

**Purpose**: Creates a Docker image with Poetry and Python pre-installed for faster CI/CD
**Purpose**: Creates a Docker image with uv and Python pre-installed for faster CI/CD

**What it includes**:

- Python 3.12
- Poetry with multiproject and polylith plugins
- uv package manager
- Git and essential build tools

**Building the image**:
Expand All @@ -408,24 +408,24 @@ The `publish-pypi.yml` workflow uses this approach:
jobs:
publish:
runs-on: ubuntu-latest
container:
image: ghcr.io/${{ github.repository }}/python-poetry:3.12
steps:
- uses: actions/checkout@v4
# Poetry and plugins are already installed!
- run: poetry build-project -C projects/agent
- uses: astral-sh/setup-uv@v4
- run: uv python install
- run: uv sync
- run: uv build --directory packages/agent
```

**Benefits of using Docker image**:

- ⚡ **Faster**: No need to install Poetry and plugins on every run
- ⚡ **Faster**: No need to install uv on every run
- 🔒 **Consistent**: Same environment across all workflows
- 💾 **Cacheable**: Image layers are cached by Docker
- 🎯 **Reproducible**: Exact same versions every time

**Image locations**:

- GitHub Container Registry: `ghcr.io/opensecflow/netdriver/python-poetry:3.12`
- GitHub Container Registry: `ghcr.io/opensecflow/netdriver/python-uv:3.12`
- Available tags: `latest`, `master`, `<branch>-<sha>`

**Benefits**:
Expand All @@ -452,7 +452,7 @@ netdriver/
│ ├── agent/ # REST API service
│ └── simunet/ # Simulation network
├── components/ # Shared components
└── projects/
└── packages/
├── agent/
│ └── pyproject.toml
└── simunet/
Expand Down Expand Up @@ -485,20 +485,18 @@ container:
password: ${{ secrets.DOCKER_PASSWORD }}
```

### 3. Verify Poetry is available
### 3. Verify uv is available

Poetry and plugins are pre-installed, so you can use them directly:
uv is pre-installed, so you can use it directly:

```yaml
- name: Verify Poetry installation
- name: Verify uv installation
run: |
poetry --version
poetry self show plugins
uv --version
```

## References

- [Poetry Documentation](https://python-poetry.org/docs/)
- [Poetry Polylith Plugin](https://github.com/DavidVujic/poetry-polylith-plugin)
- [PyPI Publishing Guide](https://packaging.python.org/tutorials/packaging-projects/)
- [uv Documentation](https://docs.astral.sh/uv/)
- [PyPI Publishing Guide](https://packaging.python.org/tutorials/packaging-packages/)
- [GitHub Actions - Python](https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python)
2 changes: 1 addition & 1 deletion .github/workflows/build-ci-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:

env:
REGISTRY: ghcr.io
IMAGE_NAME: opensecflow/netdriver/python-poetry
IMAGE_NAME: opensecflow/netdriver/python-uv

jobs:
build:
Expand Down
Loading
Loading