Skip to content

Commit f38fbe2

Browse files
sgmvsgmv
committed
add end-to-end test
1 parent f517419 commit f38fbe2

7 files changed

Lines changed: 1472 additions & 20 deletions

File tree

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,6 @@ docs/
2828
coverage/
2929
tarpaulin-report.html
3030
cobertura.xml
31+
32+
# ai assistant files
33+
skills

‎README.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ mdbook serve doc/
7373
### Running Tests
7474

7575
```bash
76-
# Run all tests
76+
# Run all unit tests
7777
cargo test
7878

7979
# Run tests with output
@@ -86,6 +86,31 @@ cargo test common::tests
8686
cargo test -- --test-threads=4
8787
```
8888

89+
### E2E Integration Tests
90+
91+
End-to-end tests validate the complete pipeline using real Docker containers (go-carbon + carbonapi).
92+
93+
#### Prerequisites
94+
- Docker installed and running
95+
- Ports available: 2003, 8080, 3005, 9999
96+
97+
#### Running E2E Tests
98+
99+
```bash
100+
# Run E2E tests (Docker containers are managed automatically)
101+
cargo test --test integration_e2e_reporter -- --ignored --nocapture
102+
```
103+
104+
The E2E test validates 4 scenarios:
105+
| Scenario | Expected Weight | Description |
106+
|----------|-----------------|-------------|
107+
| healthy | 0 | All metrics within thresholds |
108+
| degraded_slow | 1 | API response time > 1200ms |
109+
| degraded_errors | 1 | Success rate < 65% |
110+
| outage | 2 | 100% API failures |
111+
112+
For details, see [Testing Guide](doc/testing.md).
113+
89114
### Test Coverage
90115

91116
```bash
@@ -99,8 +124,6 @@ cargo tarpaulin --out Html
99124
open tarpaulin-report.html
100125
```
101126

102-
For detailed testing documentation, see [Testing Guide](doc/testing.md).
103-
104127
### JSON Schema for Configuration
105128

106129
A JSON Schema for configuration validation is auto-generated during build:

‎doc/testing.md‎

Lines changed: 126 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,40 +2,47 @@
22

33
## Overview
44

5-
This document describes the comprehensive test suite for the metrics-processor project, including test execution, coverage measurement, and regression protection.
5+
This document describes the comprehensive test suite for the metrics-processor project, including unit tests, integration tests, E2E tests, coverage measurement, and regression protection.
66

77
## Test Organization
88

99
### Test Structure
1010

1111
```
1212
tests/
13-
├── fixtures/ # Shared test fixtures and utilities
14-
│ ├── mod.rs # Module declaration
15-
│ ├── configs.rs # YAML configuration fixtures
16-
│ ├── graphite_responses.rs # Mock Graphite response data
17-
│ └── helpers.rs # Test helper functions and assertions
18-
├── documentation_validation.rs # Documentation validation tests
19-
├── integration_health.rs # Service health integration tests
20-
└── integration_api.rs # API integration tests
13+
├── fixtures/ # Shared test fixtures and utilities
14+
│ ├── mod.rs # Module declaration
15+
│ ├── configs.rs # YAML configuration fixtures
16+
│ ├── graphite_responses.rs # Mock Graphite response data
17+
│ └── helpers.rs # Test helper functions and assertions
18+
├── docker/ # Docker setup for E2E tests
19+
│ ├── docker-compose.yml # go-carbon + carbonapi containers
20+
│ ├── go-carbon.conf # Carbon storage configuration
21+
│ ├── carbonapi.yml # CarbonAPI configuration
22+
│ └── README.md # Docker setup documentation
23+
├── integration_e2e_reporter.rs # E2E tests with real Graphite
24+
├── integration_health.rs # Service health integration tests
25+
├── integration_api.rs # API integration tests
26+
└── documentation_validation.rs # Documentation validation tests
2127
2228
src/
23-
├── common.rs # + #[cfg(test)] mod tests { 11 tests }
24-
├── types.rs # + #[cfg(test)] mod tests { 6 tests }
25-
├── config.rs # + #[cfg(test)] mod tests { 7 tests }
26-
├── graphite.rs # + #[cfg(test)] mod tests { 6 tests }
27-
└── api/v1.rs # + #[cfg(test)] mod tests { 4 tests }
29+
├── common.rs # + #[cfg(test)] mod tests { 11 tests }
30+
├── types.rs # + #[cfg(test)] mod tests { 6 tests }
31+
├── config.rs # + #[cfg(test)] mod tests { 7 tests }
32+
├── graphite.rs # + #[cfg(test)] mod tests { 6 tests }
33+
└── api/v1.rs # + #[cfg(test)] mod tests { 4 tests }
2834
```
2935

3036
### Test Categories
3137

3238
1. **Unit Tests**: Located inline with source code using `#[cfg(test)]` modules
3339
2. **Integration Tests**: Located in `tests/` directory for cross-module scenarios
34-
3. **Fixtures**: Shared test data and utilities in `tests/fixtures/`
40+
3. **E2E Tests**: Full pipeline tests with real Docker containers
41+
4. **Fixtures**: Shared test data and utilities in `tests/fixtures/`
3542

3643
## Running Tests
3744

38-
### Run All Tests
45+
### Run All Unit Tests
3946

4047
```bash
4148
cargo test
@@ -50,7 +57,7 @@ cargo test common::tests
5057
# Run only config tests
5158
cargo test config::tests
5259

53-
# Run only integration tests
60+
# Run only integration tests (excluding E2E)
5461
cargo test --test integration_*
5562
```
5663

@@ -72,6 +79,108 @@ cargo test -- --test-threads=4
7279
cargo test -- --test-threads=1
7380
```
7481

82+
## E2E Integration Tests
83+
84+
The E2E tests (`integration_e2e_reporter.rs`) validate the complete metrics-processor pipeline using real Docker containers.
85+
86+
### What They Test
87+
88+
- Metrics ingestion via Carbon protocol
89+
- Query processing via CarbonAPI
90+
- Health expression evaluation
91+
- Incident creation and severity assignment
92+
- Reporter log output format and content
93+
94+
### Prerequisites
95+
96+
1. **Docker**: Must be installed and running
97+
2. **Available Ports**:
98+
- `2003` - Carbon plaintext protocol
99+
- `8080` - CarbonAPI query endpoint
100+
- `3005` - Convertor API
101+
- `9999` - Mock Status Dashboard
102+
103+
### Running E2E Tests
104+
105+
```bash
106+
# Run E2E tests (Docker containers managed automatically)
107+
cargo test --test integration_e2e_reporter -- --ignored --nocapture
108+
```
109+
110+
The test automatically:
111+
1. Restarts Docker containers to ensure clean state
112+
2. Builds convertor and reporter binaries
113+
3. Runs all 4 test scenarios
114+
4. Validates log output for each scenario
115+
116+
### Test Scenarios
117+
118+
| Scenario | Weight | Condition | Expected Log |
119+
|----------|--------|-----------|--------------|
120+
| `healthy` | 0 | All metrics OK | No incident log |
121+
| `degraded_slow` | 1 | Response time > 1200ms | `matched_expression="...api_slow..."` |
122+
| `degraded_errors` | 1 | Success rate < 65% | `matched_expression="...api_success_rate_low..."` |
123+
| `outage` | 2 | 100% failures | `matched_expression="...api_down"` |
124+
125+
### E2E Test Architecture
126+
127+
```
128+
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
129+
│ Test Code │────▶│ go-carbon │────▶│ carbonapi │
130+
│(write data) │ │ (storage) │ │ (query) │
131+
└─────────────┘ └─────────────┘ └─────────────┘
132+
│
133+
┌────────────────────────────────────────┘
134+
▼
135+
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
136+
│ Convertor │────▶│ Reporter │────▶│ Mock Status │
137+
│ (process) │ │ (alert) │ │ Dashboard │
138+
└─────────────┘ └─────────────┘ └─────────────┘
139+
│ │
140+
│ ▼
141+
│ ┌─────────────┐
142+
└────────────▶│ Log Output │◀── Test validates
143+
│ (stdout) │
144+
└─────────────┘
145+
```
146+
147+
### Data Isolation
148+
149+
Each scenario uses a unique service name (e.g., `rms_healthy`, `rms_outage`) to prevent data pollution between scenarios. This allows sequential execution without clearing Graphite data.
150+
151+
### E2E Troubleshooting
152+
153+
#### "Docker containers failed to start"
154+
```bash
155+
# Check Docker is running
156+
docker ps
157+
158+
# Check port availability
159+
lsof -i :2003 -i :8080 -i :3005 -i :9999
160+
161+
# Manually start containers
162+
cd tests/docker && docker compose up -d
163+
```
164+
165+
#### "Convertor not ready after timeout"
166+
- The convertor may need more time to start
167+
- Check for port conflicts on 3005
168+
- Review convertor stderr output
169+
170+
#### "No incident log found"
171+
```bash
172+
# Verify Graphite received data
173+
curl 'http://localhost:8080/metrics/find?query=stats.*&format=json'
174+
175+
# Check go-carbon logs
176+
docker logs metrics-processor-go-carbon
177+
```
178+
179+
#### "Log validation failed"
180+
- Verify expected expression matches the config
181+
- Check for ANSI escape codes (test strips them automatically)
182+
- Review the full log output in test output
183+
75184
## Test Coverage
76185

77186
### Measuring Coverage

‎tests/docker/carbonapi.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# CarbonAPI configuration for testing
2+
listen: ":8080"
3+
prefix: ""
4+
notFoundStatusCode: 404
5+
headersToPass: []
6+
headersToLog: []
7+
8+
upstreams:
9+
backends:
10+
- "http://go-carbon:7002"
11+
12+
timeouts:
13+
find: "2s"
14+
render: "10s"
15+
connect: "2s"
16+
17+
expireDelaySec: 600
18+
maxBatchSize: 100
19+
20+
concurrency:
21+
maxProcs: 0
22+
maxRenderRequest: 10
23+
maxBatchRenderRequest: 10
24+
25+
cache:
26+
type: "mem"
27+
size_mb: 0
28+
defaultTimeoutSec: 60
29+
30+
logger:
31+
- logger: ""
32+
file: "stdout"
33+
level: "info"
34+
encoding: "console"
35+
encodingTime: "iso8601"
36+
encodingDuration: "seconds"

‎tests/docker/docker-compose.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
services:
2+
# go-carbon for metrics storage (Carbon replacement)
3+
go-carbon:
4+
image: ghcr.io/go-graphite/go-carbon:latest
5+
container_name: metrics-processor-go-carbon
6+
user: "0:0" # Run as root to ensure write permissions
7+
ports:
8+
- "2003:2003" # Carbon plaintext
9+
- "2004:2004" # Carbon pickle
10+
- "7002:7002" # Carbonserver (for carbonapi)
11+
volumes:
12+
- whisper_data:/data/graphite/whisper
13+
- ./go-carbon.conf:/etc/go-carbon/go-carbon.conf:ro
14+
healthcheck:
15+
test: ["CMD", "nc", "-z", "localhost", "2003"]
16+
interval: 10s
17+
timeout: 5s
18+
retries: 5
19+
20+
# CarbonAPI for Graphite-compatible API
21+
carbonapi:
22+
image: quay.io/opentelekomcloud/carbonapi:v0.16.1
23+
container_name: metrics-processor-carbonapi
24+
ports:
25+
- "8080:8080" # Graphite API
26+
volumes:
27+
- ./carbonapi.yml:/etc/carbonapi.yml:ro
28+
depends_on:
29+
go-carbon:
30+
condition: service_healthy
31+
healthcheck:
32+
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/render?format=json"]
33+
interval: 10s
34+
timeout: 5s
35+
retries: 5
36+
37+
volumes:
38+
whisper_data:

‎tests/docker/go-carbon.conf‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# go-carbon configuration for testing
2+
[common]
3+
user = ""
4+
graph-prefix = "carbon.agents.{host}"
5+
metric-endpoint = "local"
6+
metric-interval = "1m0s"
7+
max-cpu = 0
8+
9+
[whisper]
10+
data-dir = "/data/graphite/whisper"
11+
workers = 1
12+
max-updates-per-second = 0
13+
sparse-create = false
14+
enabled = true
15+
16+
[cache]
17+
max-size = 1000000
18+
write-strategy = "max"
19+
20+
[udp]
21+
listen = ":2003"
22+
enabled = false
23+
24+
[tcp]
25+
listen = ":2003"
26+
enabled = true
27+
28+
[pickle]
29+
listen = ":2004"
30+
enabled = true
31+
max-message-size = 67108864
32+
33+
[carbonlink]
34+
listen = ":7007"
35+
enabled = true
36+
read-timeout = "30s"
37+
38+
[carbonserver]
39+
listen = ":7002"
40+
enabled = true
41+
buckets = 10
42+
max-globs = 100
43+
metrics-as-counters = false
44+
read-timeout = "60s"
45+
write-timeout = "60s"
46+
scan-frequency = "10s"
47+
max-metrics-globbed = 30000
48+
max-metrics-rendered = 1000
49+
file-list-cache = ""
50+
trigram-index = false
51+
internal-stats-dir = ""
52+
query-cache-enabled = false
53+
54+
[logging]
55+
file = "stdout"
56+
level = "info"
57+
encoding = "mixed"

0 commit comments

Comments
 (0)