Refactor CI workflows to reorder runner labels for improved clarity and consistency across API Docs, API Gateway, Service Adapters, Frontend, and Integration Tests; update documentation to include new runner labels and optimization recommendations.
Some checks failed
API Gateway (Java Spring Boot) / test (17) (push) Failing after 2s
API Gateway (Java Spring Boot) / test (21) (push) Failing after 3s
API Gateway (Java Spring Boot) / build (push) Has been skipped
API Gateway (Java Spring Boot) / security (push) Has been skipped
LabFusion CI/CD Pipeline / api-gateway (push) Failing after 3s
LabFusion CI/CD Pipeline / service-adapters (push) Failing after 3s
LabFusion CI/CD Pipeline / api-docs (push) Failing after 3s
Service Adapters (Python FastAPI) / test (3.1) (push) Failing after 3s
LabFusion CI/CD Pipeline / frontend (push) Failing after 2s
LabFusion CI/CD Pipeline / integration-tests (push) Has been skipped
LabFusion CI/CD Pipeline / security-scan (push) Has been skipped
Service Adapters (Python FastAPI) / test (3.11) (push) Failing after 2s
Frontend (React) / test (16) (push) Failing after 2s
Service Adapters (Python FastAPI) / test (3.12) (push) Failing after 2s
Frontend (React) / test (18) (push) Failing after 3s
Service Adapters (Python FastAPI) / test (3.9) (push) Failing after 3s
Frontend (React) / test (20) (push) Failing after 3s
Service Adapters (Python FastAPI) / build (push) Has been skipped
Service Adapters (Python FastAPI) / security (push) Has been skipped
Frontend (React) / build (push) Has been skipped
Frontend (React) / security (push) Has been skipped
Frontend (React) / lighthouse (push) Has been skipped
Docker Build and Push / build-and-push (push) Failing after 4m52s
Docker Build and Push / security-scan (push) Has been skipped
Integration Tests / integration-tests (push) Failing after 2s
Docker Build and Push / deploy-staging (push) Has been skipped
Integration Tests / performance-tests (push) Has been skipped
Docker Build and Push / deploy-production (push) Has been skipped
API Docs (Node.js Express) / test (16) (push) Failing after 2s
API Docs (Node.js Express) / test (18) (push) Failing after 2s
API Docs (Node.js Express) / test (20) (push) Failing after 2s
API Docs (Node.js Express) / build (push) Has been skipped
API Docs (Node.js Express) / security (push) Has been skipped
Some checks failed
API Gateway (Java Spring Boot) / test (17) (push) Failing after 2s
API Gateway (Java Spring Boot) / test (21) (push) Failing after 3s
API Gateway (Java Spring Boot) / build (push) Has been skipped
API Gateway (Java Spring Boot) / security (push) Has been skipped
LabFusion CI/CD Pipeline / api-gateway (push) Failing after 3s
LabFusion CI/CD Pipeline / service-adapters (push) Failing after 3s
LabFusion CI/CD Pipeline / api-docs (push) Failing after 3s
Service Adapters (Python FastAPI) / test (3.1) (push) Failing after 3s
LabFusion CI/CD Pipeline / frontend (push) Failing after 2s
LabFusion CI/CD Pipeline / integration-tests (push) Has been skipped
LabFusion CI/CD Pipeline / security-scan (push) Has been skipped
Service Adapters (Python FastAPI) / test (3.11) (push) Failing after 2s
Frontend (React) / test (16) (push) Failing after 2s
Service Adapters (Python FastAPI) / test (3.12) (push) Failing after 2s
Frontend (React) / test (18) (push) Failing after 3s
Service Adapters (Python FastAPI) / test (3.9) (push) Failing after 3s
Frontend (React) / test (20) (push) Failing after 3s
Service Adapters (Python FastAPI) / build (push) Has been skipped
Service Adapters (Python FastAPI) / security (push) Has been skipped
Frontend (React) / build (push) Has been skipped
Frontend (React) / security (push) Has been skipped
Frontend (React) / lighthouse (push) Has been skipped
Docker Build and Push / build-and-push (push) Failing after 4m52s
Docker Build and Push / security-scan (push) Has been skipped
Integration Tests / integration-tests (push) Failing after 2s
Docker Build and Push / deploy-staging (push) Has been skipped
Integration Tests / performance-tests (push) Has been skipped
Docker Build and Push / deploy-production (push) Has been skipped
API Docs (Node.js Express) / test (16) (push) Failing after 2s
API Docs (Node.js Express) / test (18) (push) Failing after 2s
API Docs (Node.js Express) / test (20) (push) Failing after 2s
API Docs (Node.js Express) / build (push) Has been skipped
API Docs (Node.js Express) / security (push) Has been skipped
This commit is contained in:
183
docs/OPTIMIZATION_RECOMMENDATIONS.md
Normal file
183
docs/OPTIMIZATION_RECOMMENDATIONS.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# LabFusion CI/CD Optimization Recommendations
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
### What We're Currently Using:
|
||||
- **Full Ubuntu Latest images** for all workloads (~2-3GB per container)
|
||||
- **Setup actions** that install tools every time (Java, Python, Node.js)
|
||||
- **Generic labels** that don't optimize for specific workloads
|
||||
|
||||
### What Each Pipeline Actually Needs:
|
||||
|
||||
#### Heavy Runner (Java + Python):
|
||||
- **Java**: JDK 17, Maven (for API Gateway)
|
||||
- **Python**: Python 3.11, pip, pytest, flake8, black, isort, bandit, safety, mypy (for Service Adapters)
|
||||
- **Docker**: For building images
|
||||
|
||||
#### Light Runner (Node.js + Frontend):
|
||||
- **Node.js**: Node 16/18/20, npm (for API Docs and Frontend)
|
||||
- **Docker**: For building images
|
||||
|
||||
#### Docker Runner:
|
||||
- **Docker**: Docker-in-Docker capabilities
|
||||
- **Basic tools**: curl, git
|
||||
|
||||
#### Security Runner:
|
||||
- **Security tools**: Trivy, OWASP ZAP, etc.
|
||||
- **Docker**: For scanning images
|
||||
|
||||
## Optimization Strategy
|
||||
|
||||
### 1. Use Specialized Images
|
||||
|
||||
Instead of full Ubuntu latest, use optimized images from [Gitea's runner-images](https://gitea.com/docker.gitea.com/runner-images):
|
||||
|
||||
```yaml
|
||||
# Heavy Runner - Java workloads
|
||||
- "java:docker://docker.gitea.com/runner-images:ubuntu-22.04-java17-maven"
|
||||
- "heavy:docker://docker.gitea.com/runner-images:ubuntu-22.04-java17-maven"
|
||||
|
||||
# Heavy Runner - Python workloads
|
||||
- "python:docker://docker.gitea.com/runner-images:ubuntu-22.04-python3.11"
|
||||
|
||||
# Light Runner - Node.js workloads
|
||||
- "nodejs:docker://docker.gitea.com/runner-images:ubuntu-22.04-node20"
|
||||
- "frontend:docker://docker.gitea.com/runner-images:ubuntu-22.04-node20"
|
||||
|
||||
# Docker Runner - Docker-in-Docker
|
||||
- "docker:docker://docker.gitea.com/runner-images:ubuntu-22.04-docker"
|
||||
|
||||
# Security Runner - Security tools
|
||||
- "security:docker://docker.gitea.com/runner-images:ubuntu-22.04-security"
|
||||
```
|
||||
|
||||
### 2. Benefits of Specialized Images
|
||||
|
||||
- **Faster startup**: Pre-installed tools mean no setup time
|
||||
- **Smaller images**: Only includes what's needed (~500MB vs 2-3GB)
|
||||
- **Better caching**: Tools are already in the image layer
|
||||
- **More reliable**: No network dependency for tool installation
|
||||
|
||||
### 3. Workflow Optimizations
|
||||
|
||||
#### Remove Redundant Setup Steps
|
||||
|
||||
**Before (current):**
|
||||
```yaml
|
||||
- name: Set up JDK 17
|
||||
uses: actions/setup-java@v4
|
||||
with:
|
||||
java-version: '17'
|
||||
distribution: 'temurin'
|
||||
```
|
||||
|
||||
**After (optimized):**
|
||||
```yaml
|
||||
# Remove this step - Java 17 and Maven are pre-installed
|
||||
# - name: Set up JDK 17
|
||||
# uses: actions/setup-java@v4
|
||||
```
|
||||
|
||||
#### Use Matrix Strategy for Node.js Versions
|
||||
|
||||
**Current approach**: Multiple setup steps
|
||||
**Optimized approach**: Use matrix with specialized images
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
node-version: [16, 18, 20]
|
||||
image:
|
||||
- "ubuntu-22.04-node16"
|
||||
- "ubuntu-22.04-node18"
|
||||
- "ubuntu-22.04-node20"
|
||||
```
|
||||
|
||||
### 4. Fallback Strategy
|
||||
|
||||
Keep `ubuntu-latest` as fallback for:
|
||||
- Complex builds that need many tools
|
||||
- Debugging when specialized images fail
|
||||
- New workloads not yet optimized
|
||||
|
||||
### 5. Performance Impact
|
||||
|
||||
#### Expected Improvements:
|
||||
- **Startup time**: 30-60 seconds faster per job
|
||||
- **Image size**: 60-70% smaller
|
||||
- **Cache efficiency**: Better layer reuse
|
||||
- **Resource usage**: Lower memory footprint
|
||||
|
||||
#### Example Timeline:
|
||||
```
|
||||
Current: Checkout (10s) + Setup Java (30s) + Setup Maven (20s) + Run Tests (60s) = 120s
|
||||
Optimized: Checkout (10s) + Run Tests (60s) = 70s
|
||||
Savings: 50s per job (42% faster)
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Update Config Files ✅
|
||||
- [x] Update `config_heavy.yaml` with specialized images
|
||||
- [x] Update `config_light.yaml` with Node.js images
|
||||
- [x] Update `config_docker.yaml` with Docker-in-Docker image
|
||||
- [x] Update `config_security.yaml` with security tools image
|
||||
|
||||
### Phase 2: Test and Validate
|
||||
- [ ] Test each runner with optimized images
|
||||
- [ ] Verify all tools are available
|
||||
- [ ] Check performance improvements
|
||||
- [ ] Validate fallback works
|
||||
|
||||
### Phase 3: Optimize Workflows
|
||||
- [ ] Remove redundant setup steps
|
||||
- [ ] Update matrix strategies
|
||||
- [ ] Add performance monitoring
|
||||
- [ ] Document changes
|
||||
|
||||
### Phase 4: Monitor and Tune
|
||||
- [ ] Monitor job execution times
|
||||
- [ ] Track resource usage
|
||||
- [ ] Fine-tune based on metrics
|
||||
- [ ] Update documentation
|
||||
|
||||
## Available Gitea Runner Images
|
||||
|
||||
Check [Gitea's runner-images repository](https://gitea.com/docker.gitea.com/runner-images) for available images:
|
||||
|
||||
### Java Images:
|
||||
- `ubuntu-22.04-java17-maven`
|
||||
- `ubuntu-22.04-java21-maven`
|
||||
- `ubuntu-22.04-java17-gradle`
|
||||
|
||||
### Python Images:
|
||||
- `ubuntu-22.04-python3.11`
|
||||
- `ubuntu-22.04-python3.12`
|
||||
- `ubuntu-22.04-python3.11-pip`
|
||||
|
||||
### Node.js Images:
|
||||
- `ubuntu-22.04-node18`
|
||||
- `ubuntu-22.04-node20`
|
||||
- `ubuntu-22.04-node21`
|
||||
|
||||
### Docker Images:
|
||||
- `ubuntu-22.04-docker`
|
||||
- `ubuntu-22.04-docker-compose`
|
||||
|
||||
### Security Images:
|
||||
- `ubuntu-22.04-security`
|
||||
- `ubuntu-22.04-trivy`
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Test the updated config files** with a simple job
|
||||
2. **Verify image availability** on Gitea's registry
|
||||
3. **Update workflows** to remove redundant setup steps
|
||||
4. **Monitor performance** improvements
|
||||
5. **Document the changes** for the team
|
||||
|
||||
## References
|
||||
|
||||
- [Gitea Runner Images](https://gitea.com/docker.gitea.com/runner-images)
|
||||
- [Gitea Actions Design](https://docs.gitea.com/usage/actions/design#act-runner)
|
||||
- [LabFusion Runner Labels](RUNNER_LABELS.md)
|
||||
180
docs/RUNNER_LABELS.md
Normal file
180
docs/RUNNER_LABELS.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# Gitea Actions Runner Labels - Technical Summary
|
||||
|
||||
Based on the [Gitea Actions Design Documentation](https://docs.gitea.com/usage/actions/design#act-runner), this document summarizes how runner labels work in Gitea Actions.
|
||||
|
||||
## Overview
|
||||
|
||||
Gitea Actions uses a label-based system to match workflow jobs with appropriate runners. The `runs-on: ubuntu-latest` directive in workflow files means the job will run on a runner that has the `ubuntu-latest` label.
|
||||
|
||||
## Label Format
|
||||
|
||||
Runner labels follow the format: `label[:schema[:args]]`
|
||||
|
||||
### Schema Types
|
||||
|
||||
1. **`host` (default)**: Run jobs directly on the host machine
|
||||
- `my_custom_label:host` - Run on host
|
||||
- `my_custom_label` - Same as above (host is default)
|
||||
|
||||
2. **`docker`**: Run jobs in Docker containers
|
||||
- `my_custom_label:docker://node:18` - Run in Node.js 18 container
|
||||
- `my_custom_label:docker://centos:7` - Run in CentOS 7 container
|
||||
|
||||
3. **`vm` (example, not implemented)**: Run in virtual machines
|
||||
- `my_custom_label:vm:ubuntu-latest` - Run in Ubuntu VM
|
||||
|
||||
## Label Mapping
|
||||
|
||||
When registering a runner, you specify which labels it can handle and how:
|
||||
|
||||
```bash
|
||||
# Example registration with custom labels
|
||||
act_runner register \
|
||||
--labels "ubuntu-latest:docker://ubuntu:22.04,self-hosted:host,heavy:docker://ubuntu:22.04"
|
||||
```
|
||||
|
||||
This means:
|
||||
- Jobs with `runs-on: ubuntu-latest` → Run in Ubuntu 22.04 Docker container
|
||||
- Jobs with `runs-on: self-hosted` → Run directly on host
|
||||
- Jobs with `runs-on: heavy` → Run in Ubuntu 22.04 Docker container
|
||||
|
||||
## LabFusion Implementation
|
||||
|
||||
For the LabFusion project, our runners use these labels:
|
||||
|
||||
### Heavy Runner
|
||||
- **Labels**: `ubuntu-latest,self-hosted,heavy,java,python`
|
||||
- **Purpose**: Java Spring Boot and Python FastAPI workloads
|
||||
- **Configuration**: Uses Docker containers for isolation
|
||||
- **Config File**: `config_heavy.yaml`
|
||||
|
||||
### Light Runner
|
||||
- **Labels**: `ubuntu-latest,self-hosted,light,nodejs,frontend`
|
||||
- **Purpose**: Node.js and React frontend workloads
|
||||
- **Configuration**: Uses Docker containers for isolation
|
||||
- **Config File**: `config_light.yaml`
|
||||
|
||||
### Docker Runner
|
||||
- **Labels**: `ubuntu-latest,self-hosted,docker,integration`
|
||||
- **Purpose**: Integration tests and Docker builds
|
||||
- **Configuration**: Full Docker-in-Docker capabilities
|
||||
- **Config File**: `config_docker.yaml`
|
||||
|
||||
### Security Runner
|
||||
- **Labels**: `ubuntu-latest,self-hosted,security,scan`
|
||||
- **Purpose**: Security scans and vulnerability assessments
|
||||
- **Configuration**: Specialized for security tools
|
||||
- **Config File**: `config_security.yaml`
|
||||
|
||||
## Label Mapping in Config Files
|
||||
|
||||
Each runner configuration file maps labels to Docker images:
|
||||
|
||||
### Heavy Runner (`config_heavy.yaml`)
|
||||
```yaml
|
||||
labels:
|
||||
- "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "heavy:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "java:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "python:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
```
|
||||
|
||||
### Light Runner (`config_light.yaml`)
|
||||
```yaml
|
||||
labels:
|
||||
- "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "light:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "nodejs:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "frontend:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
```
|
||||
|
||||
### Docker Runner (`config_docker.yaml`)
|
||||
```yaml
|
||||
labels:
|
||||
- "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "docker:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "integration:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
```
|
||||
|
||||
### Security Runner (`config_security.yaml`)
|
||||
```yaml
|
||||
labels:
|
||||
- "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "security:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
- "scan:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
```
|
||||
|
||||
## Label Priority and Matching
|
||||
|
||||
**Important**: The order of labels in `runs-on` matters! Gitea checks labels in order and uses the first match.
|
||||
|
||||
### Optimized Approach:
|
||||
```yaml
|
||||
# ✅ GOOD: Specific labels first, fallback last
|
||||
runs-on: [java, heavy, self-hosted]
|
||||
```
|
||||
|
||||
This will match:
|
||||
1. `java:docker://...` (optimized Java image) ✅
|
||||
2. `heavy:docker://...` (if java not available)
|
||||
3. `self-hosted:docker://...` (fallback to full Ubuntu)
|
||||
|
||||
### Avoid This:
|
||||
```yaml
|
||||
# ❌ BAD: Generic labels first
|
||||
runs-on: [self-hosted, heavy, java]
|
||||
```
|
||||
|
||||
This will match:
|
||||
1. `self-hosted:docker://...` (full Ubuntu image) ❌
|
||||
2. Never reaches the optimized `java` label
|
||||
|
||||
## Workflow Usage
|
||||
|
||||
In our CI/CD workflows, jobs specify which runner to use:
|
||||
|
||||
```yaml
|
||||
# API Gateway (Java) - uses optimized Java image
|
||||
runs-on: [java, heavy, self-hosted]
|
||||
|
||||
# Service Adapters (Python) - uses optimized Python image
|
||||
runs-on: [python, heavy, self-hosted]
|
||||
|
||||
# API Docs (Node.js) - uses optimized Node.js image
|
||||
runs-on: [nodejs, light, self-hosted]
|
||||
|
||||
# Frontend (React) - uses optimized Node.js image
|
||||
runs-on: [frontend, light, self-hosted]
|
||||
|
||||
# Integration Tests - uses optimized Docker image
|
||||
runs-on: [docker, integration, self-hosted]
|
||||
|
||||
# Security Scans - uses optimized security image
|
||||
runs-on: [security, scan, self-hosted]
|
||||
```
|
||||
|
||||
## Key Benefits
|
||||
|
||||
1. **Flexibility**: Mix and match labels for different workload types
|
||||
2. **Isolation**: Docker containers provide job isolation
|
||||
3. **Resource Management**: Different runners can have different resource limits
|
||||
4. **Specialization**: Dedicated runners for specific tasks (security, integration)
|
||||
5. **Scalability**: Easy to add more runners with specific label combinations
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use descriptive labels** that clearly indicate the runner's purpose
|
||||
2. **Combine multiple labels** to allow flexible job assignment
|
||||
3. **Use Docker containers** for better isolation and consistency
|
||||
4. **Match labels to workflow requirements** for optimal resource usage
|
||||
5. **Consider resource limits** when assigning labels to runners
|
||||
|
||||
## References
|
||||
|
||||
- [Gitea Actions Design Documentation](https://docs.gitea.com/usage/actions/design#act-runner)
|
||||
- [LabFusion Runners Setup](RUNNERS.md)
|
||||
- [LabFusion CI/CD Pipelines](CI_CD.md)
|
||||
@@ -115,5 +115,7 @@ runners/
|
||||
├── specs.md # Project specifications
|
||||
├── structure.txt # Project structure
|
||||
├── progress.md # Development progress tracking
|
||||
├── CI_CD.md # CI/CD pipeline documentation
|
||||
└── RUNNERS.md # Gitea runners setup and management
|
||||
├── RUNNERS.md # Gitea runners setup and management
|
||||
├── RUNNER_LABELS.md # Runner labels technical documentation
|
||||
├── OPTIMIZATION_RECOMMENDATIONS.md # CI/CD optimization recommendations
|
||||
└── CI_CD.md # CI/CD pipeline documentation
|
||||
Reference in New Issue
Block a user