Update CI workflows to utilize self-hosted runners with specific labels for API Docs, API Gateway, Service Adapters, Frontend, and Integration Tests; enhance progress documentation to reflect changes in runner configurations

docs/RUNNERS.md

@@ -0,0 +1,312 @@
+# LabFusion Gitea Runners Setup
+
+This document explains how to set up and manage multiple Gitea Actions runners for the LabFusion project using Docker Compose.
+
+## Overview
+
+The LabFusion project uses multiple specialized runners to handle different types of CI/CD workloads:
+
+- **Heavy Runner**: Java Spring Boot (API Gateway) and Python FastAPI (Service Adapters)
+- **Light Runner**: Node.js (API Docs) and React (Frontend)
+- **Docker Runner**: Integration tests and Docker builds
+- **Security Runner**: Security scans and vulnerability assessments
+
+## Quick Start
+
+### 1. Prerequisites
+
+- Docker and Docker Compose installed
+- Gitea instance running
+- Runner registration token from Gitea
+
+### 2. Configuration
+
+**Windows PowerShell:**
+```powershell
+# Copy the environment template
+Copy-Item env.runners.example .env.runners
+
+# Edit the configuration
+notepad .env.runners
+```
+
+**Linux/macOS:**
+```bash
+# Copy the environment template
+cp env.runners.example .env.runners
+
+# Edit the configuration
+nano .env.runners
+```
+
+**Manual Creation (if copy fails):**
+Create `.env.runners` file with the following content:
+
+```bash
+# Gitea instance URL (adjust port if different)
+GITEA_INSTANCE_URL=http://localhost:3000
+
+# Runner registration token (get from Gitea Admin > Actions > Runners)
+GITEA_RUNNER_TOKEN=your_runner_registration_token_here
+```
+
+**Important:** Replace `your_runner_registration_token_here` with your actual token from Gitea Admin > Actions > Runners.
+
+### 3. Start Runners
+
+**Linux/macOS:**
+```bash
+# Make script executable
+chmod +x scripts/manage-runners.sh
+
+# Start all runners
+./scripts/manage-runners.sh start
+```
+
+**Windows PowerShell:**
+```powershell
+# Start all runners
+.\scripts\manage-runners.ps1 start
+```
+
+**Docker Compose directly:**
+```bash
+docker-compose -f docker-compose.runners.yml --env-file .env.runners up -d
+```
+
+## Runner Configuration
+
+### Resource Allocation
+
+| Runner | CPU | Memory | Use Case |
+|--------|-----|--------|----------|
+| Heavy | 4 cores | 8GB | Java/Python builds |
+| Light | 2 cores | 4GB | Node.js/Frontend |
+| Docker | 6 cores | 12GB | Integration tests |
+| Security | 2 cores | 4GB | Security scans |
+
+### Labels
+
+Each runner is configured with specific labels for workload routing:
+
+- **Heavy Runner**: `ubuntu-latest,self-hosted,heavy,java,python`
+- **Light Runner**: `ubuntu-latest,self-hosted,light,nodejs,frontend`
+- **Docker Runner**: `ubuntu-latest,self-hosted,docker,integration`
+- **Security Runner**: `ubuntu-latest,self-hosted,security,scan`
+
+## Management Commands
+
+### Using the Management Scripts
+
+**Linux/macOS:**
+```bash
+# Start runners
+./scripts/manage-runners.sh start
+
+# Check status
+./scripts/manage-runners.sh status
+
+# View logs
+./scripts/manage-runners.sh logs
+./scripts/manage-runners.sh logs labfusion-runner-heavy
+
+# Restart runners
+./scripts/manage-runners.sh restart
+
+# Stop runners
+./scripts/manage-runners.sh stop
+
+# Clean up (removes all data)
+./scripts/manage-runners.sh clean
+```
+
+**Windows PowerShell:**
+```powershell
+# Start runners
+.\scripts\manage-runners.ps1 start
+
+# Check status
+.\scripts\manage-runners.ps1 status
+
+# View logs
+.\scripts\manage-runners.ps1 logs
+.\scripts\manage-runners.ps1 logs labfusion-runner-heavy
+
+# Restart runners
+.\scripts\manage-runners.ps1 restart
+
+# Stop runners
+.\scripts\manage-runners.ps1 stop
+
+# Clean up (removes all data)
+.\scripts\manage-runners.ps1 clean
+```
+
+### Direct Docker Compose Commands
+
+```bash
+# Start all runners
+docker-compose -f docker-compose.runners.yml up -d
+
+# Check status
+docker-compose -f docker-compose.runners.yml ps
+
+# View logs
+docker-compose -f docker-compose.runners.yml logs -f
+
+# Stop runners
+docker-compose -f docker-compose.runners.yml down
+
+# Remove everything (including volumes)
+docker-compose -f docker-compose.runners.yml down -v
+```
+
+## Workflow Configuration
+
+To use specific runners in your workflows, update the `runs-on` field:
+
+```yaml
+# .gitea/workflows/api-gateway.yml
+jobs:
+  test:
+    runs-on: [self-hosted, heavy]  # Uses heavy runner
+
+# .gitea/workflows/frontend.yml
+jobs:
+  test:
+    runs-on: [self-hosted, light]  # Uses light runner
+
+# .gitea/workflows/integration-tests.yml
+jobs:
+  integration:
+    runs-on: [self-hosted, docker]  # Uses docker runner
+```
+
+## Monitoring
+
+### Health Checks
+
+Each runner includes health checks that monitor:
+- Container status
+- HTTP health endpoint
+- Resource usage
+
+### Logs
+
+View logs for troubleshooting:
+
+```bash
+# All runners
+docker-compose -f docker-compose.runners.yml logs -f
+
+# Specific runner
+docker-compose -f docker-compose.runners.yml logs -f labfusion-runner-heavy
+```
+
+### Resource Monitoring
+
+```bash
+# Container resource usage
+docker stats
+
+# Volume usage
+docker system df -v
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Runners not registering / Token appears empty**
+   - **Check if `.env.runners` file exists** in the project root
+   - **Verify the file contains your actual token** (not the placeholder)
+   - **Make sure you're using the management scripts** (they include `--env-file` parameter)
+   - **If running Docker Compose directly**, add `--env-file .env.runners`:
+     ```bash
+     docker-compose -f docker-compose.runners.yml --env-file .env.runners up -d
+     ```
+
+2. **Runners not registering**
+   - Check Gitea instance URL and port
+   - Verify runner token is correct
+   - Ensure Gitea Actions is enabled
+
+3. **High resource usage**
+   - Adjust CPU/memory limits in docker-compose.runners.yml
+   - Check for stuck jobs in Gitea Actions
+
+4. **Docker socket issues**
+   - Ensure Docker socket is accessible: `/var/run/docker.sock`
+   - Check Docker daemon is running
+
+5. **Network connectivity**
+   - Verify runners can reach Gitea instance
+   - Check firewall settings
+
+### Debug Mode
+
+Enable debug logging by adding to `.env.runners`:
+
+```bash
+# Enable debug logging
+ACTIONS_RUNNER_DEBUG=1
+```
+
+### Reset Runners
+
+If runners become unresponsive:
+
+```bash
+# Stop and remove all runners
+docker-compose -f docker-compose.runners.yml down -v
+
+# Clean up volumes
+docker volume prune -f
+
+# Restart
+docker-compose -f docker-compose.runners.yml up -d
+```
+
+## Security Considerations
+
+- Runners run with Docker socket access for container builds
+- Each runner has isolated data volumes
+- Shared cache volume for build optimization
+- Resource limits prevent resource exhaustion
+- Health checks ensure runner availability
+
+## Scaling
+
+To add more runners:
+
+1. Copy a runner service in `docker-compose.runners.yml`
+2. Update the runner name and labels
+3. Adjust resource allocation
+4. Restart the compose stack
+
+Example additional runner:
+
+```yaml
+gitea-runner-additional:
+  image: gitea/act_runner:latest
+  container_name: labfusion-runner-additional
+  environment:
+    - GITEA_INSTANCE_URL=${GITEA_INSTANCE_URL}
+    - GITEA_RUNNER_REGISTRATION_TOKEN=${GITEA_RUNNER_TOKEN}
+    - GITEA_RUNNER_NAME=labfusion-runner-additional
+    - GITEA_RUNNER_LABELS=ubuntu-latest,self-hosted,additional
+  volumes:
+    - /var/run/docker.sock:/var/run/docker.sock
+    - runner-data-additional:/data
+    - shared-cache:/cache
+  restart: unless-stopped
+```
+
+## Support
+
+For issues with the runners setup:
+
+1. Check the troubleshooting section above
+2. Review runner logs
+3. Verify Gitea Actions configuration
+4. Check Docker and Docker Compose installation

docs/progress.md

@@ -217,6 +217,8 @@ The modular structure allows for easy addition of new services:
 - [x] Update flake8 line length limit to 150 characters for better readability
 - [x] Create JavaScript/Node.js tests for API docs service and frontend
 - [x] Fix Gitea Actions compatibility issues with artifact uploads
+- [x] Create Docker Compose setup for multiple Gitea runners
+- [x] Update all CI/CD pipelines with appropriate runner labels
 
 ## Resources
 - [Project Specifications](specs.md)

docs/structure.txt

@@ -97,8 +97,16 @@ labfusion/
 │   ├── Dockerfile.dev         # Development container
 │   ├── CLEAN_CODE.md          # Clean code documentation
 │   └── RESILIENCE.md          # Frontend resilience features
+# Docker Compose for Runners
+docker-compose.runners.yml   # Multi-runner Docker Compose setup
+env.runners.example          # Environment template for runners
+scripts/
+  manage-runners.sh          # Linux/macOS runner management script
+  manage-runners.ps1         # Windows PowerShell runner management script
+
 └── docs/                      # Documentation
     ├── specs.md              # Project specifications
     ├── structure.txt         # Project structure
     ├── progress.md           # Development progress tracking
-    └── CI_CD.md             # CI/CD pipeline documentation
+    ├── CI_CD.md             # CI/CD pipeline documentation
+    └── RUNNERS.md            # Gitea runners setup and management