Deployment Troubleshooting and Operations

Use this reference to diagnose operational errors, perform database health checks, and maintain long-term system stability.

متاح الآن. آخر مراجعة: 26 October 2026.

Automated Readiness Health Check

Aqiled includes a dedicated readiness health check at /health/ready (SingleSchoolReadinessHealthCheck.cs).

curl -i http://127.0.0.1:5216/health/ready

Response Codes

• HTTP 200 OK (Healthy): Database is connected, PostgreSQL 18 is validated, migrations are applied, and the school identity matches the deployment configuration.
• HTTP 503 Service Unavailable (Unhealthy): One or more dependencies failed startup checks. Inspect systemd logs for details:
  Copy

  sudo journalctl -u school-api -n 50 --no-pager
  

Startup Validation Errors

1. PostgreSQL Version Incompatibility

• Error: InvalidOperationException: DefaultConnection uses PostgreSQL 16.x. School.Api requires PostgreSQL 18.x.
• Cause: The database server is running an older major version. Aqiled pins PostgreSQL 18.x for modern exclusion constraints and RLS capabilities.
• Solution: Upgrade your database to PostgreSQL 18 using the official PGDG repository.

2. Dedicated School Identity Mismatch

• Error: InvalidOperationException: Dedicated school identity does not match deployment configuration.
• Cause: The Deployment:SchoolTenantId in /etc/school/api.env does not match the UUID in the deploymentidentity table, or multiple schools exist in the database.
• Solution:
  1. Inspect the marker in PostgreSQL:
     Copy

     SELECT schoolid FROM deploymentidentity WHERE id = 1;
     

  2. Ensure /etc/school/api.env has the exact matching UUID:
     Copy

     Deployment__SchoolTenantId=<MATCHING_UUID>
     

  3. Never delete the marker or switch Deployment:Mode back to MultiTenant to bypass this check.

3. Database Role & Row-Level Security Violations

• Error: InvalidOperationException: DefaultConnection must use a restricted role subject to tenant row security.
• Cause: DefaultConnection was configured with a superuser account, or an account with BYPASSRLS or table ownership.
• Solution: Verify your connection strings in /etc/school/api.env:
  • DefaultConnection must use school_app (NOBYPASSRLS).
  • AdminConnection must use school_admin (BYPASSRLS).
  • Neither connection may connect as the postgres superuser.

4. Missing Tenant RLS Policy

• Error: InvalidOperationException: A tenant table is missing its row security policy; apply and review the AppDb migration.
• Cause: A database table containing tenantid is missing the school_tenant_guard security policy.
• Solution: Re-run the AppDbContext migration bundle using the school_migrator user.

Networking & Runtime Issues

5. SignalR WebSocket Reconnect Loops

• Symptom: Parent notices, live chat, and attendance updates constantly fail with WebSocket connection errors in the browser console.
• Cause: Reverse proxy is stripping Upgrade and Connection headers or using HTTP/1.0.
• Solution: Verify Nginx configuration includes:
  Copy

  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  

6. Front-End API Calls Failing with 404

• Symptom: The login page renders, but clicking submit fails or sends requests to external domains.
• Cause: The front-end was built with standard npm run build instead of npm run build:dedicated.
• Solution: Rebuild front-end assets with:
  Copy

  cd front-end
  npm run build:dedicated
  sudo cp -r dist/dedicated/* /opt/school/web/current/
  

7. Background Jobs Triggering at the Wrong Hour

• Symptom: Daily Quran Hifz revision digests or data retention jobs fire during the school day rather than overnight.
• Cause: The host OS timezone is set to UTC, while the school operates in a different timezone (e.g. UTC+7).
• Solution:
  1. Set server timezone: sudo timedatectl set-timezone Asia/Bangkok.
  2. In the Aqiled School Admin dashboard, verify the school's configured timezone under System Settings.

Maintenance & Backups

Automated Daily Backup

Create a daily backup script at /usr/local/sbin/school-backup:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="/var/backups/school"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/school_db_${DATE}.dump"

# Dump database using pg_dump custom format
sudo -u postgres pg_dump -Fc -d school_db > "${BACKUP_FILE}"

# Retain backups for 14 days
find "${BACKUP_DIR}" -type f -name "*.dump" -mtime +14 -delete

Make executable and schedule in cron:

sudo chmod +x /usr/local/sbin/school-backup
# Add to /etc/cron.d/school-backup:
# 0 2 * * * root /usr/local/sbin/school-backup