> ## Documentation Index
> Fetch the complete documentation index at: https://docs.faithloft.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-host Faithloft

> Run Faithloft locally for development or deploy the current Kubernetes chart on your own infrastructure.

Use this guide to run Faithloft from source or operate your own deployment. The product is split across the `backend`, `web`, `admin`, `mobile`, and `landing` repositories. The backend resolves each church from the subdomain in the request host, so a hosted installation needs a base domain and wildcard DNS for church portals.

## Choose a setup

| Setup | What it runs | Where to start |
| - | - | - |
| Local development | PostgreSQL, PgBouncer, and Redis from Docker Compose; run the API and web apps from source | Follow [Run locally](#run-locally) |
| Self-hosted server | The checked-in k3s and Helm example deploys the API, Client Portal, Admin Console, landing site, PostgreSQL, Redis, ingress, and TLS | Follow [Deploy to a server](#deploy-to-a-server) and the runbook at `backend/deploy/k3s/README.md` |

The backend Compose file is for local dependencies. It does not start the API, Client Portal, or Admin Console. The Flutter mobile app is built separately and is not deployed by the Helm chart.

## Run locally

### Prerequisites

* Docker Engine with Docker Compose v2
* Node.js 22
* pnpm 9 for the backend
* npm for the web and admin applications
* OpenSSL for creating local signing secrets

### Start the backend

From the repository checkout, open a terminal in `backend`:

```bash theme={null}
cp .env.example .env
```

Use `.env.example` as a variable reference. Before starting the application, set separate random values for `JWT_ACCESS_SECRET` and `JWT_REFRESH_SECRET`, and replace the example platform and tenant seed passwords. These values are for local development; never reuse them on a public server. The backend needs `MASTER_DB_URL`, `TENANT_DB_URL`, `TENANT_DB_DIRECT_URL`, and `REDIS_URL`. The example file points these at the local Compose services.

Generate each signing key separately:

```bash theme={null}
openssl rand -hex 32
```

Start the local data services and prepare the demo church:

```bash theme={null}
docker compose up -d postgres pgbouncer redis
pnpm install --frozen-lockfile
pnpm migration:master:run
pnpm seed:master
pnpm run provision -- demo "Demo Church"
pnpm run seed:tenant -- demo
pnpm start:dev
```

`seed:master` creates the local platform administrator `admin@platform.local` and a demo organisation. Set `SEED_SUPERADMIN_PASSWORD` before running it. `provision` creates and migrates the demo tenant database; `seed:tenant demo` creates the passwordless local account `demo-admin@demo.local`. Without SMTP configured, sign-in codes are logged by the backend instead of emailed.

The Compose database credentials and exposed ports are for local development only. Do not expose the Compose services to the public internet or use the example passwords on a server. Use the `provision` command for a single local tenant and `migrate:tenants` to migrate existing tenant databases. Avoid running `migration:tenant:run` with its default target, which points to `tenant_demo`.

The API listens on port `3000`. In development, its health endpoint is `http://localhost:3000/health` and its Swagger UI is `http://localhost:3000/docs`.

### Start the Client Portal

In another terminal, open `web`:

```bash theme={null}
cp .env.example .env.local
npm ci
npm run dev
```

The example values point the portal at the `demo` tenant on `localhost`. Open `http://demo.localhost:3002`. The `.localhost` wildcard works in standard desktop browsers; for devices that cannot resolve it, use a local tunnel or a development domain that resolves to your computer.

### Start the Admin Console

In another terminal, open `admin`:

```bash theme={null}
cp .env.example .env.local
npm ci
npm run dev
```

The example API origin is `http://localhost:3000`. Open `http://localhost:3003` and sign in as `admin@platform.local` with the `SEED_SUPERADMIN_PASSWORD` you set before seeding.

The web and admin applications expose health checks at `/api/health` on their respective development ports.

### Run the mobile app

The mobile app is a separate Flutter build. Set its `BASE_DOMAIN`, `API_SCHEME`, `API_PORT`, `WEB_SCHEME`, and `WEB_PORT` Dart defines to match the API and portal addresses. The available settings are in `mobile/lib/config/api_config.dart`. Android emulators do not resolve subdomains of `localhost`; use a tunnel or a development domain that resolves from the device.

## Deploy to a server

This guide takes you from a blank Ubuntu VPS to a working Faithloft install on your own domain. It uses k3s, Helm, Traefik, CloudNativePG, and, depending on how traffic reaches the server, cert-manager or a Cloudflare Tunnel. It is an operations deployment, not the local Compose file. Replace `example.org` with your domain throughout.

The chart deploys these workloads:

* `faithloft-backend`: the API, autoscaled between 2 and 6 replicas with rolling updates.
* `faithloft-worker`: background jobs and schedulers, exactly 1 replica.
* The Client Portal, the Admin Console and the landing site, 2 replicas each.
* PostgreSQL (CloudNativePG) with 2 instances (a primary and an automatic-failover standby) and a pooler, and Redis.

The high-availability defaults are API 2, web 2, admin 2 and Postgres 2 instances. On a single server this protects against pod or process failure, not against the whole server going down.

Redis has no password, but a NetworkPolicy allows only the API and the worker to reach it.

### Requirements

* An Ubuntu 22.04 or newer VPS. This guide was tested on 8 vCPU and 11 GB RAM. 4 vCPU and 8 GB RAM is the suggested minimum; this is an estimate, not a tested floor.
* About 20 GB of disk for PostgreSQL, plus room for container images.
* A domain whose DNS is hosted on Cloudflare.
* Root or `sudo` access on the server.

### Choose how traffic reaches the server

| | Path A: public IP | Path B: Cloudflare Tunnel |
| - | - | - |
| Use when | The server has a public IP with TCP 80 and 443 open | The server is behind NAT, a home or office router, or a provider that blocks inbound ports |
| Open ports | 80 and 443 | None |
| TLS | Issued in the cluster by cert-manager | Terminates at Cloudflare |
| DNS records | `A` records, DNS only (grey cloud) | Proxied `CNAME` records created by `cloudflared` |

To check which path you need, run this from another network, not from the server:

```bash theme={null}
nc -vz <public-ip> 443
```

If it cannot connect, use Path B.

### Install k3s with the Traefik config

Write the Traefik configuration before you install k3s, so Traefik starts with it:

```bash theme={null}
sudo mkdir -p /var/lib/rancher/k3s/server/manifests
sudo tee /var/lib/rancher/k3s/server/manifests/traefik-config.yaml >/dev/null <<'YAML'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        http:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true
    service:
      spec:
        externalTrafficPolicy: Local
YAML
```

`externalTrafficPolicy: Local` preserves the real client IP, which the backend needs for throttling and lockout controls.

If another web server already holds ports 80 and 443, stop it first:

```bash theme={null}
sudo systemctl disable --now nginx
```

Install k3s and copy the kubeconfig:

```bash theme={null}
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v1.36.5+k3s1' K3S_KUBECONFIG_MODE=644 sh -
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
```

### Install Helm, cert-manager, CloudNativePG, and the barman plugin

Install the Helm CLI using the [official method](https://helm.sh/docs/intro/install/), then confirm it works:

```bash theme={null}
helm version
```

Install cert-manager with the recursive-nameserver flags. Without them, issuing one certificate for both the apex and the wildcard can fail with `NXDOMAIN looking up TXT for _acme-challenge`:

```bash theme={null}
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.2 \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --set "extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=1.1.1.1:53\,1.0.0.1:53}"
```

Path B does not need cert-manager, but installing it does no harm. You can set `certManager.enabled=false` in your values file instead.

Install CloudNativePG and the barman-cloud plugin:

```bash theme={null}
kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.1.yaml
kubectl -n cnpg-system rollout status deployment/cnpg-controller-manager
kubectl apply -f https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.15.1/manifest.yaml
```

### Get the container images

Choose one option.

**Option a: GitHub Container Registry.** Each repository's `.github/workflows/image.yml` builds an image on every push to `main`. Set the repository variables from the table below, then create a pull secret from a GitHub token with `read:packages`:

```bash theme={null}
kubectl create namespace faithloft
kubectl -n faithloft create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io \
  --docker-username='<github-username>' \
  --docker-password='<github-pat-with-read-packages>' \
  --docker-email='<github-email>'
```

**Option b: build on the server, no registry.** Install Docker, then for each of the `backend`, `web`, `admin`, and `landing` repositories, clone it and build the image with the build arguments below:

```bash theme={null}
curl -fsSL https://get.docker.com | sh
git clone <repository-url>
cd <repository>
sudo docker build <build-args> --build-arg SOURCE_COMMIT=<sha> -t <image>:<tag> .
sudo docker save <image>:<tag> | sudo k3s ctr -n k8s.io images import -
```

In your values file, set each image `repository` and `tag` to what you built, set `pullPolicy: Never`, and set `imagePullSecrets: []`.

Build-time settings for the Next.js apps are baked into the image. Set them as `--build-arg` options for option b, or as repository variables with the same names for option a:

| Image | Build arguments |
| - | - |
| Web | `NEXT_PUBLIC_API_BASE_DOMAIN=example.org`, `NEXT_PUBLIC_API_SCHEME=https`, `NEXT_PUBLIC_API_PORT=` (empty) |
| Admin | `NEXT_PUBLIC_API_ORIGIN=https://api.example.org` |
| Landing | `NEXT_PUBLIC_SITE_URL=https://example.org` |
| All images | `SOURCE_COMMIT=<sha>` |

If you change your domain later, rebuild the web, admin, and landing images. Changing Helm values does not rewrite them.

### Create the namespace and secrets

Create the namespace if you have not already:

```bash theme={null}
kubectl create namespace faithloft
```

Create the password for the `church` PostgreSQL role. Keep the `P` variable in your shell; the next step uses it.

```bash theme={null}
P="$(openssl rand -hex 24)"
kubectl -n faithloft create secret generic faithloft-pg-church \
  --type=kubernetes.io/basic-auth \
  --from-literal=username=church \
  --from-literal=password="$P"
kubectl -n faithloft label secret faithloft-pg-church cnpg.io/reload=true
```

If you open a new shell later, read the password back with `kubectl -n faithloft get secret faithloft-pg-church -o jsonpath='{.data.password}' | base64 -d`.

Generate the backend env file in one step. The script writes every required key with fresh secrets and keeps the file private. The image sets `NODE_ENV`, so it is not listed.

```bash theme={null}
mkdir -p ~/faithloft
umask 077
PORTAL_PROXY_SECRET="$(openssl rand -hex 32)"
ADMIN_PROXY_SECRET="$(openssl rand -hex 32)"
cat > ~/faithloft/backend.env <<ENV
# Base domain used to route church subdomains
BASE_DOMAIN=example.org
# Browser origins allowed to call the API
CORS_ORIGINS=https://example.org,https://www.example.org
# Platform database, direct connection
MASTER_DB_URL=postgresql://church:${P}@faithloft-pg-rw.faithloft.svc:5432/master
# Tenant requests, through the pooler
TENANT_DB_URL=postgresql://church:${P}@faithloft-pg-pooler-rw.faithloft.svc:5432/master
# Tenant provisioning and migrations, direct connection
TENANT_DB_DIRECT_URL=postgresql://church:${P}@faithloft-pg-rw.faithloft.svc:5432/master
# Redis for rate limits and background queues
REDIS_URL=redis://faithloft-redis.faithloft.svc:6379
# Signing secret for access tokens
JWT_ACCESS_SECRET=$(openssl rand -hex 32)
# Signing secret for refresh tokens
JWT_REFRESH_SECRET=$(openssl rand -hex 32)
# Pepper for hashing sign-in codes
LOGIN_CODE_PEPPER=$(openssl rand -hex 32)
# Encryption key for payment credentials
PAYMENTS_ENC_KEY=$(openssl rand -base64 32)
# Versioned encryption keys for pastoral notes
PASTORAL_NOTES_ENC_KEYS=v1:$(openssl rand -base64 32)
# Versioned encryption keys for children's notes
KIDS_NOTES_ENC_KEYS=v1:$(openssl rand -base64 32)
# Signs client-IP headers from the Client Portal
PORTAL_PROXY_SECRET=${PORTAL_PROXY_SECRET}
# Signs client-IP headers from the Admin Console
ADMIN_PROXY_SECRET=${ADMIN_PROXY_SECRET}
# Contact address required in production
CHILD_SAFETY_ESCALATION_EMAIL=safety@example.org
# Password for admin@platform.local
SEED_SUPERADMIN_PASSWORD=$(openssl rand -base64 18)
# Sender address for outgoing email
NOTIFICATIONS_EMAIL_FROM=noreply@example.org
ENV
```

<Warning>
  Back up `~/faithloft/backend.env` securely and never change the encryption keys (`PAYMENTS_ENC_KEY`, `PASTORAL_NOTES_ENC_KEYS`, `KIDS_NOTES_ENC_KEYS`). Losing or replacing them makes the encrypted data unreadable.
</Warning>

Create the application secrets. The `FAITHLOFT_PROXY_SECRET` in `faithloft-web` must equal the backend `PORTAL_PROXY_SECRET`, and the one in `faithloft-admin` must equal `ADMIN_PROXY_SECRET`:

```bash theme={null}
kubectl -n faithloft create secret generic faithloft-backend \
  --from-env-file="$HOME/faithloft/backend.env"
kubectl -n faithloft create secret generic faithloft-web \
  --from-literal=FAITHLOFT_PROXY_SECRET="$PORTAL_PROXY_SECRET"
kubectl -n faithloft create secret generic faithloft-admin \
  --from-literal=FAITHLOFT_PROXY_SECRET="$ADMIN_PROXY_SECRET"
```

For Path A, create a Cloudflare API token with the `Zone:DNS:Edit` and `Zone:Zone:Read` permissions for your zone. Store it in the `cert-manager` namespace:

```bash theme={null}
kubectl -n cert-manager create secret generic cloudflare-api-token \
  --from-literal=api-token='<cloudflare-api-token>'
```

### Configure DNS

**Path A: public IP.** In Cloudflare DNS, create these records and set both to **DNS only** (grey cloud):

| Type | Name | Content |
| - | - | - |
| `A` | `@` | `<public IP>` |
| `A` | `*` | `<public IP>` |

Proxying (orange cloud) would hide client IPs and conflict with the in-cluster TLS. Remove any parking-page `A` record and any `www` `CNAME`.

**Path B: Cloudflare Tunnel.** Remove any conflicting `A` records first. On your laptop, [install `cloudflared`](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/downloads/), then run:

```bash theme={null}
cloudflared tunnel login
cloudflared tunnel create faithloft
cloudflared tunnel route dns faithloft example.org
cloudflared tunnel route dns faithloft '*.example.org'
```

`tunnel create` prints the tunnel ID and writes `~/.cloudflared/<ID>.json`. The two `route dns` commands create proxied `CNAME` records that point to `<ID>.cfargotunnel.com`. Copy the JSON file to the server, then create the credentials secret:

```bash theme={null}
kubectl -n faithloft create secret generic cloudflared-credentials \
  --from-file=credentials.json=<file>
```

<Note>
  A tunnel created in the Zero Trust dashboard is remotely managed, and its dashboard routes override the chart's local routes. If you use the dashboard, add the same routes there, in this order:

  1. `example.org` to the landing site.
  2. `api` to the API.
  3. `*` with path `^/(v1|socket\.io)(/|$)` to the API.
  4. `admin` to the Admin Console.
  5. `*` to the Client Portal.
</Note>

Cloudflare Universal SSL covers the apex and `*.example.org`. Deeper subdomains need Advanced Certificate Manager.

### Add a dedicated edge node (optional)

Path B only. Run the tunnel connectors on a second, small node so that a connector survives the loss of either node. The cluster has two nodes: `main`, the k3s server that runs every workload, and `edge`, an agent that runs only tunnel connectors.

On the server, read the join token from `/var/lib/rancher/k3s/server/node-token`. On the edge machine, join it as an agent:

```bash theme={null}
curl -sfL https://get.k3s.io | K3S_URL=https://<server-private-ip>:6443 K3S_TOKEN=<token> INSTALL_K3S_VERSION='v1.36.5+k3s1' sh -
```

Label and taint the node so that only tolerating pods are scheduled on it:

```bash theme={null}
kubectl label node <edge> faithloft.org/role=edge
kubectl taint node <edge> faithloft.org/edge=true:NoSchedule
```

Add the toleration to the `tunnel` block of your values file:

```yaml theme={null}
tunnel:
  tolerations:
    - {key: faithloft.org/edge, operator: Equal, value: "true", effect: NoSchedule}
```

`spreadAcrossNodes` stays `true`, which places one connector per node. Replicas above the node count stay Pending.

Cloudflare balances traffic across the connectors on both nodes. If the edge node dies, the connector on the main node keeps serving. The edge node holds no data. If the main node dies, the site is down.

### Write the values file

Keep `~/faithloft/values.yaml` outside Git. Both paths need the base domain and the image tags. Set the image `repository` too if you use your own registry. For option b, also set `pullPolicy: Never` on each image and `imagePullSecrets: []`. Add `postgres.backup` now if you have a bucket (see [Backups](#backups)).

**Path A**, using the Let's Encrypt staging server for the first run:

```yaml theme={null}
baseDomain: example.org
certManager:
  email: ops@example.org
  acmeServer: https://acme-staging-v02.api.letsencrypt.org/directory
backend:
  image:
    tag: "<backend-sha>"
web:
  image:
    tag: "<web-sha>"
admin:
  image:
    tag: "<admin-sha>"
landing:
  image:
    tag: "<landing-sha>"
monitor:
  enabled: true
```

After the first run succeeds, delete the `acmeServer` line and delete the staging certificate secret so cert-manager issues a production certificate:

```bash theme={null}
kubectl -n faithloft delete secret faithloft-wildcard-tls
kubectl -n faithloft get certificate
```

**Path B**, using the tunnel:

```yaml theme={null}
baseDomain: example.org
certManager:
  enabled: false
tunnel:
  enabled: true
  tunnelId: "<ID>"
backend:
  image:
    tag: "<backend-sha>"
web:
  image:
    tag: "<web-sha>"
admin:
  image:
    tag: "<admin-sha>"
landing:
  image:
    tag: "<landing-sha>"
monitor:
  enabled: true
```

### Install and initialize

On a first install, run the migrations before the worker and API start. Run these commands from the `backend` checkout.

<Steps>
  <Step title="Install the chart">
    ```bash theme={null}
    helm install faithloft backend/deploy/helm/faithloft -n faithloft -f ~/faithloft/values.yaml
    ```

    Adjust the chart path if you are not in the directory that contains `backend`.
  </Step>

  <Step title="Wait for PostgreSQL">
    ```bash theme={null}
    kubectl -n faithloft get cluster faithloft-pg
    ```

    Wait until the cluster reports healthy. The API and worker pods restart until PostgreSQL is ready; this is expected.
  </Step>

  <Step title="Stop the worker and the API">
    ```bash theme={null}
    kubectl -n faithloft scale deploy/faithloft-worker --replicas=0
    kubectl -n faithloft wait --for=delete pod -l app.kubernetes.io/instance=faithloft,app.kubernetes.io/component=worker --timeout=120s
    kubectl -n faithloft scale deploy/faithloft-backend --replicas=0
    kubectl -n faithloft wait --for=delete pod -l app.kubernetes.io/instance=faithloft,app.kubernetes.io/component=backend --timeout=120s
    ```
  </Step>

  <Step title="Run the migrations">
    Use the backend image you installed. For images built on the server, add `--image-pull-policy=Never` and `"imagePullPolicy":"Never"` in the overrides, and drop the `imagePullSecrets`.

    ```bash theme={null}
    kubectl -n faithloft run faithloft-migration --rm -it --restart=Never \
      --override-type=strategic \
      --image=ghcr.io/emilio-kariuki/church-backend:<sha> \
      --overrides='{"spec":{"imagePullSecrets":[{"name":"ghcr-pull"}],"containers":[{"name":"faithloft-migration","image":"ghcr.io/emilio-kariuki/church-backend:<sha>","envFrom":[{"secretRef":{"name":"faithloft-backend"}}]}]}}' \
      --command -- sh -c 'pnpm migration:master:run && pnpm seed:master && pnpm migrate:tenants'
    ```
  </Step>

  <Step title="Clear Redis (first install only)">
    ```bash theme={null}
    kubectl -n faithloft exec faithloft-redis-0 -- redis-cli FLUSHALL
    ```

    Run this only on a first install with a new Redis volume. Never run it on a reinstall that reuses the Redis volume, because it deletes queued work.
  </Step>

  <Step title="Start the worker and the API">
    ```bash theme={null}
    kubectl -n faithloft scale deploy/faithloft-worker --replicas=1
    kubectl -n faithloft scale deploy/faithloft-backend --replicas=2
    ```

    With autoscaling on, 2 is the floor (`backend.autoscaling.minReplicas`). After this, the autoscaler takes over.
  </Step>
</Steps>

Within about a minute, the worker provisions the demo church database, `tenant_demo`. Confirm it in the worker logs:

```bash theme={null}
kubectl -n faithloft logs deploy/faithloft-worker | grep "Provisioned tenant database"
```

Never run `migration:tenant:run`. Its default target is `tenant_demo`.

### Verify

```bash theme={null}
kubectl -n faithloft get pods
kubectl -n faithloft get certificate
```

The `get certificate` check applies to Path A only. For Path A, run the checks below from outside your network.

| URL | Expected |
| - | - |
| `https://example.org` | 200 |
| `https://api.example.org/health/ready` | 200 |
| `https://admin.example.org/api/health` | 200 |
| `https://demo.example.org` | 307 to `/login` |
| `https://demo.example.org/v1/branding` | 200 |

### Monitoring and alerts

The chart can run a small monitor that pushes alerts to your phone through ntfy. Create the secret before you install, or run the install again after creating it. Pick a long random topic name, because anyone who knows it can read your alerts.

```bash theme={null}
kubectl -n faithloft create secret generic faithloft-monitor \
  --from-literal=NTFY_URL=https://ntfy.sh/<topic>
```

Install the ntfy app on your phone and subscribe to the same topic. With `monitor.enabled: true` in your values file, a "monitor started" message arrives after the install.

The monitor alerts when a container restarts or crash-loops, when a workload has fewer ready replicas than expected, when a node is NotReady, when Postgres is unhealthy or fails over, and when a health check fails twice in a row. It sends a recovery message when the problem clears.

A monitor inside the cluster cannot report that the server itself is down. To cover that, add a `HEARTBEAT_URL` key with a ping URL from a service such as healthchecks.io. The monitor pings it every minute, and the service alerts you through its ntfy, email or other integration when the pings stop.

### High availability (optional)

One server cannot survive its own loss. For real high availability, run 3 k3s servers with embedded etcd. A single server, or a server plus agents, has no API server after a failure, so nothing is rescheduled and Postgres cannot fail over.

To convert an existing single server, back up `/var/lib/rancher/k3s/server/db`, add `cluster-init: true` to `/etc/rancher/k3s/config.yaml`, and restart k3s. Remove any existing agent first with `k3s-agent-uninstall.sh` on that machine and `kubectl delete node <name>`. Then join each new server:

```bash theme={null}
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v1.36.5+k3s1' K3S_TOKEN=<token> sh -s - server --server https://<first-server-ip>:6443 --node-ip <this-node-ip>
```

If you use an edge node, remove its taint with `kubectl taint node <n> faithloft.org/edge-` once all 3 servers should share the workloads. Recommended counts:

| Component | Count |
| - | - |
| API | `backend.autoscaling.minReplicas: 3` (one per node) |
| web, admin, landing | 2 each |
| pooler | 2 |
| Postgres | `postgres.instances: 3`, spread across nodes by default |
| Redis | 1 (a node loss means a brief queue and rate-limit hiccup) |
| tunnel connectors | 3 |
| worker | always 1 |

Volumes are tied to their node, so Redis only returns when its node does. Each Postgres instance has its own volume and fails over by promotion.

### Sign in for the first time

* Admin Console: `https://admin.example.org`. Sign in as `admin@platform.local` with the `SEED_SUPERADMIN_PASSWORD` from your env file.
* Church portal: `https://demo.example.org`.

Without SMTP configured, sign-in codes are only logged:

```bash theme={null}
kubectl -n faithloft logs deploy/faithloft-backend --all-containers | grep -i code
```

Create your real churches in the Admin Console. The worker provisions each church database automatically.

### Autoscaling the API

The API autoscales on CPU. By default it keeps at least 2 pods (`backend.autoscaling.minReplicas`) and grows to at most 6 (`maxReplicas`) when average CPU passes 70% of the 500m request. The worker is never autoscaled. A new pod takes 30 to 60 seconds to become Ready, so a sudden spike is absorbed by the always-on floor first.

Measured on 2026-10-10, one API pod handled about 1,500 concurrent active users (2 to 8 seconds of think time) at about 0.9 core. Plan about 1 core of node CPU per pod. Watch it with:

```bash theme={null}
kubectl -n faithloft get hpa
```

If you turn autoscaling on for a release that already runs a fixed count, the API briefly drops to 1 pod (about 20 seconds) until the autoscaler restores the floor. Make that switch outside peak hours.

Each API pod can hold up to 150 pooler connections, so the pooler needs room for `(maxReplicas + 1 surge + 1 worker) × 150` clients. The default `max_client_conn` of 2000 covers a `maxReplicas` up to 11; raise `postgres.pooler.maxClientConn` before going higher.

### Deploy updates with zero downtime

1. Put the new image tags in your values file.

2. Stop the worker and wait for it to go:

   ```bash theme={null}
   kubectl -n faithloft scale deploy/faithloft-worker --replicas=0
   kubectl -n faithloft wait --for=delete pod -l app.kubernetes.io/instance=faithloft,app.kubernetes.io/component=worker --timeout=120s
   ```

3. Run `pnpm migration:master:run` and then `pnpm migrate:tenants` from the temporary migration pod, using the new image tag. The old API keeps serving meanwhile.

4. Upgrade the chart. The API rolls with no downtime and the worker returns on the new image:

   ```bash theme={null}
   helm upgrade --install faithloft backend/deploy/helm/faithloft -n faithloft -f ~/faithloft/values.yaml
   ```

5. Check the result and confirm the API READY count is between `minReplicas` and `maxReplicas` and the worker is `1/1`:

   ```bash theme={null}
   kubectl -n faithloft get deploy faithloft-backend faithloft-worker
   ```

   `helm upgrade` restores the worker but not an API you scaled to 0, because the autoscaled chart omits `replicas`. In that case run `kubectl -n faithloft scale deploy/faithloft-backend --replicas=2` (your `minReplicas`) and the autoscaler resumes.

Migrations in a zero-downtime release may only add things: nullable or defaulted columns, new tables, and new indexes. The old API runs against the new schema while it rolls, so dropping or renaming something the old code reads must wait for a later release (expand now, contract later). Background job payloads follow the same rule, because old API pods keep enqueueing jobs that the new worker processes. For a release that cannot follow this rule, use the breaking-release path in `backend/deploy/k3s/README.md`.

To roll back, run `helm rollback faithloft`. This does not roll back the database schema.

### Automatic deploys (optional)

On a single k3s server without an image registry, `backend/deploy/k3s/autodeploy/` provides a systemd timer that runs every 2 minutes. It fetches `main` of the backend, client, admin, and landing repositories. When something changed it builds the images on the host and imports them into k3s. For a backend change it then stops the worker, runs the migrations, runs `helm upgrade`, checks every rollout, and sends progress and failure notifications to ntfy.

Setup in short:

1. Create one read-only deploy key per repository with `gh repo deploy-key add <key>.pub --repo <owner>/<repo>`, add a `~/.ssh/config` alias for each (use `ssh.github.com` on port 443 if outbound port 22 is blocked), and clone the four repositories through the aliases.
2. Install the script with `sudo install -m 755 faithloft-autodeploy.sh /usr/local/bin/faithloft-autodeploy`, copy `autodeploy.env.example` to `/etc/faithloft-autodeploy.env` with mode 600 and edit it, copy the two systemd units to `/etc/systemd/system`, and run `sudo systemctl enable --now faithloft-autodeploy.timer`.
3. Use `faithloft-autodeploy --status`, `--force all`, and `--dry-run` to inspect or force a run, and `journalctl -u faithloft-autodeploy` for logs.

The first run has no recorded state and deploys all four apps. Migrations must follow the expand/contract rule above; put `[deploy:breaking]` in a backend commit message to stop the API as well during that release. A failed deploy sends a high-priority ntfy message and does not roll back automatically: run `helm rollback faithloft -n faithloft` to recover. A failed deploy is attempted once: the same commits are not retried until any repository gets a new commit, or you run `faithloft-autodeploy --force all` after fixing the problem. If Coolify still runs, merging to `main` also deploys Coolify production. The full runbook is in `backend/deploy/k3s/README.md`.

### Backups

Configure backups before you store real data.

1. Create an S3-compatible bucket.

2. Create the credentials secret:

   ```bash theme={null}
   kubectl -n faithloft create secret generic <name> \
     --from-literal=ACCESS_KEY_ID='<access-key-id>' \
     --from-literal=ACCESS_SECRET_KEY='<access-secret-key>'
   ```

3. Set the `postgres.backup` values in `~/faithloft/values.yaml`:

   ```yaml theme={null}
   postgres:
     backup:
       enabled: true
       destinationPath: s3://<bucket>/backups
       endpointURL: https://<s3-endpoint>
       s3CredentialsSecret: <name>
   ```

4. Run `helm upgrade`, take an on-demand backup, and complete the restore drill. The exact manifests are in section 12 of `backend/deploy/k3s/README.md`.

Back up `~/faithloft/backend.env` separately and securely. It holds the encryption keys that the database backups depend on.

### Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| The site does not load from outside but works on the server | Inbound ports are blocked by NAT or your provider | Use Path B, the Cloudflare Tunnel |
| The certificate is stuck, or shows `NXDOMAIN looking up TXT _acme-challenge` | cert-manager is missing the recursive-nameserver flags | Reinstall cert-manager with the `extraArgs` flags, delete the Certificate, then run `helm upgrade` again |
| Port 80 is already in use when installing k3s | Another web server is running | Stop it, for example `sudo systemctl disable --now nginx` |
| Backend or worker in `CrashLoopBackOff` with `ECONNREFUSED 5432` right after install | PostgreSQL is not ready yet | Wait until `kubectl -n faithloft get cluster faithloft-pg` is healthy |
| A web pod times out when it reaches Redis | Expected: the NetworkPolicy allows only the API and the worker | No action needed |
| Client IPs in rate limits are Cloudflare addresses (Path A) | DNS records are proxied (orange cloud) | Set the `A` records to DNS only (grey cloud) |
| The Admin Console still links to `faithloft.org` | Known gap: `NEXT_PUBLIC_TENANT_HOST` is not a Dockerfile build argument yet | None yet; this is recorded in the runbook |

### Reference

* `backend/deploy/k3s/README.md` is the full runbook. It covers cutover from Coolify, Redis state, landmines, and the breaking-release path.
* `backend/deploy/helm/faithloft/values.yaml` lists every chart value.
* `backend/src/config/env.validation.ts` lists every backend environment key.

## Optional integrations

The core app can start without every external provider, but related features remain limited until you configure one:

| Capability | Configuration | Without it |
| - | - | - |
| Email | SMTP settings such as `SMTP_HOST`, `SMTP_USER`, and `SMTP_PASS` | Messages render and log, but are not delivered |
| File uploads | S3-compatible storage settings such as `S3_BUCKET`, credentials, and public URL | Upload endpoints report that storage is unavailable |
| Push notifications | Firebase service account or credentials file | Push sends are logged instead of delivered |
| AI drafting | OpenAI-compatible `AI_BASE_URL` and `AI_API_KEY` | The backend uses its configured offline/manual fallback |
| Google sign-in | Backend `GOOGLE_OAUTH_CLIENT_ID`, `GOOGLE_OAUTH_CLIENT_SECRET`, `GOOGLE_OAUTH_PUBLIC_BASE_URL` (the API's public HTTPS origin) and `PAYMENTS_ENC_KEY`; register one redirect URI, `<api origin>/v1/auth/google/callback`. Mobile still uses `GOOGLE_CLIENT_IDS` | Web Google sign-in reports that it is not configured |

Use `backend/.env.example` and `backend/src/config/env.validation.ts` as the authoritative environment variable reference. Client build-time settings are listed in each app's `.env.example` or configuration source.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.