Self-host Faithloft

Use this guide to run Faithloft from source or operate your own deployment. The product is split across the backend, web, admin, mobile, and landing repositories. The backend resolves each church from the subdomain in the request host, so a hosted installation needs a base domain and wildcard DNS for church portals.

Choose a setup

Setup	What it runs	Where to start
Local development	PostgreSQL, PgBouncer, and Redis from Docker Compose; run the API and web apps from source	Follow Run locally
Self-hosted server	The checked-in k3s and Helm example deploys the API, Client Portal, Admin Console, landing site, PostgreSQL, Redis, ingress, and TLS	Follow Deploy to a server and the runbook at `backend/deploy/k3s/README.md`

The backend Compose file is for local dependencies. It does not start the API, Client Portal, or Admin Console. The Flutter mobile app is built separately and is not deployed by the Helm chart.

Run locally

Prerequisites

Docker Engine with Docker Compose v2
Node.js 22
pnpm 9 for the backend
npm for the web and admin applications
OpenSSL for creating local signing secrets

Start the backend

From the repository checkout, open a terminal in backend:

cp .env.example .env

Use .env.example as a variable reference. Before starting the application, set separate random values for JWT_ACCESS_SECRET and JWT_REFRESH_SECRET, and replace the example platform and tenant seed passwords. These values are for local development; never reuse them on a public server. The backend needs MASTER_DB_URL, TENANT_DB_URL, TENANT_DB_DIRECT_URL, and REDIS_URL. The example file points these at the local Compose services. Generate each signing key separately:

openssl rand -hex 32

Start the local data services and prepare the demo church:

docker compose up -d postgres pgbouncer redis
pnpm install --frozen-lockfile
pnpm migration:master:run
pnpm seed:master
pnpm run provision -- demo "Demo Church"
pnpm run seed:tenant -- demo
pnpm start:dev

seed:master creates the local platform administrator [email protected] and a demo organisation. Set SEED_SUPERADMIN_PASSWORD before running it. provision creates and migrates the demo tenant database; seed:tenant demo creates the passwordless local account [email protected]. Without SMTP configured, sign-in codes are logged by the backend instead of emailed. The Compose database credentials and exposed ports are for local development only. Do not expose the Compose services to the public internet or use the example passwords on a server. Use the provision command for a single local tenant and migrate:tenants to migrate existing tenant databases. Avoid running migration:tenant:run with its default target, which points to tenant_demo. The API listens on port 3000. In development, its health endpoint is http://localhost:3000/health and its Swagger UI is http://localhost:3000/docs.

Start the Client Portal

In another terminal, open web:

cp .env.example .env.local
npm ci
npm run dev

The example values point the portal at the demo tenant on localhost. Open http://demo.localhost:3002. The .localhost wildcard works in standard desktop browsers; for devices that cannot resolve it, use a local tunnel or a development domain that resolves to your computer.

Start the Admin Console

In another terminal, open admin:

cp .env.example .env.local
npm ci
npm run dev

The example API origin is http://localhost:3000. Open http://localhost:3003 and sign in as [email protected] with the SEED_SUPERADMIN_PASSWORD you set before seeding. The web and admin applications expose health checks at /api/health on their respective development ports.

Run the mobile app

The mobile app is a separate Flutter build. Set its BASE_DOMAIN, API_SCHEME, API_PORT, WEB_SCHEME, and WEB_PORT Dart defines to match the API and portal addresses. The available settings are in mobile/lib/config/api_config.dart. Android emulators do not resolve subdomains of localhost; use a tunnel or a development domain that resolves from the device.

Deploy to a server

This guide takes you from a blank Ubuntu VPS to a working Faithloft install on your own domain. It uses k3s, Helm, Traefik, CloudNativePG, and, depending on how traffic reaches the server, cert-manager or a Cloudflare Tunnel. It is an operations deployment, not the local Compose file. Replace example.org with your domain throughout. The chart deploys these workloads:

faithloft-backend: the API, autoscaled between 2 and 6 replicas with rolling updates.
faithloft-worker: background jobs and schedulers, exactly 1 replica.
The Client Portal, the Admin Console and the landing site, 2 replicas each.
PostgreSQL (CloudNativePG) with 2 instances (a primary and an automatic-failover standby) and a pooler, and Redis.

The high-availability defaults are API 2, web 2, admin 2 and Postgres 2 instances. On a single server this protects against pod or process failure, not against the whole server going down. Redis has no password, but a NetworkPolicy allows only the API and the worker to reach it.

Requirements

An Ubuntu 22.04 or newer VPS. This guide was tested on 8 vCPU and 11 GB RAM. 4 vCPU and 8 GB RAM is the suggested minimum; this is an estimate, not a tested floor.
About 20 GB of disk for PostgreSQL, plus room for container images.
A domain whose DNS is hosted on Cloudflare.
Root or sudo access on the server.

Choose how traffic reaches the server

	Path A: public IP	Path B: Cloudflare Tunnel
Use when	The server has a public IP with TCP 80 and 443 open	The server is behind NAT, a home or office router, or a provider that blocks inbound ports
Open ports	80 and 443	None
TLS	Issued in the cluster by cert-manager	Terminates at Cloudflare
DNS records	`A` records, DNS only (grey cloud)	Proxied `CNAME` records created by `cloudflared`

To check which path you need, run this from another network, not from the server:

nc -vz <public-ip> 443

If it cannot connect, use Path B.

Install k3s with the Traefik config

Write the Traefik configuration before you install k3s, so Traefik starts with it:

sudo mkdir -p /var/lib/rancher/k3s/server/manifests
sudo tee /var/lib/rancher/k3s/server/manifests/traefik-config.yaml >/dev/null <<'YAML'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        http:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true
    service:
      spec:
        externalTrafficPolicy: Local
YAML

externalTrafficPolicy: Local preserves the real client IP, which the backend needs for throttling and lockout controls. If another web server already holds ports 80 and 443, stop it first:

sudo systemctl disable --now nginx

Install k3s and copy the kubeconfig:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v1.36.5+k3s1' K3S_KUBECONFIG_MODE=644 sh -
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Install Helm, cert-manager, CloudNativePG, and the barman plugin

Install the Helm CLI using the official method, then confirm it works:

helm version

Install cert-manager with the recursive-nameserver flags. Without them, issuing one certificate for both the apex and the wildcard can fail with NXDOMAIN looking up TXT for _acme-challenge:

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.2 \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --set "extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=1.1.1.1:53\,1.0.0.1:53}"

Path B does not need cert-manager, but installing it does no harm. You can set certManager.enabled=false in your values file instead. Install CloudNativePG and the barman-cloud plugin:

kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.1.yaml
kubectl -n cnpg-system rollout status deployment/cnpg-controller-manager
kubectl apply -f https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.15.1/manifest.yaml

Get the container images

Choose one option. Option a: GitHub Container Registry. Each repository’s .github/workflows/image.yml builds an image on every push to main. Set the repository variables from the table below, then create a pull secret from a GitHub token with read:packages:

kubectl create namespace faithloft
kubectl -n faithloft create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io \
  --docker-username='<github-username>' \
  --docker-password='<github-pat-with-read-packages>' \
  --docker-email='<github-email>'

Option b: build on the server, no registry. Install Docker, then for each of the backend, web, admin, and landing repositories, clone it and build the image with the build arguments below:

curl -fsSL https://get.docker.com | sh
git clone <repository-url>
cd <repository>
sudo docker build <build-args> --build-arg SOURCE_COMMIT=<sha> -t <image>:<tag> .
sudo docker save <image>:<tag> | sudo k3s ctr -n k8s.io images import -

In your values file, set each image repository and tag to what you built, set pullPolicy: Never, and set imagePullSecrets: []. Build-time settings for the Next.js apps are baked into the image. Set them as --build-arg options for option b, or as repository variables with the same names for option a:

Image	Build arguments
Web	`NEXT_PUBLIC_API_BASE_DOMAIN=example.org`, `NEXT_PUBLIC_API_SCHEME=https`, `NEXT_PUBLIC_API_PORT=` (empty)
Admin	`NEXT_PUBLIC_API_ORIGIN=https://api.example.org`
Landing	`NEXT_PUBLIC_SITE_URL=https://example.org`
All images	`SOURCE_COMMIT=<sha>`

If you change your domain later, rebuild the web, admin, and landing images. Changing Helm values does not rewrite them.

Create the namespace and secrets

Create the namespace if you have not already:

kubectl create namespace faithloft

Create the password for the church PostgreSQL role. Keep the P variable in your shell; the next step uses it.

P="$(openssl rand -hex 24)"
kubectl -n faithloft create secret generic faithloft-pg-church \
  --type=kubernetes.io/basic-auth \
  --from-literal=username=church \
  --from-literal=password="$P"
kubectl -n faithloft label secret faithloft-pg-church cnpg.io/reload=true

If you open a new shell later, read the password back with kubectl -n faithloft get secret faithloft-pg-church -o jsonpath='{.data.password}' | base64 -d. Generate the backend env file in one step. The script writes every required key with fresh secrets and keeps the file private. The image sets NODE_ENV, so it is not listed.

mkdir -p ~/faithloft
umask 077
PORTAL_PROXY_SECRET="$(openssl rand -hex 32)"
ADMIN_PROXY_SECRET="$(openssl rand -hex 32)"
cat > ~/faithloft/backend.env <<ENV
# Base domain used to route church subdomains
BASE_DOMAIN=example.org
# Browser origins allowed to call the API
CORS_ORIGINS=https://example.org,https://www.example.org
# Platform database, direct connection
MASTER_DB_URL=postgresql://church:${P}@faithloft-pg-rw.faithloft.svc:5432/master
# Tenant requests, through the pooler
TENANT_DB_URL=postgresql://church:${P}@faithloft-pg-pooler-rw.faithloft.svc:5432/master
# Tenant provisioning and migrations, direct connection
TENANT_DB_DIRECT_URL=postgresql://church:${P}@faithloft-pg-rw.faithloft.svc:5432/master
# Redis for rate limits and background queues
REDIS_URL=redis://faithloft-redis.faithloft.svc:6379
# Signing secret for access tokens
JWT_ACCESS_SECRET=$(openssl rand -hex 32)
# Signing secret for refresh tokens
JWT_REFRESH_SECRET=$(openssl rand -hex 32)
# Pepper for hashing sign-in codes
LOGIN_CODE_PEPPER=$(openssl rand -hex 32)
# Encryption key for payment credentials
PAYMENTS_ENC_KEY=$(openssl rand -base64 32)
# Versioned encryption keys for pastoral notes
PASTORAL_NOTES_ENC_KEYS=v1:$(openssl rand -base64 32)
# Versioned encryption keys for children's notes
KIDS_NOTES_ENC_KEYS=v1:$(openssl rand -base64 32)
# Signs client-IP headers from the Client Portal
PORTAL_PROXY_SECRET=${PORTAL_PROXY_SECRET}
# Signs client-IP headers from the Admin Console
ADMIN_PROXY_SECRET=${ADMIN_PROXY_SECRET}
# Contact address required in production
[email protected]
# Password for [email protected]
SEED_SUPERADMIN_PASSWORD=$(openssl rand -base64 18)
# Sender address for outgoing email
[email protected]
ENV

Back up ~/faithloft/backend.env securely and never change the encryption keys (PAYMENTS_ENC_KEY, PASTORAL_NOTES_ENC_KEYS, KIDS_NOTES_ENC_KEYS). Losing or replacing them makes the encrypted data unreadable.

Create the application secrets. The FAITHLOFT_PROXY_SECRET in faithloft-web must equal the backend PORTAL_PROXY_SECRET, and the one in faithloft-admin must equal ADMIN_PROXY_SECRET:

kubectl -n faithloft create secret generic faithloft-backend \
  --from-env-file="$HOME/faithloft/backend.env"
kubectl -n faithloft create secret generic faithloft-web \
  --from-literal=FAITHLOFT_PROXY_SECRET="$PORTAL_PROXY_SECRET"
kubectl -n faithloft create secret generic faithloft-admin \
  --from-literal=FAITHLOFT_PROXY_SECRET="$ADMIN_PROXY_SECRET"

For Path A, create a Cloudflare API token with the Zone:DNS:Edit and Zone:Zone:Read permissions for your zone. Store it in the cert-manager namespace:

kubectl -n cert-manager create secret generic cloudflare-api-token \
  --from-literal=api-token='<cloudflare-api-token>'

Configure DNS

Path A: public IP. In Cloudflare DNS, create these records and set both to DNS only (grey cloud):

Type	Name	Content
`A`	`@`	`<public IP>`
`A`	`*`	`<public IP>`

Proxying (orange cloud) would hide client IPs and conflict with the in-cluster TLS. Remove any parking-page A record and any www CNAME. Path B: Cloudflare Tunnel. Remove any conflicting A records first. On your laptop, install cloudflared, then run:

cloudflared tunnel login
cloudflared tunnel create faithloft
cloudflared tunnel route dns faithloft example.org
cloudflared tunnel route dns faithloft '*.example.org'

tunnel create prints the tunnel ID and writes ~/.cloudflared/<ID>.json. The two route dns commands create proxied CNAME records that point to <ID>.cfargotunnel.com. Copy the JSON file to the server, then create the credentials secret:

kubectl -n faithloft create secret generic cloudflared-credentials \
  --from-file=credentials.json=<file>

A tunnel created in the Zero Trust dashboard is remotely managed, and its dashboard routes override the chart’s local routes. If you use the dashboard, add the same routes there, in this order:

example.org to the landing site.
api to the API.
* with path ^/(v1|socket\.io)(/|$) to the API.
admin to the Admin Console.
* to the Client Portal.

Cloudflare Universal SSL covers the apex and *.example.org. Deeper subdomains need Advanced Certificate Manager.

Add a dedicated edge node (optional)

Path B only. Run the tunnel connectors on a second, small node so that a connector survives the loss of either node. The cluster has two nodes: main, the k3s server that runs every workload, and edge, an agent that runs only tunnel connectors. On the server, read the join token from /var/lib/rancher/k3s/server/node-token. On the edge machine, join it as an agent:

curl -sfL https://get.k3s.io | K3S_URL=https://<server-private-ip>:6443 K3S_TOKEN=<token> INSTALL_K3S_VERSION='v1.36.5+k3s1' sh -

Label and taint the node so that only tolerating pods are scheduled on it:

kubectl label node <edge> faithloft.org/role=edge
kubectl taint node <edge> faithloft.org/edge=true:NoSchedule

Add the toleration to the tunnel block of your values file:

tunnel:
  tolerations:
    - {key: faithloft.org/edge, operator: Equal, value: "true", effect: NoSchedule}

spreadAcrossNodes stays true, which places one connector per node. Replicas above the node count stay Pending. Cloudflare balances traffic across the connectors on both nodes. If the edge node dies, the connector on the main node keeps serving. The edge node holds no data. If the main node dies, the site is down.

Write the values file

Keep ~/faithloft/values.yaml outside Git. Both paths need the base domain and the image tags. Set the image repository too if you use your own registry. For option b, also set pullPolicy: Never on each image and imagePullSecrets: []. Add postgres.backup now if you have a bucket (see Backups). Path A, using the Let’s Encrypt staging server for the first run:

baseDomain: example.org
certManager:
  email: [email protected]
  acmeServer: https://acme-staging-v02.api.letsencrypt.org/directory
backend:
  image:
    tag: "<backend-sha>"
web:
  image:
    tag: "<web-sha>"
admin:
  image:
    tag: "<admin-sha>"
landing:
  image:
    tag: "<landing-sha>"
monitor:
  enabled: true

After the first run succeeds, delete the acmeServer line and delete the staging certificate secret so cert-manager issues a production certificate:

kubectl -n faithloft delete secret faithloft-wildcard-tls
kubectl -n faithloft get certificate

Path B, using the tunnel:

baseDomain: example.org
certManager:
  enabled: false
tunnel:
  enabled: true
  tunnelId: "<ID>"
backend:
  image:
    tag: "<backend-sha>"
web:
  image:
    tag: "<web-sha>"
admin:
  image:
    tag: "<admin-sha>"
landing:
  image:
    tag: "<landing-sha>"
monitor:
  enabled: true

Install and initialize

On a first install, run the migrations before the worker and API start. Run these commands from the backend checkout.

Install the chart

helm install faithloft backend/deploy/helm/faithloft -n faithloft -f ~/faithloft/values.yaml

Adjust the chart path if you are not in the directory that contains backend.

Wait for PostgreSQL

kubectl -n faithloft get cluster faithloft-pg

Wait until the cluster reports healthy. The API and worker pods restart until PostgreSQL is ready; this is expected.

Stop the worker and the API

kubectl -n faithloft scale deploy/faithloft-worker --replicas=0
kubectl -n faithloft wait --for=delete pod -l app.kubernetes.io/instance=faithloft,app.kubernetes.io/component=worker --timeout=120s
kubectl -n faithloft scale deploy/faithloft-backend --replicas=0
kubectl -n faithloft wait --for=delete pod -l app.kubernetes.io/instance=faithloft,app.kubernetes.io/component=backend --timeout=120s

Run the migrations

Use the backend image you installed. For images built on the server, add --image-pull-policy=Never and "imagePullPolicy":"Never" in the overrides, and drop the imagePullSecrets.

kubectl -n faithloft run faithloft-migration --rm -it --restart=Never \
  --override-type=strategic \
  --image=ghcr.io/emilio-kariuki/church-backend:<sha> \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"ghcr-pull"}],"containers":[{"name":"faithloft-migration","image":"ghcr.io/emilio-kariuki/church-backend:<sha>","envFrom":[{"secretRef":{"name":"faithloft-backend"}}]}]}}' \
  --command -- sh -c 'pnpm migration:master:run && pnpm seed:master && pnpm migrate:tenants'

Clear Redis (first install only)

kubectl -n faithloft exec faithloft-redis-0 -- redis-cli FLUSHALL

Run this only on a first install with a new Redis volume. Never run it on a reinstall that reuses the Redis volume, because it deletes queued work.

Start the worker and the API

kubectl -n faithloft scale deploy/faithloft-worker --replicas=1
kubectl -n faithloft scale deploy/faithloft-backend --replicas=2

With autoscaling on, 2 is the floor (backend.autoscaling.minReplicas). After this, the autoscaler takes over.

Within about a minute, the worker provisions the demo church database, tenant_demo. Confirm it in the worker logs:

kubectl -n faithloft logs deploy/faithloft-worker | grep "Provisioned tenant database"

Never run migration:tenant:run. Its default target is tenant_demo.

Verify

kubectl -n faithloft get pods
kubectl -n faithloft get certificate

The get certificate check applies to Path A only. For Path A, run the checks below from outside your network.

URL	Expected
`https://example.org`	200
`https://api.example.org/health/ready`	200
`https://admin.example.org/api/health`	200
`https://demo.example.org`	307 to `/login`
`https://demo.example.org/v1/branding`	200

Monitoring and alerts

The chart can run a small monitor that pushes alerts to your phone through ntfy. Create the secret before you install, or run the install again after creating it. Pick a long random topic name, because anyone who knows it can read your alerts.

kubectl -n faithloft create secret generic faithloft-monitor \
  --from-literal=NTFY_URL=https://ntfy.sh/<topic>

Install the ntfy app on your phone and subscribe to the same topic. With monitor.enabled: true in your values file, a “monitor started” message arrives after the install. The monitor alerts when a container restarts or crash-loops, when a workload has fewer ready replicas than expected, when a node is NotReady, when Postgres is unhealthy or fails over, and when a health check fails twice in a row. It sends a recovery message when the problem clears. A monitor inside the cluster cannot report that the server itself is down. To cover that, add a HEARTBEAT_URL key with a ping URL from a service such as healthchecks.io. The monitor pings it every minute, and the service alerts you through its ntfy, email or other integration when the pings stop.

High availability (optional)

One server cannot survive its own loss. For real high availability, run 3 k3s servers with embedded etcd. A single server, or a server plus agents, has no API server after a failure, so nothing is rescheduled and Postgres cannot fail over. To convert an existing single server, back up /var/lib/rancher/k3s/server/db, add cluster-init: true to /etc/rancher/k3s/config.yaml, and restart k3s. Remove any existing agent first with k3s-agent-uninstall.sh on that machine and kubectl delete node <name>. Then join each new server:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v1.36.5+k3s1' K3S_TOKEN=<token> sh -s - server --server https://<first-server-ip>:6443 --node-ip <this-node-ip>

If you use an edge node, remove its taint with kubectl taint node <n> faithloft.org/edge- once all 3 servers should share the workloads. Recommended counts:

Component	Count
API	`backend.autoscaling.minReplicas: 3` (one per node)
web, admin, landing	2 each
pooler	2
Postgres	`postgres.instances: 3`, spread across nodes by default
Redis	1 (a node loss means a brief queue and rate-limit hiccup)
tunnel connectors	3
worker	always 1

Volumes are tied to their node, so Redis only returns when its node does. Each Postgres instance has its own volume and fails over by promotion.

Admin Console: https://admin.example.org. Sign in as [email protected] with the SEED_SUPERADMIN_PASSWORD from your env file.
Church portal: https://demo.example.org.

Without SMTP configured, sign-in codes are only logged:

kubectl -n faithloft logs deploy/faithloft-backend --all-containers | grep -i code

Create your real churches in the Admin Console. The worker provisions each church database automatically.

Autoscaling the API

The API autoscales on CPU. By default it keeps at least 2 pods (backend.autoscaling.minReplicas) and grows to at most 6 (maxReplicas) when average CPU passes 70% of the 500m request. The worker is never autoscaled. A new pod takes 30 to 60 seconds to become Ready, so a sudden spike is absorbed by the always-on floor first. Measured on 2026-10-10, one API pod handled about 1,500 concurrent active users (2 to 8 seconds of think time) at about 0.9 core. Plan about 1 core of node CPU per pod. Watch it with:

kubectl -n faithloft get hpa

If you turn autoscaling on for a release that already runs a fixed count, the API briefly drops to 1 pod (about 20 seconds) until the autoscaler restores the floor. Make that switch outside peak hours. Each API pod can hold up to 150 pooler connections, so the pooler needs room for (maxReplicas + 1 surge + 1 worker) × 150 clients. The default max_client_conn of 2000 covers a maxReplicas up to 11; raise postgres.pooler.maxClientConn before going higher.

Deploy updates with zero downtime

Put the new image tags in your values file.

Stop the worker and wait for it to go:

kubectl -n faithloft scale deploy/faithloft-worker --replicas=0
kubectl -n faithloft wait --for=delete pod -l app.kubernetes.io/instance=faithloft,app.kubernetes.io/component=worker --timeout=120s

Run pnpm migration:master:run and then pnpm migrate:tenants from the temporary migration pod, using the new image tag. The old API keeps serving meanwhile.

Upgrade the chart. The API rolls with no downtime and the worker returns on the new image:

helm upgrade --install faithloft backend/deploy/helm/faithloft -n faithloft -f ~/faithloft/values.yaml

Check the result and confirm the API READY count is between minReplicas and maxReplicas and the worker is 1/1:
kubectl -n faithloft get deploy faithloft-backend faithloft-worker
helm upgrade restores the worker but not an API you scaled to 0, because the autoscaled chart omits replicas. In that case run kubectl -n faithloft scale deploy/faithloft-backend --replicas=2 (your minReplicas) and the autoscaler resumes.

Migrations in a zero-downtime release may only add things: nullable or defaulted columns, new tables, and new indexes. The old API runs against the new schema while it rolls, so dropping or renaming something the old code reads must wait for a later release (expand now, contract later). Background job payloads follow the same rule, because old API pods keep enqueueing jobs that the new worker processes. For a release that cannot follow this rule, use the breaking-release path in backend/deploy/k3s/README.md. To roll back, run helm rollback faithloft. This does not roll back the database schema.

Automatic deploys (optional)

On a single k3s server without an image registry, backend/deploy/k3s/autodeploy/ provides a systemd timer that runs every 2 minutes. It fetches main of the backend, client, admin, and landing repositories. When something changed it builds the images on the host and imports them into k3s. For a backend change it then stops the worker, runs the migrations, runs helm upgrade, checks every rollout, and sends progress and failure notifications to ntfy. Setup in short:

Create one read-only deploy key per repository with gh repo deploy-key add <key>.pub --repo <owner>/<repo>, add a ~/.ssh/config alias for each (use ssh.github.com on port 443 if outbound port 22 is blocked), and clone the four repositories through the aliases.
Install the script with sudo install -m 755 faithloft-autodeploy.sh /usr/local/bin/faithloft-autodeploy, copy autodeploy.env.example to /etc/faithloft-autodeploy.env with mode 600 and edit it, copy the two systemd units to /etc/systemd/system, and run sudo systemctl enable --now faithloft-autodeploy.timer.
Use faithloft-autodeploy --status, --force all, and --dry-run to inspect or force a run, and journalctl -u faithloft-autodeploy for logs.

The first run has no recorded state and deploys all four apps. Migrations must follow the expand/contract rule above; put [deploy:breaking] in a backend commit message to stop the API as well during that release. A failed deploy sends a high-priority ntfy message and does not roll back automatically: run helm rollback faithloft -n faithloft to recover. A failed deploy is attempted once: the same commits are not retried until any repository gets a new commit, or you run faithloft-autodeploy --force all after fixing the problem. If Coolify still runs, merging to main also deploys Coolify production. The full runbook is in backend/deploy/k3s/README.md.

Backups

Configure backups before you store real data.

Create an S3-compatible bucket.

Create the credentials secret:

kubectl -n faithloft create secret generic <name> \
  --from-literal=ACCESS_KEY_ID='<access-key-id>' \
  --from-literal=ACCESS_SECRET_KEY='<access-secret-key>'

Set the postgres.backup values in ~/faithloft/values.yaml:

postgres:
  backup:
    enabled: true
    destinationPath: s3://<bucket>/backups
    endpointURL: https://<s3-endpoint>
    s3CredentialsSecret: <name>

Run helm upgrade, take an on-demand backup, and complete the restore drill. The exact manifests are in section 12 of backend/deploy/k3s/README.md.

Back up ~/faithloft/backend.env separately and securely. It holds the encryption keys that the database backups depend on.

Troubleshooting

Symptom	Cause	Fix
The site does not load from outside but works on the server	Inbound ports are blocked by NAT or your provider	Use Path B, the Cloudflare Tunnel
The certificate is stuck, or shows `NXDOMAIN looking up TXT _acme-challenge`	cert-manager is missing the recursive-nameserver flags	Reinstall cert-manager with the `extraArgs` flags, delete the Certificate, then run `helm upgrade` again
Port 80 is already in use when installing k3s	Another web server is running	Stop it, for example `sudo systemctl disable --now nginx`
Backend or worker in `CrashLoopBackOff` with `ECONNREFUSED 5432` right after install	PostgreSQL is not ready yet	Wait until `kubectl -n faithloft get cluster faithloft-pg` is healthy
A web pod times out when it reaches Redis	Expected: the NetworkPolicy allows only the API and the worker	No action needed
Client IPs in rate limits are Cloudflare addresses (Path A)	DNS records are proxied (orange cloud)	Set the `A` records to DNS only (grey cloud)
The Admin Console still links to `faithloft.org`	Known gap: `NEXT_PUBLIC_TENANT_HOST` is not a Dockerfile build argument yet	None yet; this is recorded in the runbook

Reference

backend/deploy/k3s/README.md is the full runbook. It covers cutover from Coolify, Redis state, landmines, and the breaking-release path.
backend/deploy/helm/faithloft/values.yaml lists every chart value.
backend/src/config/env.validation.ts lists every backend environment key.

Optional integrations

The core app can start without every external provider, but related features remain limited until you configure one:

Capability	Configuration	Without it
Email	SMTP settings such as `SMTP_HOST`, `SMTP_USER`, and `SMTP_PASS`	Messages render and log, but are not delivered
File uploads	S3-compatible storage settings such as `S3_BUCKET`, credentials, and public URL	Upload endpoints report that storage is unavailable
Push notifications	Firebase service account or credentials file	Push sends are logged instead of delivered
AI drafting	OpenAI-compatible `AI_BASE_URL` and `AI_API_KEY`	The backend uses its configured offline/manual fallback
Google sign-in	Backend `GOOGLE_OAUTH_CLIENT_ID`, `GOOGLE_OAUTH_CLIENT_SECRET`, `GOOGLE_OAUTH_PUBLIC_BASE_URL` (the API’s public HTTPS origin) and `PAYMENTS_ENC_KEY`; register one redirect URI, `<api origin>/v1/auth/google/callback`. Mobile still uses `GOOGLE_CLIENT_IDS`	Web Google sign-in reports that it is not configured

Use backend/.env.example and backend/src/config/env.validation.ts as the authoritative environment variable reference. Client build-time settings are listed in each app’s .env.example or configuration source.

Start here

Staff portal

Integrations

Departments

Developers

Choose a setup

Run locally

Prerequisites

Start the backend

Start the Client Portal

Start the Admin Console

Run the mobile app

Deploy to a server

Requirements

Choose how traffic reaches the server

Install k3s with the Traefik config

Install Helm, cert-manager, CloudNativePG, and the barman plugin

Get the container images

Create the namespace and secrets

Configure DNS

Add a dedicated edge node (optional)

Write the values file

Install and initialize

Verify

Monitoring and alerts

High availability (optional)

Autoscaling the API

Deploy updates with zero downtime

Automatic deploys (optional)

Backups

Troubleshooting

Reference

Optional integrations

​Choose a setup

​Run locally

​Prerequisites

​Start the backend

​Start the Client Portal

​Start the Admin Console

​Run the mobile app

​Deploy to a server

​Requirements

​Choose how traffic reaches the server

​Install k3s with the Traefik config

​Install Helm, cert-manager, CloudNativePG, and the barman plugin

​Get the container images

​Create the namespace and secrets

​Configure DNS

​Add a dedicated edge node (optional)

​Write the values file

​Install and initialize

​Verify

​Monitoring and alerts

​High availability (optional)

​Sign in for the first time

​Autoscaling the API

​Deploy updates with zero downtime

​Automatic deploys (optional)

​Backups

​Troubleshooting

​Reference

​Optional integrations

Choose a setup

Run locally

Prerequisites

Start the backend

Start the Client Portal

Start the Admin Console

Run the mobile app

Deploy to a server

Requirements

Choose how traffic reaches the server

Install k3s with the Traefik config

Install Helm, cert-manager, CloudNativePG, and the barman plugin

Get the container images

Create the namespace and secrets

Configure DNS

Add a dedicated edge node (optional)

Write the values file

Install and initialize

Verify

Monitoring and alerts

High availability (optional)

Sign in for the first time

Autoscaling the API

Deploy updates with zero downtime

Automatic deploys (optional)

Backups

Troubleshooting

Reference

Optional integrations