From e93492b5ea8afbe64da5c4e952d3911f9e07aa0d Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Mon, 14 Sep 2026 09:17:38 -0600 Subject: [PATCH 1/7] docs(learn): add Administration production reliability track Ten Learn > Administration guides derived from Jeff's "Operating Reliable Harper Applications" field guide, drafted by Aleks Haugom with Claude and moved into the repo largely as delivered. Replaces the Administration "Coming Soon" placeholder and redirects its old URL to the first guide. Guides, in sidebar order: 1. How Harper Runs in Production 2. Health Checks and Traffic Admission 3. Sizing a Harper Cluster 4. Operating Replication 5. Monitoring and Triage 6. Safe Deployments and Rollback 7. Backup and Recovery 8. Engineering RPO, RTO, and Uptime 9. Production Readiness Checklist 10. Reliability Plan Template Only content change from the delivered drafts: the get_components link in guide 1 pointed at an anchor that does not exist on the operations page; it now points at components/applications#get_components. Co-Authored-By: Aleks Haugom Co-Authored-By: Claude Fable 5.1 --- learn/administration/backup-and-recovery.mdx | 340 +++++++++++++++++ learn/administration/coming-soon.md | 1 - .../engineering-rpo-rto-and-uptime.mdx | 224 ++++++++++++ .../health-checks-and-traffic-admission.mdx | 314 ++++++++++++++++ .../how-harper-runs-in-production.mdx | 273 ++++++++++++++ .../administration/monitoring-and-triage.mdx | 293 +++++++++++++++ .../administration/operating-replication.mdx | 342 ++++++++++++++++++ .../production-readiness-checklist.mdx | 230 ++++++++++++ .../reliability-plan-template.mdx | 199 ++++++++++ .../safe-deployments-and-rollback.mdx | 328 +++++++++++++++++ .../sizing-a-harper-cluster.mdx | 261 +++++++++++++ redirects.ts | 1 + 12 files changed, 2805 insertions(+), 1 deletion(-) create mode 100644 learn/administration/backup-and-recovery.mdx delete mode 100644 learn/administration/coming-soon.md create mode 100644 learn/administration/engineering-rpo-rto-and-uptime.mdx create mode 100644 learn/administration/health-checks-and-traffic-admission.mdx create mode 100644 learn/administration/how-harper-runs-in-production.mdx create mode 100644 learn/administration/monitoring-and-triage.mdx create mode 100644 learn/administration/operating-replication.mdx create mode 100644 learn/administration/production-readiness-checklist.mdx create mode 100644 learn/administration/reliability-plan-template.mdx create mode 100644 learn/administration/safe-deployments-and-rollback.mdx create mode 100644 learn/administration/sizing-a-harper-cluster.mdx diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx new file mode 100644 index 000000000..54bfa9f88 --- /dev/null +++ b/learn/administration/backup-and-recovery.mdx @@ -0,0 +1,340 @@ +--- +title: Backup and Recovery +sidebar_position: 7 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Replication gives you availability. It does not give you recovery, and the reason +is worth stating plainly: replication faithfully propagates whatever you write, +including the bad write. A destructive operation, a schema mistake, or a release +that corrupted data reaches every peer at the speed of your convergence time. + +Backups cover the failure classes replication cannot. This guide is about +deciding what you need before you pick an operation, getting a copy somewhere the +node's failure cannot reach, and knowing your restore constraints before you are +in an incident rather than during one. + +## What You Will Learn + +- The five questions that determine your backup design, answered before any + command +- Which backup mechanism you actually have, since it depends on your storage + engine +- Why a copy on the node that created it is not a backup, and the specific way + copying one wrong breaks it +- What `verify_backup` does and does not check +- Which databases can be restored with the server running, and which cannot +- Why restoring a database to undo a code bug is usually the wrong move + +## Prerequisites + +- A cluster with a `super_user` credential, and CLI access to at least one node +- Knowledge of your storage engine per database, from your service boundary + inventory in + [How Harper Runs in Production](./how-harper-runs-in-production.mdx) +- [Operating Replication](./operating-replication.mdx), because a restore in a + replicated cluster is a replication event as much as a storage one +- A non-production database you are willing to destroy + +## Decide before you choose a command + +Answer these first. Every operation below is easy, and every one of them is the +wrong choice for some of these answers. + +1. **Which failure classes must you survive?** Node loss, storage loss, a bad + deployment, an accidental destructive operation, a corrupting application bug, + a site or region event, and a control-plane or credential loss are seven + different problems. Replication addresses the first one well and the sixth one + partially. It addresses none of the rest. +2. **What is the recovery point and recovery time per database?** Not per cluster. + A catalog table and an order ledger rarely deserve the same answer, and paying + ledger-grade backup cadence for catalog data is how backup cost becomes a + reason to reduce frequency. +3. **What granularity do you need?** Harper backs up and restores whole databases. + There is no per-table restore. If your recovery story requires restoring one + table, that requirement has to be met by database layout or by forward repair, + and it is far cheaper to learn that now. +4. **Where must copies live?** How many independent locations, under what + retention, and with what access control. +5. **Who is authorized to execute a restore, and how is that logged?** A restore + destroys current data by design. It deserves a named authority and an audit + trail. + +## Know which mechanism you have + +| Mechanism | Engines | What it is | +| ----------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| **Managed backups** | | Incremental, verifiable backups in a server-side repository under `storage.backupPath` | +| **Snapshot download** | | [`get_backup`](/reference/v5/backups/operations#get_backup) streams a snapshot over HTTP, with no server-side artifact | + +A backup is a whole-database copy: all tables, the transaction log, and any +file-backed blobs. So a restored database keeps its `read_audit_log` history as of +the backup point. + +Two limitations to check against your own deployment now rather than later: + +- **Managed backups require RocksDB.** For LMDB databases your options are + `get_backup` or volume snapshots. +- **One storage root per database.** A database whose tables use per-table `path` + storage configs spans multiple root stores and cannot be backed up with these + operations at all. Database-level custom storage paths are fine. If someone has + configured per-table paths, your backup strategy for that database does not + exist yet and you need to know that before launch. + +## Create and retain managed backups + + + + +```json +{ + "operation": "create_backup", + "database": "data" +} +``` + +Through a running server this returns a `job_id` immediately. Poll +[`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, +which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result +as the completion signal, not the original response. + + + + +```bash +harper create_backup database=data +harper list_backups database=data +harper purge_backups database=data keep_count=7 +``` + +Every backup operation runs from the CLI under the same name. With the server +running the CLI forwards to the server; with it stopped the command operates +directly on the files. + + + + +Backups of the same database share unchanged RocksDB data files, so the first one +copies everything and later ones copy only what changed. Shared files are +reference-counted, so deleting a backup removes only files no remaining backup +references. + +:::warning +The incremental behavior applies to the RocksDB data files only. **The +transaction-log snapshot, and the blob snapshot for a database with file-backed +blobs, are copied in full on every backup.** With a large audit-retention window +or many blobs, frequent backups cost considerably more disk than the +data-only view suggests. Pass `exclude_blobs: true` to skip blobs when that is +appropriate, and size your backup volume against the full cost rather than the +incremental one. +::: + +Do not use `list_backups` sizes for capacity planning. The `size` and +`file_count` fields come from the RocksDB backup engine and exclude the +transaction-log and blob snapshots, so each entry undercounts, while the shared +files between entries mean summing them overcounts. Measure the repository +directory instead. + +## Get a copy off the node + +This is the step most likely to be missing, and the one where doing it slightly +wrong produces a copy that cannot be restored. + +**Managed backups live on the node that created them.** The repository is a local +directory, and RocksDB shares files across backup IDs, so **a backup ID is not a +self-contained folder.** Two consequences: + +- A disaster-recovery copy has to take the entire per-database repository, + `/`, not an individual backup. +- It has to do that while no backup operation is running, or from an atomic + filesystem snapshot. A live recursive copy can race `create_backup`, + `delete_backup`, or `purge_backups` and produce an unrestorable copy. + +An unrestorable copy is worse than no copy, because it will pass a "backups exist" +check and fail during an incident. + +The simpler off-host path, and the one to prefer unless you specifically need +retained managed backups off the node: + +```bash +# Pull a snapshot of the current state from a running node +harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz + +# Or pull from another node, which also clones that node's database onto this one +harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz +``` + +Note that `get_backup` always streams the current state. It cannot download a +historical managed backup, so it is a way to take a fresh off-host copy, not a way +to export your retention history. + +Whichever path you choose, the destination must not share a failure domain with +the source. A second directory on the same volume survives a deleted file and +nothing else. + +## Verification is not optional + +```json +{ + "operation": "verify_backup", + "database": "data", + "backup_id": 1, + "verify_checksum": true +} +``` + +`verify_backup` checks the RocksDB file sizes, their checksums when +`verify_checksum` is `true`, which is slower, and the framing of the +transaction-log snapshot, which is always checked. + +**The blob snapshot is not verified.** If your database has file-backed blobs, +verification does not tell you they are intact, and only a real restore does. + +More generally, a verified backup is a well-formed backup, not a proven recovery. +The only evidence that your recovery works is a restore you have actually +performed, which is why the drill at the end of this guide is the point of it. + +## Know your restore constraints before the incident + +RocksDB is single-writer, so an in-place restore requires the database to be fully +closed first. That produces hard constraints you cannot negotiate during an +outage: + +| Database | Online restore, server running | Offline restore, server stopped | +| ---------------------------------------------- | ------------------------------------ | ------------------------------- | +| A user database no loaded component holds open | Yes, restored in place | Yes | +| A user database a loaded component holds open | No, the job ends in `ERROR` | Yes | +| The `system` database | No, rejected before a job is created | Yes | + +Because the restore runs as a background job, a component holding the database +open does not fail your request. It fails inside the job, so you find out from +`get_job` rather than from the response. Harper does not track which component +uses which database, so it cannot selectively stop one. + +Two more constraints worth writing into your procedure: + +- **`target_database` requires the server stopped.** Restoring into a separate + database rather than overwriting the source is CLI-only with Harper down. The + target must not already exist or must be an empty directory. +- **An interrupted restore leaves the database unloadable.** On a crash or power + loss mid-restore, Harper marks the database as incompletely restored and refuses + to load it on the next start. Recover by rerunning `restore_backup` for the same + database and `backup_id`. Do not try to load or hand-repair the directory. + +## Restoring in a replicated cluster + +A restore is a point-in-time rollback of one node's data. In a cluster, that node +then has to rejoin peers that never rolled back, so sequence it deliberately: + +1. Take the node out of rotation with the availability flag, per + [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + Never restore a node that is serving traffic. +2. Decide what should happen to replication for the restored database, and do that + deliberately. Whether the restored data should propagate, or be overwritten by + peers, is a decision with two very different outcomes, and it is not a decision + to improvise. +3. Perform the restore, online or offline according to the table above. +4. Verify data expectations locally before any peer sees the node. +5. Re-admit the node through the full return sequence, including convergence + verification. + +## What backups cannot fix + +If a release changed the meaning of persisted data, restoring the database is +usually the wrong response. A restore rolls back every write in the window, +including all the correct ones from the same period, so you can violate your +recovery point objective in the course of fixing a code bug. + +For that failure class, define forward repair instead: a targeted correction, or a +replay from the transaction log, that fixes the affected records and leaves the +rest alone. Decide which of your failure classes get restore and which get forward +repair while you are calm, and record the decision. See +[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this +interacts with release reversal. + +### Prove it + +A backup you have never restored is a hypothesis. Run this on a non-production +cluster and record the numbers: + +1. Create a managed backup, then verify it with `verify_checksum: true`. +2. Take an off-host copy using the correct procedure for your mechanism, quiesced + or from an atomic snapshot if you are copying a managed repository. +3. Destroy the source database. +4. Restore it, taking the node out of rotation first, and time from decision to + restored service. That is your measured recovery time. +5. Determine how much data was actually lost against the backup timestamp. That is + your measured recovery point. +6. Validate correctness through the application, not just at the storage layer. + Row counts agreeing is not the same as the journey working. +7. Repeat for the `system` database specifically, offline, since it has different + constraints and it is the one people never rehearse. + +Both measured numbers feed +[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they +do not meet your stated targets, one of the two has to change, and it is better +that it is the target than a promise you cannot keep. + +## Operational notes + +- **Retention is a policy, not a side effect of disk space.** Use + `purge_backups` with an explicit `keep_count` that matches a written retention + decision. +- **Backup operations are `super_user` through the server, and filesystem + permissions offline.** An operator with shell access on a node can restore + without an API credential, so protect the host accordingly. +- **Schedule backups per database, matched to that database's recovery point.** + A single cluster-wide cadence overspends on some databases and underspends on + the ones that matter. +- **Alert on backup age, and on job failure.** A `create_backup` job that fails + silently produces a gap you will find at the worst possible time. Backup age + exceeding your recovery point objective is a pageable condition. +- **Record the storage engine per database in your inventory** and re-check after + migrations, since your entire mechanism choice depends on it. + +## Readiness checklist + +- [ ] Failure classes enumerated, with the mechanism that addresses each +- [ ] Recovery point and recovery time stated per database, not per cluster +- [ ] Storage engine confirmed per database, and the mechanism chosen accordingly +- [ ] No database in scope uses per-table storage paths, or its exclusion is known + and accepted +- [ ] Backup cadence matches the stated recovery point per database +- [ ] Backup volume sized against the full cost, including non-incremental + transaction-log and blob snapshots +- [ ] An off-host copy exists in a destination that does not share a failure + domain +- [ ] Managed repository copies take the whole `/` directory, + quiesced or from an atomic snapshot +- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least + periodically +- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Restore constraints documented for user databases, component-held databases, + and `system` +- [ ] Restore authority named, and restore execution logged +- [ ] Restore drill completed and dated, with measured recovery time and recovery + point +- [ ] `system` database restore rehearsed offline +- [ ] Forward repair defined for corruption caused by application code + +## Additional Resources + +- [Backups overview](/reference/v5/backups/overview) for how managed backups work, + the full limitation list, and manual restore examples +- [Backup operations](/reference/v5/backups/operations) for every parameter of + `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, + `purge_backups`, `restore_backup`, and `get_backup` +- [Storage configuration](/reference/v5/configuration/options#storage) for + `storage.backupPath`, `storage.path`, and `storage.blobPaths` +- [Jobs](/reference/v5/operations-api/operations#jobs) and + [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking + long-running backup operations +- [CLI operations](/reference/v5/cli/operations-api-commands) and + [remote operations](/reference/v5/cli/overview#remote-operations) for the + offline and `target=` forms +- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for + transaction history usable in forward repair +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for + turning the measured numbers into commitments diff --git a/learn/administration/coming-soon.md b/learn/administration/coming-soon.md deleted file mode 100644 index d30962bdc..000000000 --- a/learn/administration/coming-soon.md +++ /dev/null @@ -1 +0,0 @@ -# Coming Soon diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx new file mode 100644 index 000000000..958c625d5 --- /dev/null +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -0,0 +1,224 @@ +--- +title: Engineering RPO, RTO, and Uptime +sidebar_position: 8 +--- + +Recovery point, recovery time, and availability are usually written down once, in +a document nobody consults, as three numbers that were never checked against a +mechanism. This guide is about making them real: stating them per journey, +mapping each failure scenario to the mechanism that addresses it, and reconciling +the targets against numbers you have actually measured. + +If a target and a measurement disagree, one of them has to change. The point of +doing this deliberately is that you get to choose which, rather than finding out +during an incident that the answer was the target all along. + +## What You Will Learn + +- How to state recovery point, recovery time, and availability so they are + testable rather than aspirational +- A scenario-to-mechanism map, with what has to be proven for each to count +- How to convert an availability target into a monthly budget, and what each tier + demands of your automation +- How to reconcile stated targets against the numbers measured in earlier guides +- Why declaring your degraded modes is as important as declaring your targets + +## Prerequisites + +- Measured numbers from the earlier guides: per-node capacity, convergence time, + node re-entry time, release reversal time, and restore time +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity model +- [Backup and Recovery](./backup-and-recovery.mdx) for the recovery mechanisms +- A named owner who can approve or reject a target, because this exercise produces + decisions rather than data + +## State the three per journey + +| Term | The question it answers | How to state it so it is testable | +| ------------------------ | ------------------------------------ | --------------------------------------------------------------------------- | +| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | +| Recovery time objective | How long until service is restored | Per journey, measured from detection to verified service, not from decision | +| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | + +Three details make the difference between a testable statement and a slogan. + +**Per journey, not per cluster.** A catalog browse and an order submission fail +differently, matter differently, and recover differently. One cluster-wide number +overcommits on the cheap journey and undercommits on the expensive one. + +**Recovery time starts at detection, not at decision.** The gap between something +breaking and someone knowing is part of the outage, and it is often the largest +part. Measuring from the moment a human decided to act produces a number that +flatters your automation and misleads your planning. + +**Name the measurement boundary.** Availability measured at your CDN edge and +availability measured at a Harper node are different quantities, and the +difference is exactly the part of the stack you may not control. Whichever you +choose, say so in the same sentence as the number. + +## Map scenarios to mechanisms + +For each scenario, know the mechanism, and know what must be proven for the +mechanism to count. The right-hand column is the whole exercise. + +| Scenario | Primary mechanism | What must be proven | +| ------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | +| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | +| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | +| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | +| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | +| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | +| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | +| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | +| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | +| Corruption propagated by replication | Point-in-time restore plus a replication decision | A rehearsed sequence that includes what happens to peers | +| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | +| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | + +Two rows in that table are the ones most often missing. Node replacement is +budgeted as though it were a restart, when it is a full synchronization and can be +much slower. And the downstream dependency row usually has no answer at all, which +means the answer is whatever the code happens to do. + +## Convert availability into a budget + +An availability target is a quantity of unavailability you may spend per month. + +| Monthly target | Approximate maximum unavailability | What the tier demands | +| -------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| 99.9% | 43 min 50 sec | A human can be paged, diagnose, and act within some incidents. One untested maintenance window can consume the month | +| 99.95% | 21 min 55 sec | Automated traffic removal and a rehearsed release reversal become necessary rather than nice | +| 99.99% | 4 min 23 sec | Node failure has to be close to transparent. Detection, traffic removal, and validation must be automated, since no human response fits | + +Figures assume a 30.44 day month. Your contractual definition, exclusions, and +measurement boundary may differ. + +Read down that right-hand column rather than the middle one. The tier you choose +determines how much automation you are committing to build, not how many nodes you +buy. Adding nodes without automating detection and traffic removal moves you +across the table's rows without moving you down its column. + +Treat the budget as spendable. If the month's remaining budget is small, that is +an argument for pausing discretionary change, and if it is large, that is +permission to ship. An error budget that never changes anyone's behavior is +just a number in a document. + +## Reconcile targets against measurements + +This is the step that turns the exercise into engineering. Fill in both columns +and look for rows where they disagree. + +| Commitment | Measured from | Your target | Your measurement | +| --------------------------- | ---------------------------------------------------------------------------------------- | ----------- | ---------------- | +| Surviving capacity | The peak-load drain drill in [Sizing](./sizing-a-harper-cluster.mdx) | | | +| Time to remove traffic | The drain sequence in [Health Checks](./health-checks-and-traffic-admission.mdx) | | | +| Node re-entry time | The same drill, including convergence | | | +| Convergence time under load | The sentinel measurement in [Operating Replication](./operating-replication.mdx) | | | +| Full synchronization time | A node replacement at your data volume | | | +| Release reversal time | The stopped-rollout rehearsal in [Safe Deployments](./safe-deployments-and-rollback.mdx) | | | +| Restore time | The restore drill in [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Actual data loss on restore | The same drill, against the backup timestamp | | | +| Time to detection | The fault injection drill in [Monitoring and Triage](./monitoring-and-triage.mdx) | | | + +Three reconciliations catch most problems: + +- **Backup cadence against recovery point.** If you back up every six hours, your + recovery point cannot be one hour, no matter what the document says. +- **Restore time plus detection time against recovery time.** Recovery time + includes noticing. A twenty minute restore behind a forty minute detection gap + is a one hour recovery. +- **Convergence time against your admission gate.** If your gate admits a node + faster than it converges, you are serving stale data on purpose and calling it + availability. + +Where a target and a measurement disagree, the resolution is one of three things: +invest in the mechanism, relax the target, or accept the gap explicitly with an +owner and a date. All three are legitimate. Leaving it unreconciled is not. + +## Declare your degraded modes + +Between fully working and fully down there is a range of states, and if you have +not decided what they should be, the code has decided for you. + +For each critical journey, write down what happens when a dependency is +unavailable, when a node is behind on replication, and when the cluster is below +its capacity budget. Reads served from slightly stale data may be entirely +acceptable for a catalog and entirely unacceptable for a balance check, and the +right answer differs by journey rather than by system. + +Then decide who can declare a degraded mode, and whether the declaration is +manual or automatic. A degraded mode nobody is authorized to invoke is not a +degraded mode. + +### Prove it + +Two passes, and both are needed. + +**Tabletop the whole scenario table.** For each row, walk through who detects it, +what they do, what mechanism carries the recovery, and what evidence confirms +success. Rows where the group cannot answer without speculating are the gaps, and +finding them costs an hour rather than an outage. + +**Then run one real drill per scenario class**, spread over a quarter rather than +attempted in a day. The classes are: node loss, node return, node replacement, +release reversal, restore, and dependency failure. Record the date, the measured +numbers, what surprised you, and what you changed as a result. + +The surprises are the deliverable. A drill that goes exactly as expected has +confirmed your documentation. A drill that does not has found the thing that +would have hurt you. + +## Operational notes + +- **Recovery numbers expire.** Every measurement here is a property of a Harper + version, a data volume, a topology, and a set of components. Re-measure after + version upgrades and significant data growth, and date every number you record. +- **Uptime and recovery targets belong in the same document as the mechanism.** A + target stored separately from its supporting evidence drifts from reality within + a quarter. +- **Do not let a single incident rewrite your targets.** Adjust targets from + measurement and business need, not from the last thing that went wrong. +- **Exclusions matter as much as the number.** Planned maintenance, third-party + dependency failures, and client-side problems are usually excluded from an + availability calculation. Whether yours excludes them changes the number + substantially, so it belongs in writing. +- **A target you cannot measure is not a target.** If nothing in your monitoring + produces the number in your commitment, the first investment is measurement, not + more nodes. + +## Readiness checklist + +- [ ] Recovery point stated per database, in time units +- [ ] Recovery time stated per journey, measured from detection +- [ ] Availability stated per journey, with the measurement boundary named +- [ ] Exclusions documented +- [ ] Every scenario in the map has a named mechanism +- [ ] Every mechanism has its "what must be proven" satisfied or explicitly + outstanding +- [ ] Availability target converted to minutes per month +- [ ] Error budget policy states what changes when the budget runs low +- [ ] Backup cadence reconciled against recovery point +- [ ] Restore time plus detection time reconciled against recovery time +- [ ] Convergence time reconciled against the traffic admission gate +- [ ] Every unreconciled gap has an owner and a date +- [ ] Degraded modes declared per journey, with authority to invoke them +- [ ] Tabletop completed and dated +- [ ] At least one real drill per scenario class completed and dated + +## Additional Resources + +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity + invariant behind the surviving-capacity commitment +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) + for drain, admission, and failback timing +- [Operating Replication](./operating-replication.mdx) for convergence measurement +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release + reversal timing +- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and + mechanism limits +- [Monitoring and Triage](./monitoring-and-triage.mdx) for time to detection +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the + gate this feeds +- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record + the results diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx new file mode 100644 index 000000000..249e9ddf6 --- /dev/null +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -0,0 +1,314 @@ +--- +title: Health Checks and Traffic Admission +sidebar_position: 2 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +The default health check on most load balancers asks one question: did something +answer on this port? A Harper node answers that question correctly while still +being the wrong place to send a user request, because a node can be running before +its databases have synchronized, while it is catching up on missed transactions, or +while a dependency it needs for one particular journey is down. + +This guide builds four separate signals so your traffic layer can tell the +difference between "the process is alive" and "this node will serve this journey +correctly," and then uses them to drain and return a node safely. + +## What You Will Learn + +- Why liveness, availability, readiness, and journey verification are four signals + rather than one, and which layer consumes each +- How to install and drive the `@harperdb/status-check` component to control + whether a node advertises itself as available +- How to write an application readiness route that checks only what the routed + journey needs +- The order of operations for draining a node and returning it to service +- What makes a readiness endpoint safe under load, and what makes one dangerous + +## Prerequisites + +- A Harper cluster with at least two nodes, so draining one leaves a service + behind ([Fabric](/fabric) or self-managed) +- A `super_user` credential for the Operations API +- A traffic layer you can configure health checks on: a load balancer, a CDN + origin group, or a service mesh +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your + ports written down + +## Four signals, not one + +| Signal | Where it lives | What it proves | Who consumes it | +| -------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------- | +| Process liveness | Any trivial response on `9925` or `9926` | Harper is running | Process supervisor, restart policy | +| Node availability flag | `GET /status` on `9926`, from the `status-check` component | An operator or automation says this node should serve | Traffic layer health check | +| Application readiness | A component route you write on `9926` | This node can serve this journey right now | Traffic layer, or your own gating | +| Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | + +The distinction that does the most work is the second one. The availability flag is +not a measurement, it is a declaration. It exists so that you, or your automation, +can take a node out of rotation deliberately, before doing something to it, and +put it back afterwards. Nothing else in this list can be set by an operator, and +nothing else is safe to use as the primary routing signal. + +## Install the availability flag + +[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper +component that adds a `/status` route on the application port. Deploy it like any +other component: + +```json +{ + "operation": "deploy_component", + "project": "status-check", + "package": "@harperdb/status-check", + "restart": true +} +``` + +Or declare it in your application's `config.yaml`, which is the better option +because it makes the availability contract part of your deployed artifact rather +than a manual step someone has to remember on a new node: + +```yaml +status-check: + package: '@harperdb/status-check' +``` + +Once deployed, the route answers with a status code rather than a body: + +- `GET /status` returns `200` when the node is available, `404` when it is not +- `POST /status` marks the node available (authenticated) +- `DELETE /status` marks the node unavailable (authenticated) + + + + +```bash +# Is this node advertising itself as available? +curl -s -o /dev/null -w '%{http_code}\n' https://my-node.example.com:9926/status + +# Take it out of rotation +curl -s -X DELETE https://my-node.example.com:9926/status -u 'admin:password' + +# Put it back +curl -s -X POST https://my-node.example.com:9926/status -u 'admin:password' +``` + + + + +```javascript +const base = 'https://my-node.example.com:9926/status'; +const auth = { Authorization: 'Basic ' + btoa('admin:password') }; + +const res = await fetch(base); +console.log(res.status); // 200 available, 404 unavailable + +await fetch(base, { method: 'DELETE', headers: auth }); // out of rotation +await fetch(base, { method: 'POST', headers: auth }); // back in +``` + + + + +Point your traffic layer's health check at `GET /status` on `9926`, not at the +Operations API and not at your application's root. A `404` is the node telling the +traffic layer to stop sending work, and it will keep saying so until something +sets it back. + +:::warning +Whatever mechanism you use to persist the availability flag, keep it out of +replication scope. A flag that replicates would let one node's maintenance state +propagate to its peers, which turns a routine drain into an outage. If you are +scoping replication by hand, confirm this explicitly rather than assuming it. See +[Operating Replication](./operating-replication.mdx). +::: + +### A note on `set_status` + +The Operations API also offers +[`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) +for application-defined status values, with types for primary, maintenance, and +availability. + +These are a coordination primitive for your own automation, not a health report, +and not a substitute for a real readiness check. Nothing in Harper acts on a value +you set through them. Prefer the `status-check` component for traffic admission, +because its contract is an HTTP status code that a load balancer can consume +directly, and reach for `set_status` when you need to coordinate something between +your own scripts. + +## Write an application readiness route + +Liveness and the availability flag both answer questions about the node. +Readiness answers a question about the journey: if traffic arrives for this route +right now, will it succeed? + +The rule that keeps this useful is to check only the dependencies the routed +journey actually needs. A readiness route that checks everything will report a +node unready because of a subsystem that route never touches, and you will have +converted a partial degradation into a full outage yourself. + +Add a resource to your application's `resources.js`: + +```javascript +export class Readiness extends Resource { + static async get() { + const checks = {}; + + // A bounded read against the table this journey serves. + // Keep it to a single primary-key lookup, never a scan. + try { + await tables.Product.get('readiness-probe-sentinel'); + checks.data = 'ok'; + } catch (error) { + checks.data = 'failed'; + } + + // Only the downstream dependencies this route needs. + try { + const res = await fetch('https://pricing.internal.example.com/health', { + signal: AbortSignal.timeout(500), + }); + checks.pricing = res.ok ? 'ok' : 'failed'; + } catch (error) { + checks.pricing = 'failed'; + } + + const ready = Object.values(checks).every((v) => v === 'ok'); + + return new Response(JSON.stringify({ ready, checks, version: process.env.APP_VERSION }), { + status: ready ? 200 : 503, + headers: { 'Content-Type': 'application/json' }, + }); + } +} +``` + +Enable the `jsResource` plugin in `config.yaml` if it is not already, and the +route is served at `/Readiness` on the application port. See +[Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for +the resource and export mechanics. + +Two details in that example are the point of it. The timeout on the downstream +call means a slow dependency cannot make your readiness check hang, which would +make the node look dead to a probe rather than unready. And returning the version +means that when you are staring at a dashboard during a rollout, the readiness +response itself tells you which build answered. + +## Drain a node and bring it back + +The order matters, and the verification steps between them are the parts people +skip. + + + + +1. **Declare it unavailable.** `DELETE /status` on the target node. +2. **Verify traffic actually stopped.** Watch request volume on the target fall to + zero and rise on its peers. Do not trust the configured weight; check the + measured request count from + [analytics](/reference/v5/analytics/overview) or your traffic layer's own + metrics. Health check intervals, DNS TTLs, and client-side connection reuse all + add delay here, and the delay is yours to measure. +3. **Confirm the peers can carry it.** The remaining nodes have to be inside their + capacity budget with this node gone. If they are not, stop: see + [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). +4. **Do the work.** Restart, upgrade, reconfigure, or investigate. + + + + +1. **Confirm the databases are current.** `cluster_status` should show the expected + peer sockets connected for every database this node serves, and convergence + should be complete rather than in progress. +2. **Confirm readiness passes locally.** Call your `/Readiness` route directly + against the node, bypassing the traffic layer. +3. **Run the journey synthetic against the node directly.** A real read, or a safe + write, through the same path a user would take. +4. **Declare it available.** `POST /status`. +5. **Hold before restoring full weight.** Give it a stability window at partial + traffic and watch error rate and latency against its peers before treating it as + fully back. + + + + +Step 5 on the return side is the one worth defending in a review. Failback is a +change like any other, and a node that has just synchronized under load is the +most likely one to surprise you. Design failback, do not just design failover. + +## Readiness hygiene + +- **Keep the response cheap and bounded.** A readiness check that performs a broad + scan or writes data will amplify an incident, because it runs at probe frequency + across every node at exactly the moment the system is already struggling. +- **Never make it the only routing signal.** Liveness plus availability plus + readiness, consumed at the right layers. +- **Probe from more than one location** where your traffic layer supports it. A + single probe point cannot distinguish a network path problem from a node problem. +- **Make it observable.** Record response code, latency, the reason for a failure, + and the node and component version. A readiness check whose failures you cannot + explain after the fact is a check you will end up ignoring. +- **Version the contract.** If you change what readiness means, that is a change + to the traffic admission policy, and it deserves the same care as a code + release. + +### Prove it + +On a non-production cluster, restart one node under representative read and write +load, using the full drain and return sequence above. Record three numbers: how +long from `DELETE /status` until measured traffic on that node reaches zero, +whether any user-visible errors occurred during the transition, and how long from +process start until the node legitimately passed all three return gates. + +That third number is your real node re-entry time, and it is almost always longer +than people assume, because it includes convergence rather than just startup. + +## Operational notes + +- **A returning node is not the same as a new node.** A node whose databases have + never synchronized downloads them in full. A node that was offline and comes back + catches up on the transactions it missed. Both need to finish before traffic + arrives, but they take very different amounts of time, so do not budget for the + first when you are planning a routine restart. +- **Set unavailable before recovery work, not after.** Any operation that touches + data on a node, including a restore, should happen with the node out of rotation. +- **Configuration changes need a restart to take effect**, so a node that has been + reconfigured but not restarted is running the old configuration while reporting + the new one. Sequence the restart into the same maintenance window. +- **Fabric provides its own cluster-level health and routing.** These signals still + matter, because the availability flag and your readiness route are what Fabric's + routing has to consult. + +## Readiness checklist + +- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than + deployed by hand +- [ ] Traffic layer health check points at `GET /status` on `9926` +- [ ] Availability flag confirmed to be outside replication scope +- [ ] An application readiness route exists, scoped to one journey's dependencies +- [ ] Every downstream call in the readiness route has a timeout +- [ ] Readiness response includes the component version +- [ ] A critical-journey synthetic runs against the public route, separately from + the liveness probe +- [ ] Drain sequence documented, with measured time-to-zero-traffic +- [ ] Return sequence documented, with a stability window before full weight +- [ ] Measured node re-entry time recorded, including convergence + +## Additional Resources + +- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component + source and options +- [Operations API operations](/reference/v5/operations-api/operations) for + `cluster_status`, `system_information`, and the status operations +- [Components overview](/reference/v5/components/overview) for the full list of + Harper-maintained components, including the Prometheus exporter +- [Analytics overview](/reference/v5/analytics/overview) for per-node request and + latency data +- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) + for custom resources and the `jsResource` plugin +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers + can absorb a drained node diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx new file mode 100644 index 000000000..f9690246e --- /dev/null +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -0,0 +1,273 @@ +--- +title: How Harper Runs in Production +sidebar_position: 1 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +You have built an application, deployed it from a pipeline, and now real users are +going to depend on it. Operating Harper is not the same job as operating an +application tier in front of a database tier, because Harper does not have those +two tiers. One process holds your component code, the HTTP stack, the local +database, and the replication client. That shape removes several problems you may +be used to solving, and it changes where the remaining ones live. + +This guide is the map for the rest of the Administration track. It covers what a +single node actually contains, what that means for failure and scale, and how to +take an inventory of your own deployment before you design anything around it. + +## What You Will Learn + +- What a Harper node contains, and which port carries which kind of traffic +- Why the node is your unit of failure and your unit of scale, and what that + removes from your operating burden as well as what it adds +- The three ways a Harper runbook differs from a two-tier runbook +- How to inventory your service boundary with the Operations API, so later guides + have something concrete to work from + +## Prerequisites + +- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a + [local installation](../getting-started/install-and-connect-harper.mdx) +- A `super_user` credential for the Operations API +- An application deployed to it + ([Create your First Application](../getting-started/create-your-first-application.mdx)) + +## What one node contains + +A Harper node is a single process running your components, an HTTP server, a local +storage engine, and peer replication. There is no network hop between your +application code and the data it reads, no separate cache tier to keep coherent, +and no connection pool to tune between tiers. + +Three ports carry the traffic you will operate around: + +| Port | Serves | Who should reach it | +| ------ | --------------------------------------------------------------------------- | --------------------------------- | +| `9926` | Application traffic: REST, WebSocket, MQTT-over-WebSocket, component routes | Your traffic layer and your users | +| `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | +| `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | + +These are documented defaults, not guarantees about your cluster. Confirm the live +values rather than assuming them, because a replication port in particular can be +inherited from other configuration: + +```json +{ + "operation": "get_configuration" +} +``` + +Read back `http.port`, `operationsApi.network.port`, and the `replication` block. +Record what you find. Later guides in this track assume you know these numbers for +your own deployment. + +:::tip +Keep `9925` off any public route. The Operations API can deploy components, read +logs, and read configuration, so it is an administrative surface, not an +application one. See [security overview](/reference/v5/security/overview). +::: + +## The node is your unit of failure and your unit of scale + +Because one process holds the runtime and the local data together, there is no +internal application-to-database seam that can fail over independently. The node +is the practical unit of service failure. + +Start with what that removes, because it is the larger half of the trade: + +- No cross-tier network latency on data access, and no tail latency from a + saturated connection pool between tiers +- No cache invalidation problem between an application cache and a database of + record, because they are the same thing +- No partial-outage state where the application tier is healthy and the data tier + is not, which is the failure mode that produces the most confusing incidents +- One capacity number to measure and one thing to size + +Then the consequence: when you lose a node, you lose a whole slice of your +service, not one layer of it. This is why the first real design decision in +[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak +throughput. It is how many nodes you are willing to lose at once, and whether the +survivors can carry the load. + +Scale works on the same unit. Adding capacity means adding a node that carries +both request handling and data, so a scaling event is also a data movement event. +That is not a problem, but it is a thing with a duration, and traffic should not +arrive until it finishes. + +## Three differences that change your runbook + +### A process that is up is not a node that should take traffic + +A Harper process will answer a TCP connection and return an HTTP response before +it is a good place to send a user request. A new or replacement node has to +synchronize the databases it serves before its answers are correct. A returning +node has to catch up on transactions it missed. + +So liveness and admission are two different decisions, and a load balancer health +check that only proves liveness will route users to a node that is technically +running and functionally wrong. This is the whole subject of +[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), +and it is the single most common gap in a first production deployment. + +### Replication is peer-to-peer and scoped, so verify your own scope + +Harper peers exchange data over WebSockets with mTLS on the secure replication +port and discover each other through configured routes. There is no primary. Data +mutations and transactions replicate; some things do not, and the scope is +configurable per database and per table. + +The important operating habit is not memorizing a default. It is checking what +your cluster actually replicates, because the answer depends on your +configuration, your version, and whether anyone has scoped it since: + +```json +{ + "operation": "cluster_status" +} +``` + +The response lists each peer connection and, within it, one socket per database +per peer. That tells you which databases are actually flowing, which is the +question that matters during an incident. What is in scope, what is deliberately +out of it, and how to prove convergence rather than just connection are covered in +[Operating Replication](./operating-replication.mdx). + +### Reversal is a redeploy, not an infrastructure event + +Your application ships as a component, deployed with +[`deploy_component`](/reference/v5/operations-api/operations#deploy_component) +from an immutable reference. Rolling back means deploying the previous immutable +reference. There is no image to rebuild, no instance to replace, and no cluster to +rebuild to undo a bad release. + +That makes reversal fast enough to be a real option under pressure, which in turn +makes it worth designing for deliberately rather than improvising. It also means +code rollback, configuration rollback, and data recovery are three separate +actions with three different blast radii, and conflating them during an incident +is how a bad release becomes a data loss event. See +[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). + +## Inventory your service boundary + +Everything else in this track builds on knowing what you have. Run these four +operations against each node and write down the answers. + + + + +```bash +curl -s -X POST https://my-node.example.com:9925/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"system_information","attributes":["system","cpu","memory","disk","threads"]}' +``` + + + + +```javascript +await fetch('https://my-node.example.com:9925/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ + operation: 'system_information', + attributes: ['system', 'cpu', 'memory', 'disk', 'threads'], + }), +}); +``` + + + + +:::warning +The `attributes` array silently drops names it does not recognize, so a typo +returns a smaller response rather than an error. The valid values are `system`, +`time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, +`metrics`, and `threads`. If a response is missing a section you asked for, check +the spelling before you check the node. +::: + +Then: + +- [`get_components`](/reference/v5/components/applications#get_components) for + what is deployed, including which extensions are present +- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) + for what changed and when +- `cluster_status` for the peers and databases actually connected + +Fill in a boundary record you can keep: + +| Item | Value for your deployment | +| ----------------------- | ----------------------------------------------------- | +| Harper version | From `system_information` | +| Node count and roles | From `cluster_status` and your topology intent | +| Ports in use | From `get_configuration` | +| Databases | Which exist, and which replicate | +| Storage engine | Per database, since it constrains your backup options | +| Components deployed | From `get_components`, with the version of each | +| Critical journeys | The user-facing paths that must work, named | +| Downstream dependencies | Anything Harper calls that can fail independently | + +The last two rows are the ones people skip and the ones that matter most. A +healthy Harper process cannot compensate for a failed downstream dependency or for +application logic returning wrong answers, so an operating model that only watches +Harper will miss the incidents your users actually notice. + +### Prove it + +Before moving on, confirm the picture is real rather than assumed. On a +non-production cluster, stop one node and watch what happens to the others: +whether peers keep serving, how long the remaining nodes take to show the change +in `cluster_status`, and what your traffic layer does about it. You are not +measuring anything precisely yet. You are checking that the boundary you wrote +down matches the system you have. + +## Operational notes + +- **Fabric and self-managed differ in what you own, not in how Harper behaves.** + On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline + are managed for you. The failure unit, the replication model, and the admission + problem are identical. +- **Version parity across nodes is an operating requirement, not a nicety.** + Mixed versions in a cluster change replication and deployment behavior. Record + the version per node in your boundary inventory and alert on drift. +- **Configuration changes made through the API take effect on restart.** A + `set_configuration` call that has not been followed by a restart or + `restart_service` leaves a node running something other than its stated + configuration. Track pending changes in your change record. +- **`get_status` reports a `restartRequired` flag, but it tracks component and + code restarts rather than configuration changes.** Do not rely on it to tell you + a configuration change is still pending. + +## Readiness checklist + +- [ ] Ports confirmed from `get_configuration`, not assumed from documentation +- [ ] Operations API on `9925` is not reachable from the public internet +- [ ] Harper version recorded per node, with an alert on drift +- [ ] Databases listed, with replication scope confirmed via `cluster_status` +- [ ] Storage engine recorded per database +- [ ] Components and versions recorded from `get_components` +- [ ] Critical user journeys named and written down +- [ ] Downstream dependencies named, with their own failure behavior understood + +## Additional Resources + +- [HTTP server reference](/reference/v5/http/overview) for the application port and + server architecture +- [Operations API overview](/reference/v5/operations-api/overview) and the + [full operation list](/reference/v5/operations-api/operations) +- [Replication overview](/reference/v5/replication/overview) for the peer model, + mTLS, routes, and scope +- [Components overview](/reference/v5/components/overview) for the component model + and the available extensions +- [Database overview](/reference/v5/database/overview) for storage engines and + transaction boundaries +- [Configuration overview](/reference/v5/configuration/overview) for + `harper-config.yaml` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how + application code reaches these nodes diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx new file mode 100644 index 000000000..f25621455 --- /dev/null +++ b/learn/administration/monitoring-and-triage.mdx @@ -0,0 +1,293 @@ +--- +title: Monitoring and Triage +sidebar_position: 5 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +A dashboard is useful when it lets an operator place a symptom. Users are seeing +errors: is it the edge, the traffic layer, one node, the replication path, a +downstream dependency, or the change someone shipped twenty minutes ago? A +dashboard that shows CPU across the cluster cannot answer that, which is why it +gets ignored during incidents. + +This guide builds monitoring around the boundaries you can actually act on, names +the specific Harper metrics worth alerting on, and gives you a triage sequence +short enough to run under pressure. + +## What You Will Learn + +- The layers a Harper symptom can live in, and the minimum signal for each +- Which Harper metrics are worth an alert, by name, and which are only worth a + dashboard +- How to get metrics out of Harper, on Fabric and self-managed +- How to write log entries that are still useful during an incident +- A triage sequence that narrows a Harper incident in about five minutes + +## Prerequisites + +- A cluster with your application deployed and taking traffic +- A `super_user` credential for the Operations API +- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or + a Prometheus-compatible system for self-managed +- [Operating Replication](./operating-replication.mdx), since replication signals + are half of what you will watch + +## Monitor at the boundaries you operate + +| Layer | Minimum signals | Where they come from | +| ------------------ | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | +| User and edge | Success, latency, correctness, broken out by geography and cohort | Your synthetic journey, real-user telemetry, CDN and traffic layer logs | +| HTTP and component | Request rate, error rate, latency percentiles, active component version, availability flag | `success` and `duration` metrics, `GET /status`, your readiness route | +| Node resources | CPU, memory, worker utilization, task queue latency, disk, restarts | `system_information`, `utilization`, `main-thread-utilization`, host or container metrics | +| Replication | Peer connections, per-database sockets, latency, convergence lag | `cluster_status`, `replication-latency`, `system.hdb_nodes` | +| Data | Read and write rate, transaction commit time, queue depth, freshness | `transaction-commit-time`, write and read transaction queue depth, your own sentinel | +| Storage | Database size, volume free space, table growth | `database-size`, `storage-volume`, `table-size` | +| Change | Component and config version, cohort, operator, start and end, outcome | `list_deployments`, `get_deployment`, `get_components`, `get_configuration` read-back | +| Recovery | Backup age, job state, verification result, last restore drill | `list_backups`, `verify_backup`, `get_job` | + +The Change row is the one teams leave out and the one that resolves incidents +fastest. Most production symptoms correlate with something a human did, so being +able to overlay deployments onto a latency graph is worth more than another +resource metric. + +## The signals worth an alert + +Harper records a large standard metric set automatically. Most of it belongs on a +dashboard. This much belongs on a pager: + +| Alert on | Metric or source | Why this one | +| -------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | +| Journey success rate below SLO | `success`, by resource path and method | The only signal that maps directly to user impact | +| Journey latency at your SLO percentile | `duration`, by resource path and method | Alert on the percentile in your SLO, never on the mean | +| Worker saturation | `utilization`, per worker thread | Percentage of time the worker was processing requests. This is your real headroom signal | +| Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | +| Replication convergence lag | `replication-latency`, or the receive-time gap in `cluster_status` | A node serving stale data looks healthy on every other signal | +| Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | +| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | +| Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | +| Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | +| Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | + +Two notes on how to set these. Alert on the percentile that appears in your SLO, +because a mean latency graph will look fine through an incident that is failing +your slowest ten percent of users. And set saturation thresholds from the +per-node capacity work in +[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a +generic number, since the point where latency leaves your SLO is specific to your +application and your data. + +## Get the metrics out + +Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into +`hdb_analytics`. You can query those directly, but for a real monitoring setup +export them. + + + + +Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards +over Harper's analytics without you building a pipeline, which is the fastest path +to the alert list above. + + + + +Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) +and scrape it into whatever you already run: + +```json +{ + "operation": "deploy_component", + "project": "prometheus-exporter", + "package": "@harperdb/prometheus-exporter", + "restart": true +} +``` + + + + +To find out what is actually available on your version rather than guessing from +documentation: + +```json +{ + "operation": "list_metrics", + "metric_types": ["builtin", "custom"] +} +``` + +Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for +the shape of any one of them. + +Your application can add its own metrics with +[`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), +which is how you get business-level signals such as checkout completion onto the +same dashboard as node saturation. That correlation is usually what tells you +whether a technical symptom matters. + +:::tip +Check `analytics.replicate` in the +[analytics configuration](/reference/v5/analytics/overview#analytics-configuration) +before building dashboards. Whether metrics stay node-local or replicate changes +what a cluster-wide query means, and it is easier to decide that deliberately than +to discover it after building panels on the wrong assumption. +::: + +## Logs you can use during an incident + +Harper's `logger` global writes structured entries from component code: + +```javascript +logger.info('order submitted', { + component: 'orders-api', + version: process.env.APP_VERSION, + node: process.env.HARPER_NODE_NAME, + requestId: context.requestId, + operation: 'submit', + durationMs: elapsed, + outcome: 'ok', +}); +``` + +Include the component and its version, the node, a request or trace identifier, +the operation, its duration, the outcome, and a safe error class. That field set +is what lets you answer "was this only the new version" and "was this only one +node" without guessing, and those are the first two questions in almost every +incident. + +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The +default is `warn`, and `notify` is always logged regardless of level. Choose +levels deliberately: a production log at `debug` is a log nobody can read, and one +at `error` only has the incidents in it and none of the context. + +`console.log` output does not reach the log files unless `logging.console` is +enabled, so unstructured console output is not a production record. Centralize +logs off the node, because the node you most need logs from is the one you are +about to restart. + +Read logs through the API when you need them from a specific node: + +```json +{ + "operation": "read_log", + "limit": 200, + "level": "error" +} +``` + +## The five-minute triage sequence + +Run these in order. The goal is not diagnosis, it is narrowing. + +1. **Confirm and bound the impact.** What journey, starting exactly when, in which + geography or cohort, and which component version is live. Without a start time + you cannot correlate anything. + +2. **Compare the public route against a direct node call.** Request the same + journey through your traffic layer, then directly against each node on `9926`. + If direct calls succeed and the public route fails, you are looking at the + traffic layer or the availability flag, not at Harper. + +3. **Check admission state on every node.** `GET /status` on `9926` and your + readiness route. A node advertising unavailable is a node deliberately or + accidentally out of rotation, and finding that here saves a lot of time. + +4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then + diff: + + ```json + { + "operation": "system_information", + "attributes": ["cpu", "memory", "threads", "harperdb_processes"] + } + ``` + + Differences between peers are more informative than absolute values, because + they tell you whether this is one node or the whole cluster. + +5. **Check the replication path.** `cluster_status` for the databases this journey + needs. Look for a missing peer, `connected: false`, or a widening gap between + `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. + +6. **Correlate with the last change.** `list_deployments` for what shipped and + when, and `get_configuration` read back against what you believe is configured. + Remember that a configuration change applied without a restart leaves a node + running something other than its stated configuration. + +7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, + deactivate a feature, restore prior traffic weights, or isolate suspected data. + Record the decision, who made it, and the next decision deadline. + +Two rules make this sequence work. Silence is a failed gate: if telemetry is +missing for the thing you are checking, treat that as a negative signal rather +than skipping the step. And containment comes before root cause. You can diagnose +after users are being served again. + +### Prove it + +Pick a fault and inject it on a non-production cluster, then time yourself. Good +candidates: saturate one node's workers, block the replication port on one peer, +make a downstream dependency return errors, or deploy a component that fails on +one route. + +Measure three things. How long until an alert fired. How long until an operator +following the sequence above could name the layer. And whether any step gave a +misleading answer, which is the most valuable output of the drill, because a +misleading signal during a real incident costs more than a missing one. + +## Operational notes + +- **Node-level dashboards, not cluster averages.** A cluster average conceals the + single node doing twice the work, which is the most common cause of a latency + complaint that looks like nothing on a dashboard. +- **Watch measured request distribution, not configured weights.** Per-node + request counts are the ground truth. Sticky sessions, DNS caching, and + connection reuse all skew actual distribution away from intent. +- **Annotate deployments onto your graphs.** If your monitoring supports + annotations, feed `list_deployments` into them. This single change resolves more + incidents faster than any additional metric. +- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for + reconstructing what changed in a table. Know before an incident whether you have + it on, because turning it on afterwards does not help. +- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose + configuration and source detail, so they are `super_user` operations for a + reason. Restrict and log their use. + +## Readiness checklist + +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, + replication, data, or change +- [ ] Alerts on journey success and latency at the SLO percentile, not the mean +- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived + from measured per-node capacity +- [ ] Alerts on replication convergence lag and missing peers +- [ ] Alerts on storage headroom and backup age +- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus + exporter self-managed +- [ ] `analytics.replicate` setting known and deliberate +- [ ] Structured logging includes component, version, node, request id, operation, + duration, and outcome +- [ ] Logs centralized off the node +- [ ] Deployments annotated onto dashboards +- [ ] Triage sequence written down where on-call can find it +- [ ] Fault injection drill completed, with time-to-detection recorded + +## Additional Resources + +- [Analytics overview](/reference/v5/analytics/overview) for the full standard + metric catalog and configuration options +- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` + and `describe_metric` +- [Grafana integration](/fabric/grafana-integration) for Fabric dashboards +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) + for self-managed metric scraping +- [Logging overview](/reference/v5/logging/overview), + [configuration](/reference/v5/logging/configuration), and + [operations](/reference/v5/logging/operations) +- [HTTP API reference](/reference/v5/http/api) for `server.recordAnalytics()` +- [Fabric logging](/fabric/logging) for log access on Fabric +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the + gates these signals feed diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx new file mode 100644 index 000000000..617f5c53f --- /dev/null +++ b/learn/administration/operating-replication.mdx @@ -0,0 +1,342 @@ +--- +title: Operating Replication +sidebar_position: 4 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Replication is how a Harper cluster stays available: peers exchange data directly +over WebSockets, there is no primary, and a node that falls behind catches up on +its own. The [replication reference](/reference/v5/replication/overview) covers +how to configure that. This guide covers the part that only matters once real +users depend on it, which is how to tell whether replication is actually working. + +The distinction that runs through this guide is between a connection and +convergence. A connected socket proves two nodes can talk. It does not prove the +data a user is about to read is current. Those are different claims, and only one +of them is what your traffic admission decision depends on. + +## What You Will Learn + +- How to determine what your cluster actually replicates, rather than assuming a + default +- What moves between peers automatically, what does not, and which of those will + surprise you +- How to read `cluster_status` as an operator, including which timing field + actually indicates a node is behind +- How to prove convergence with a sentinel rather than inferring it from + connection state +- Which application behaviors replication cannot make safe, and what to do about + them instead + +## Prerequisites + +- A cluster of at least three nodes, so you can interrupt one peer and still + observe the others +- A `super_user` credential for the Operations API +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your + service boundary inventory +- Familiarity with how your cluster was joined, either through + `harper-config.yaml` routes or the + [clustering operations](/reference/v5/replication/clustering) + +## Know your own scope + +By default Harper replicates all data in all databases. Scope can be narrowed two +ways: per database in configuration, and per table in the schema. + +```yaml +replication: + databases: + - data + - system +``` + +```graphql +type LocalTableForNode @table(replicate: false) { + id: ID! + name: String! +} +``` + +All tables in a replicated database replicate unless the table opts out. So the +scope you are operating is the product of a config list, a set of schema +directives, and any directional routes someone added later. Do not reconstruct it +from memory. Read it off the running cluster: + + + + +```bash +curl -s -X POST https://my-node.example.com:9925/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"cluster_status"}' +``` + + + + +```javascript +await fetch('https://my-node.example.com:9925/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ operation: 'cluster_status' }), +}); +``` + + + + +There is one socket per database per peer, so the sockets present in the response +are the ground truth about what is flowing. Run this from every node, not one. +Replication direction can be constrained per route, so node A's view of the +cluster is not necessarily node B's view, and a one-sided picture is how +directional configuration mistakes survive into production. + +:::warning +Whether the `system` database is in your replication scope is the highest-stakes +scoping question in a Harper cluster, because `system` holds `hdb_user`, +`hdb_role`, and `hdb_nodes`. Users and roles propagate only when `system` is +replicated, and a node that receives `system` must be trusted with its contents, +including encrypted secret rows. Confirm your own answer from `cluster_status` and +your configuration, and read +[replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) +before changing it. +::: + +## What moves automatically, and what does not + +| Object or change | Behavior | What it means for you | +| --------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | +| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | +| A brand new node | Downloads the full database from the first node it connects to, then enters incremental replication | Budget the transfer, and expect load on the source node while it happens | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | +| Component deployment | `deploy_component` supports `"replicated": true` for cluster-wide propagation | Useful once an artifact is trusted. Keep single-node deployment available for validation | +| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | +| Users and roles | Propagate only when the `system` database is in replication scope | If `system` is out of scope, identity must be provisioned on every node by your own automation | +| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | +| Destructive schema changes | Behavior is operation and version dependent | Treat as high risk. Write an exact procedure, and verify the result on every node | + +The two rows most likely to bite you are the returning node and the node registry. + +On the returning node: it is common to see a routine restart budgeted as though it +were a full resynchronization, which makes teams avoid restarts they should be +comfortable with. A node whose databases have never synced does download them in +full. A node that was briefly offline catches up on what it missed. Measure both +on your own data volume once, and use the right number for the right situation. + +On the registry: a node's advertised record is derived from its configuration +file, and it replicates. That means a topology constraint you applied +imperatively is superseded the next time that node restarts or reloads +components, and the node quietly goes back to advertising itself more broadly +than you intended. + +:::danger +When replicating configuration, only send cluster-appropriate parameters. +Replicating a node-local value such as a port, `node.hostname`, a file path, TLS +material, or `replication.hostname`, `url`, or `routes` overwrites every peer's +own local value. To apply a cluster-wide change safely, use +`set_configuration` with `"replicated": true` for the parameter, then +`restart_service` with `"replicated": true`, which restarts nodes one at a time. +::: + +## Read `cluster_status` like an operator + +A trimmed response, with the fields that matter: + +```json +{ + "type": "cluster-status", + "node_name": "server-1.example.net", + "is_enabled": true, + "connections": [ + { + "url": "wss://server-2.example.net:9933", + "name": "server-2.example.net", + "database_sockets": [ + { + "database": "data", + "connected": true, + "latency": 0.7, + "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", + "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", + "lastReceivedLocalTime": "Wed, 12 Feb 2025 19:09:31 GMT" + } + ] + } + ] +} +``` + +What each field is telling you: + +- **`connected`** is the liveness of this one database's socket to this one peer. + A missing peer, or a peer present with `connected: false`, is actionable before + users notice. +- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained + growth rather than on a single sample. +- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one + of your commits. If it stops advancing while you are still writing, your writes + are not landing on that peer. +- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest + transaction you have received. +- **`lastReceivedLocalTime`** is your own clock when you received it. + +The last two are the pair that matters. **A widening gap between +`lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and +working through a backlog.** That is the signal to alert on for convergence, and +it is the one that tells you a returning node is not ready for traffic yet. +`sendingMessage` appears while a transaction is actively being sent and is absent +when the socket is idle, so its absence is not a fault. + +Inventory `system.hdb_nodes` alongside this and compare it to your intended +topology. Configuration intent and live peer state should agree, and the node's +own row is in there too, not just its peers. + +### What to alert on + +- A peer missing entirely from `connections`, or `connected: false` on a database + that serves a critical journey +- Sustained growth in `latency`, judged against your own baseline +- `lastCommitConfirmed` not advancing on a peer while writes are occurring +- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your + admission budget +- Repeated reconnects, which are visible in the logs even when a point-in-time + status check looks healthy +- Version or configuration drift between peers + +## Prove convergence, not connection + +Socket state cannot tell you that a specific business record is current. For +anything where the answer matters, write a sentinel and read it back from the +peer. + +```javascript +// On the source node: write a sentinel with a known value +const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' }; +await tables.OpsProbe.put(marker.id, marker); +``` + +```bash +# On the target node: read it back and compare writtenAt +curl -s https://server-2.example.net:9926/OpsProbe/convergence-probe \ + -u 'admin:password' +``` + +The interval between the write and the moment the peer returns the new value is +your measured convergence time for that database, under whatever load the cluster +is carrying at the time. Run it under load, not on an idle cluster, and record the +result. That number is what your node admission gate should be compared against, +and it is an input to the RPO work in +[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +The `replication-latency` metric from +[analytics](/reference/v5/analytics/overview#replication-metrics) gives you the +continuous version of the same measurement: the difference between the source +commit timestamp and local time, reported per node, database, and table. Use the +sentinel for a definitive answer during a change, and the metric for a dashboard. + +## Consistency belongs in the application design + +Replication makes data available on every peer. It does not make every peer agree +at every instant, and a distributed decision made on a node that has not yet +converged can observe stale state and act on it. + +This is not a Harper limitation to work around, it is the property that lets any +node serve any request without a coordinator. But it means a specific class of +operation is unsafe if you write it as a plain read followed by a write: + +- Claim-once actions: redeeming a code, assigning a unique handle, awarding a + one-per-customer offer +- Hard floors: inventory that must not go negative, a balance that must not + overdraw +- Global limits: a rate limit or quota enforced across the whole cluster +- State machine transitions where two nodes could both believe they are making + the same transition + +For each of these, choose one of three designs: route all decisions for a given +key to a single owner, use a serialization mechanism so the conflict is resolved +in one place, or delegate to an external coordinator. Then test your +read-after-write expectations through the actual public route rather than against +a single node, because a single node always looks consistent to itself. + +### Prove it + +On a non-production cluster under write load: + +1. Record baseline `cluster_status` timing fields on every node. +2. Interrupt replication to one peer, by stopping the node or blocking the + replication port, while writes continue elsewhere. +3. Watch what your monitoring reports, and how long it takes to say anything. Note + whether `connected` flipped, whether latency alerted, and how long until a + human would have known. +4. Verify your application's actual behavior on the isolated node. Does the + critical journey fail, serve stale data, or serve correctly? All three are + possible and you should know which. +5. Restore the peer, then measure convergence with the sentinel until it is + current. +6. Compare the convergence time against the admission gate in + [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + If your gate would have admitted the node before step 5 finished, the gate is + wrong. + +## Operational notes + +- **mTLS is required on the secure replication port and cannot be disabled.** + Certificate expiry is therefore a replication outage, so treat certificate + lifetime as an operational deadline with its own alert. See + [certificate management](/reference/v5/security/certificate-management). +- **Gossip discovery means one route can join a whole cluster.** A node that + connects to one peer discovers the rest. That is convenient and it means an + accidental route can widen your topology more than you intended. +- **Sharding is a separate scope control.** If you use + [sharding](/reference/v5/replication/sharding), not every node holds every + record, so "this node is converged" and "this node can answer this query + locally" become different questions. +- **Analytics data has its own replication setting.** See `analytics.replicate` in + the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) + before you build dashboards that assume metrics are or are not cluster-wide. +- **Keep node-local operational state out of scope.** Availability flags, + maintenance markers, and anything else that describes one node rather than the + cluster should not replicate, or a routine drain propagates to peers. + +## Readiness checklist + +- [ ] Replication scope read off the running cluster, from every node, not + reconstructed from memory +- [ ] Whether the `system` database is in scope is a documented, deliberate + decision +- [ ] Identity provisioning procedure exists and matches that decision +- [ ] Tables requiring one transaction boundary confirmed to be in one database +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in + `add_node` calls +- [ ] Measured convergence time recorded under load, per critical database +- [ ] Alerts configured on missing peers, sustained latency growth, stalled + `lastCommitConfirmed`, and the remote-to-local receive gap +- [ ] Non-commutative operations identified and given an owner, a serialization + point, or an external coordinator +- [ ] Replication certificate expiry dates tracked with an alert +- [ ] Replication interruption and recovery exercise completed and dated + +## Additional Resources + +- [Replication overview](/reference/v5/replication/overview) for routes, scope, + controlled flow, and securing connections +- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, + `set_node`, and the `cluster_status` response in full +- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes + hold a given record +- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for + `replication-latency` and byte counters +- [Certificate management](/reference/v5/security/certificate-management) and + [certificate verification](/reference/v5/security/certificate-verification) +- [Database schema](/reference/v5/database/schema) for the `replicate` table + directive +- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals + on a dashboard diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx new file mode 100644 index 000000000..29cabc5aa --- /dev/null +++ b/learn/administration/production-readiness-checklist.mdx @@ -0,0 +1,230 @@ +--- +title: Production Readiness Checklist +sidebar_position: 9 +--- + +This is the aggregated launch gate for the whole Administration track. Every item +appears in one of the earlier guides, which is where the reasoning lives. This +page exists so you have one thing to work down before a launch, and one thing to +hand to a reviewer who asks how you operate Harper. + +Copy it into your own runbook and adapt it. It is a starting point for your gate, +not a substitute for having one. + +## How to use this + +Each item is a claim about your deployment. A claim counts when there is evidence: +a measured number with a date, a written procedure, a configured alert, or a +completed drill. "We know about that" is not evidence, and neither is a passing +intention. + +Some items will not apply to you, and that is fine. Mark them not applicable with +a reason rather than deleting them, so a reviewer can see the decision was made +rather than missed. + +Three things are worth deciding before you start: + +- **Who signs off.** A gate with no named approver is a document, not a gate. +- **What "outstanding" means.** Some items can be accepted as gaps with an owner + and a date. Decide in advance which ones cannot. +- **When you run it again.** This is not a one-time launch artifact. Re-run it + after a Harper version upgrade, a topology change, or significant data growth, + since most of the measured numbers expire. + +## Service definition + +From [How Harper Runs in Production](./how-harper-runs-in-production.mdx). + +- [ ] Critical user journeys named, with an owner for each +- [ ] Ports confirmed from `get_configuration` rather than assumed +- [ ] Operations API not reachable from the public internet +- [ ] Harper version recorded per node, with an alert on drift +- [ ] Databases listed, with replication scope confirmed from `cluster_status` +- [ ] Storage engine recorded per database +- [ ] Components and versions recorded from `get_components` +- [ ] Downstream dependencies named, with their failure behavior understood +- [ ] Fabric or self-managed responsibilities understood and divided + +## Capacity and topology + +From [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). + +- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to + change operators +- [ ] Per-node throughput measured with your own application code, data shape, and + query mix +- [ ] Measurement taken with a peer synchronizing, not on an idle cluster +- [ ] Target utilization derived from latency at your SLO percentile +- [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers +- [ ] Measurement conditions recorded alongside the number, and dated +- [ ] Topology's "what must be proven" satisfied, including independence of failure + domains +- [ ] Maintenance policy states what happens when a drain would breach `F` +- [ ] Storage growth thresholds set, separately from request-rate thresholds + +## Health and traffic admission + +From +[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + +- [ ] Liveness, availability flag, application readiness, and journey synthetic all + exist as separate signals +- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than by + hand +- [ ] Traffic layer health check points at `GET /status` on the application port +- [ ] Availability flag confirmed to be outside replication scope +- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every + downstream call +- [ ] Readiness response includes the component version +- [ ] Drain sequence documented, with measured time to zero traffic +- [ ] Return sequence documented, including convergence verification and a + stability window before full weight +- [ ] Measured node re-entry time recorded + +## Replication + +From [Operating Replication](./operating-replication.mdx). + +- [ ] Replication scope read off the running cluster, from every node +- [ ] Whether the `system` database is in scope is a deliberate, documented + decision +- [ ] Identity provisioning procedure matches that decision +- [ ] Tables that must commit together confirmed to be in one database +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in + `add_node` calls +- [ ] Measured convergence time recorded under load, per critical database +- [ ] Alerts on missing peers, sustained latency growth, stalled + `lastCommitConfirmed`, and the receive-time gap +- [ ] Non-commutative operations identified and given an owner, a serialization + point, or an external coordinator +- [ ] Replication certificate expiry tracked with an alert +- [ ] Replication interruption and recovery exercise completed and dated + +## Deployment and rollback + +From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). + +- [ ] Artifacts immutable, referenced by pinned version rather than by branch +- [ ] Deploy and expose are separate actions in the pipeline +- [ ] Previous known-good artifact addressable, and redeploying it rehearsed +- [ ] Cohort ladder defined, from smallest useful sample to full exposure +- [ ] Stop thresholds and hold times declared before a rollout begins +- [ ] Named decision owner for advance, hold, stop, and roll back +- [ ] Pipeline polls the deployment record and the restart job separately +- [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately +- [ ] Configuration changes go through the same change record as code +- [ ] `get_configuration` read back after every configuration change +- [ ] Node-local parameters never replicated +- [ ] Schema changes follow expand then contract, with destructive operations + documented and verified per node +- [ ] Forward repair defined for changes that alter persisted meaning +- [ ] Measured release reversal time recorded from a rehearsal + +## Observability + +From [Monitoring and Triage](./monitoring-and-triage.mdx). + +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, + replication, data, or change +- [ ] Alerts on journey success and latency at the SLO percentile, not the mean +- [ ] Alerts on worker utilization and task queue latency, with thresholds derived + from measured capacity +- [ ] Alerts on replication convergence lag +- [ ] Alerts on storage headroom and backup age +- [ ] Metrics exported off the node +- [ ] `analytics.replicate` setting known and deliberate +- [ ] Structured logs include component, version, node, request id, operation, + duration, and outcome +- [ ] Logs centralized off the node +- [ ] Deployments annotated onto dashboards +- [ ] Triage sequence written where on-call can find it +- [ ] Fault injection drill completed, with time to detection recorded + +## Backup and recovery + +From [Backup and Recovery](./backup-and-recovery.mdx). + +- [ ] Failure classes enumerated, each with the mechanism that addresses it +- [ ] Recovery point and recovery time stated per database +- [ ] Mechanism chosen per database according to its storage engine +- [ ] No database in scope uses per-table storage paths, or the exclusion is known + and accepted +- [ ] Backup cadence matches the stated recovery point +- [ ] Backup volume sized against the full cost, including the non-incremental + transaction-log and blob snapshots +- [ ] Off-host copy exists in a destination that does not share a failure domain +- [ ] Managed repository copies take the whole per-database directory, quiesced or + from an atomic snapshot +- [ ] `verify_backup` runs on a schedule +- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Restore constraints documented for user databases, component-held databases, + and `system` +- [ ] Restore authority named, and restore execution logged +- [ ] Restore drill completed and dated, with measured recovery time and actual + data loss +- [ ] `system` database restore rehearsed offline + +## Objectives + +From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +- [ ] Recovery point, recovery time, and availability stated per journey with the + measurement boundary named +- [ ] Exclusions documented +- [ ] Every scenario in the scenario map has a named mechanism +- [ ] Availability target converted to minutes per month +- [ ] Error budget policy states what changes when the budget runs low +- [ ] Backup cadence reconciled against recovery point +- [ ] Restore time plus detection time reconciled against recovery time +- [ ] Convergence time reconciled against the admission gate +- [ ] Every unreconciled gap has an owner and a date +- [ ] Degraded modes declared per journey, with authority to invoke them + +## Security and access + +- [ ] Least privilege enforced for operators, applications, and automation +- [ ] `super_user` credentials inventoried, with a rotation procedure +- [ ] Operations API restricted to administrative networks +- [ ] TLS material inventoried, with expiry alerts for both application and + replication certificates +- [ ] Deploy credentials held in a secret store, not in pipeline configuration +- [ ] Administrative access logged, including CLI access on the hosts +- [ ] Filesystem access on nodes understood to permit offline restore without an + API credential + +## Ownership and runbooks + +- [ ] On-call rotation and escalation path defined +- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster + expansion +- [ ] Every runbook has been executed by someone other than its author +- [ ] Change record location known, and used +- [ ] Incident and postmortem process defined +- [ ] A named owner for this checklist, and a date for the next review + +## Exercises completed + +Record the date of the most recent run of each. An undated exercise is an +undocumented one. + +| Exercise | Guide | Last run | Result | +| ----------------------------------------- | ------------------------------------------------------------------------ | -------- | ------ | +| Service boundary inventory | [How Harper Runs in Production](./how-harper-runs-in-production.mdx) | | | +| Drain and return under load | [Health Checks](./health-checks-and-traffic-admission.mdx) | | | +| Peak-load drain and rejoin | [Sizing](./sizing-a-harper-cluster.mdx) | | | +| Replication interruption and recovery | [Operating Replication](./operating-replication.mdx) | | | +| Fault injection and time to detection | [Monitoring and Triage](./monitoring-and-triage.mdx) | | | +| Stopped rollout and reversal | [Safe Deployments](./safe-deployments-and-rollback.mdx) | | | +| Restore drill, user database | [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Restore drill, `system` database, offline | [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Scenario tabletop | [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) | | | + +## Additional Resources + +- [Reliability Plan Template](./reliability-plan-template.mdx) for the document + that holds the answers this checklist asks for +- Every guide in this section, linked per group above +- [Security overview](/reference/v5/security/overview) and + [certificate management](/reference/v5/security/certificate-management) +- [Configuration options](/reference/v5/configuration/options) for the settings + referenced throughout diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx new file mode 100644 index 000000000..f580ef251 --- /dev/null +++ b/learn/administration/reliability-plan-template.mdx @@ -0,0 +1,199 @@ +--- +title: Reliability Plan Template +sidebar_position: 10 +--- + +The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks +the questions. This is the document that holds the answers. + +Copy it, keep one per service, and keep it where your on-call can reach it during +an incident rather than in a wiki nobody remembers. It is deliberately short. +A reliability plan that takes a day to read is a reliability plan nobody consults +at 3am. + +Example values are included in italics to show the intended level of specificity. +Replace them. + +--- + +## 1. Service identity + +| Field | Value | +| ------------------ | ---------------------------------------------------------- | +| Service name | _orders-api_ | +| Business owner | _name, team_ | +| Technical owner | _name, team_ | +| On-call rotation | _link to schedule_ | +| Escalation path | _first, second, and decision authority for a stop_ | +| Harper deployment | _Fabric cluster, or self-managed, with cluster identifier_ | +| Harper version | _5.2.x, per node_ | +| Plan last reviewed | _date_ | +| Next review due | _date_ | + +## 2. Critical journeys + +The user-facing paths that must work. Everything else in this plan is stated per +journey, so this table defines the scope of all of it. + +| Journey | Description | Databases and tables involved | Downstream dependencies | Owner | +| ---------------- | ----------------------------- | ----------------------------- | ----------------------- | ----- | +| _submit order_ | _POST through the public API_ | _orders.orders, orders.items_ | _pricing, payments_ | | +| _browse catalog_ | _cached read_ | _catalog.products_ | _none_ | | + +## 3. Objectives + +| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective | Exclusions | +| ------- | ------------------- | -------------------- | ----------------------- | ------------------------ | ------------------------------------------ | +| | _99.95%_ | _at the CDN edge_ | _15 min from detection_ | _5 min_ | _planned maintenance, third-party outages_ | + +Monthly unavailability budget: _21 min 55 sec at 99.95%_ + +Error budget policy: _what changes when the remaining budget falls below a +threshold, and who decides_ + +## 4. Architecture and topology + +- Node count and `F`, the tolerated simultaneous loss: _5 nodes, F = 1_ +- Failure domain layout: _how nodes are distributed, and the proof that domains are independent_ +- Traffic layer and routing: _what fronts Harper, and how it selects nodes_ +- Replication scope: _which databases, which tables excluded, and whether `system` is in scope_ +- Sharding in use: _yes or no, and where_ +- Storage engine per database: _per database_ + +## 5. Capacity + +| Measurement | Value | Conditions | Date measured | +| -------------------------- | ----- | ------------------------------------------------ | ------------- | +| Peak required throughput | | _per journey_ | | +| Tested per-node throughput | | _data volume, query mix, peer synchronizing y/n_ | | +| Target utilization | | _derived from latency at the SLO percentile_ | | +| Surviving capacity at `F` | | _measured in the drain drill, not calculated_ | | + +Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept +with documented duration_ + +## 6. Traffic admission + +- Liveness probe: _endpoint, interval, threshold_ +- Availability flag: _`status-check` deployed how, and who may set it_ +- Readiness route: _path, and the dependencies it checks_ +- Journey synthetic: _what it does, from where, how often_ +- Drain sequence: _link to runbook_ +- Return sequence, including stability window: _link to runbook_ +- Measured time to zero traffic after drain: _value, date_ +- Measured node re-entry time including convergence: _value, date_ + +## 7. Replication + +- Measured convergence time under load, per critical database: _value, date_ +- Measured full synchronization time for a new node: _value, date_ +- Admission gate threshold, and how it compares to convergence: _value_ +- Non-commutative operations and their handling: _operation, and the owner, serialization point, or coordinator_ +- Certificate expiry dates and alert lead time: _dates_ + +## 8. Deployment and change + +- Artifact source and reference format: _registry or git, pinned version format_ +- Cohort ladder: _one node, internal, 5 percent, 25 percent, full_ +- Stop thresholds and hold times: _stated numerically_ +- Decision authority for advance, hold, stop, roll back: _name or role_ +- Restart mode used, and how completion is confirmed: _`rolling`, polling the job_ +- Measured release reversal time: _value, date_ +- Forward repair procedures for data-affecting releases: _link_ +- Change record location: _link_ + +## 9. Observability + +- Dashboard location: _link_ +- Alerts configured, with thresholds and destinations: _table or link_ +- Metric export path: _Grafana on Fabric, or Prometheus scrape_ +- Log destination and retention: _system, retention period_ +- Deployment annotations enabled: _yes or no_ +- Triage runbook: _link_ +- Measured time to detection, per drill: _value, date_ + +## 10. Backup and recovery + +| Database | Mechanism | Cadence | Retention | Off-host destination | Last verified | Last restore drill | Measured restore time | Measured data loss | +| -------- | --------- | ------- | --------- | -------------------- | ------------- | ------------------ | --------------------- | ------------------ | +| | | | | | | | | | + +- Restore authority: _who may execute a restore_ +- Restore constraints per database: _online, or offline only because a component holds it open_ +- `system` database restore procedure: _offline only, link to runbook_ +- Failure classes assigned to restore versus forward repair: _table_ + +## 11. Scenario responses + +| Scenario | Mechanism | Runbook | What proves it works | Last exercised | +| ---------------------------------- | --------- | ------- | -------------------- | -------------- | +| _single node loss_ | | | | | +| _node return after restart_ | | | | | +| _node replacement_ | | | | | +| _failure domain loss_ | | | | | +| _region loss_ | | | | | +| _bad release, no data change_ | | | | | +| _bad release that changed data_ | | | | | +| _accidental destructive operation_ | | | | | +| _storage loss on one node_ | | | | | +| _certificate expiry_ | | | | | +| _downstream dependency outage_ | | | | | + +## 12. Degraded modes + +| Journey | Condition | Declared behavior | Who may invoke | Automatic or manual | +| ------- | ------------------------------- | ------------------------------------------- | -------------- | ------------------- | +| | _pricing service unavailable_ | _serve last known price, flag the response_ | | | +| | _node behind on replication_ | _remove from rotation_ | | | +| | _cluster below capacity budget_ | _shed non-critical traffic_ | | | + +## 13. Security and access + +- Operator roles and privileges: _who has what_ +- `super_user` credential inventory and rotation: _link, schedule_ +- Operations API network restriction: _how enforced_ +- Certificate inventory and expiry alerts: _link_ +- Deploy credential custody: _secret store, rotation method_ +- Administrative access logging, including host-level: _system_ + +## 14. Open risks + +Accepted gaps, each with an owner and a date. This section existing and being +honest is worth more than it being empty. + +| Risk | Impact if realized | Why accepted | Owner | Review date | +| ---- | ------------------ | ------------ | ----- | ----------- | +| | | | | | + +## 15. Revision history + +| Date | Author | Change | Trigger | +| ---- | ------ | ------ | ---------------------------------------------------- | +| | | | _launch, version upgrade, topology change, incident_ | + +--- + +## Keeping this current + +Most of the numbers in this plan are properties of a Harper version, a data +volume, a topology, and a set of components. All four change, so the numbers +expire. Re-measure and revise after: + +- A Harper version upgrade +- A topology change, including adding or removing nodes +- Significant data growth +- Any incident that produced a surprise + +Date every number. An undated measurement in a reliability plan is worse than a +blank field, because a blank field prompts someone to go and measure. + +## Additional Resources + +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the + gate that populates this plan +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how + to derive sections 3 and 11 +- [Backup and Recovery](./backup-and-recovery.mdx) for section 10 +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for + section 8 +- [Monitoring and Triage](./monitoring-and-triage.mdx) for section 9 diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx new file mode 100644 index 000000000..41310686c --- /dev/null +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -0,0 +1,328 @@ +--- +title: Safe Deployments and Rollback +sidebar_position: 6 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an +immutable artifact onto your cluster from a pipeline. This guide is about what +happens around that call: how to limit who sees a change, what evidence justifies +expanding it, and how to reverse it when the evidence says stop. + +Harper makes reversal unusually cheap, because your application is a component +deployed from an immutable reference, so rolling back is deploying the previous +reference. There is no image to rebuild and no instance to replace. That only +helps if the previous reference is still addressable and someone has done it +before under calm conditions, which is what makes rollback a designed capability +rather than a hope. + +## What You Will Learn + +- How to separate build, deploy, activate, and expose into four decisions with + four different control points +- The change loop, and what counts as evidence at each step +- Five rollout patterns and the Harper operations behind each +- What `"restart": true` and `"restart": "rolling"` actually do, which is not what + most people assume +- How to classify a change so you reverse the right thing, and why reversing the + wrong thing can cause data loss + +## Prerequisites + +- A cluster of at least three nodes, so you can hold one out of rotation and still + meet capacity +- A working pipeline that deploys from an immutable reference + ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), + since every pattern here depends on being able to move traffic +- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here + depends on being able to compare versions + +## Separate four decisions + +Most bad deployments come from collapsing these into one action. + +| Decision | The question | Harper control point | +| ------------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | +| **Build** | What immutable artifact exists? | A pinned package reference, a versioned tarball, a checksum, a dependency lock | +| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `replicated` true or false | +| **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | +| **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | + +Once these are separate you get options that do not exist otherwise. You can +install an artifact on every node and activate it nowhere. You can activate a +route and expose it only to internal traffic. And when something is wrong you can +reverse exposure in seconds without touching what is installed, which is the +fastest containment action available to you. + +`urlPath` mounts a component at an HTTP path. +`host` serves it on a virtual hostname. Both are +persisted on the component's root config entry, so they are part of the deployed +state rather than a runtime toggle. See +[HTTP middleware routing](/reference/v5/http/overview#middleware-routing). + +## The change loop + +1. **Preflight.** Confirm the target version, the component inventory from + `get_components`, `cluster_status` convergence, peer capacity with one node + held out, backup posture if data is at risk, and that the previous known-good + artifact is still addressable. + +2. **Limit.** Choose the smallest cohort that produces useful evidence. One + drained node, an internal cohort, a low-risk geography, a tenant set, or a + small weighted slice. Smaller is better right up until the sample is too small + to distinguish signal from noise. + +3. **Observe.** Compare the new and old versions on the same metrics: request + success, latency at your SLO percentile, worker saturation, logs and traces, + data correctness, replication behavior, and downstream errors. Comparison + against the other cohort is the point. Absolute numbers on the new version tell + you much less. + +4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared + before you started, with a named owner. Missing telemetry is a failed gate, not + a pass. + +5. **Expand.** Increase exposure only after minimum sample and hold conditions + pass, and keep enough healthy capacity in the old version to reverse. + +6. **Close.** Verify uniform artifact and configuration across nodes, restore + intended traffic, record the actual outcome, and keep the evidence with the + change record. + +Declaring thresholds in step 4 before step 2 is the part that gets skipped and the +part that matters. A threshold invented while looking at a live graph is not a +threshold, it is a negotiation, and it always resolves toward shipping. + +## Five rollout patterns + +| Pattern | How Harper is used | Fits when, and watch out for | +| ----------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | +| Isolated node validation | Drain one node, deploy with `"replicated": false`, validate, re-admit a small cohort | Strongest infrastructure and component validation. Requires N+1 capacity and node targeting | +| Replicated rolling deployment | `deploy_component` with `"replicated": true` and `"restart": "rolling"` | Efficient once you trust the artifact. Rolling alone is not progressive delivery, add traffic and telemetry gates | +| Feature-targeted release | Deploy compatible code broadly, activate by flag, header, tenant, or geography | Best when behavior can separate from code placement. Guard flag ownership and cleanup | +| Parallel environment | Deploy to a separate node pool, shift traffic after validation | Strong isolation and fast reversal. Write ownership and replication between pools need real design | +| Regional progression | Promote the same artifact region by region, exposing each after local gates | Good locality and blast radius control. One region does not predict every traffic shape | + +Isolated node validation, concretely: + + + + +```json +{ + "operation": "deploy_component", + "project": "orders-api", + "package": "https://artifacts.example.net/orders-api-2.4.1.tgz", + "replicated": false, + "restart": true +} +``` + +Drain the node first, deploy, run your readiness route and journey synthetic +against it directly, then admit a bounded cohort. + + + + +```json +{ + "operation": "deploy_component", + "project": "orders-api", + "package": "https://artifacts.example.net/orders-api-2.4.1.tgz", + "replicated": true, + "restart": "rolling" +} +``` + +Same immutable reference, now cluster-wide. Poll the returned `restartJobId` with +[`get_job`](/reference/v5/operations-api/operations#get_job), and poll +[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) until +the deployment reports success rather than treating the response as completion. + + + + +Prefer a pinned version or an immutable tarball over a moving branch reference. +A branch moves, so you can neither audit what was running yesterday nor redeploy +it. + +## What restart actually does + +This is worth reading carefully, because the naming invites a wrong assumption. + +**`"restart": true`** restarts the HTTP worker threads on the node handling the +call, and waits for that restart to finish before responding. A successful +response therefore means every worker on that node is serving the new code. Until +a worker has been replaced it is still running the previous code, and on platforms +where replacements share a listening port it keeps accepting connections during +the changeover. The wait follows the restart's own progress rather than a fixed +timeout, so the response can take tens of seconds on a slow install with many +worker threads. + +**`"restart": "rolling"`** does not restart inline. It starts a replicated +`restart_service` job and returns a `restartJobId` for you to poll. Use this when +your caller has a short request timeout. + +Two consequences for your pipeline. A caller with an aggressive HTTP timeout +should use `"rolling"` and poll, because giving up on `true` does not stop the +restart, it just leaves you without the result. And a failed restart does not fail +the deploy: the component is installed and replicated either way, so your pipeline +needs to check both outcomes separately rather than assuming one implies the +other. + +Two parameters worth setting deliberately on replicated deploys: + +- `deployment_timeout` is how long a peer waits + for the replicated payload before failing, defaulting to 120000 ms. Raise it for + large components or slow links. +- `ignore_replication_errors` treats a peer that + fails to receive the deploy as non-fatal. By default a failed peer makes the + whole operation return a non-2xx status, while the component is still deployed + on the origin node. Decide which behavior you want before you need it, because + the default leaves you in a mixed-version state with a failed response, and that + is a confusing thing to reason about mid-incident. + +## Configuration changes are deployments too + +A configuration change carries the same risk as a code change and gets less +ceremony, which is backwards. + +`set_configuration` supports `"replicated": true` +to apply a change across the cluster in one call, with per-node outcomes in the +response. To finish the change cluster-wide, follow with `restart_service` using +`"replicated": true`, which restarts nodes one at a time. + +```json +{ + "operation": "set_configuration", + "logging_level": "info", + "replicated": true +} +``` + +:::danger +Only replicate cluster-appropriate parameters. Node-local values such as ports, +`node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or +`routes` would overwrite every peer's own values. Replicating one of these is a +cluster-wide outage delivered in a single API call. +::: + +Two more things to hold onto. A change takes effect only after a restart, so a +node that has been reconfigured and not restarted is running the old +configuration while reporting the new one. And `get_status` reports a +`restartRequired` flag, but it tracks component and code restarts rather than +configuration changes, so it will not tell you a configuration change is still +pending. Track pending configuration in your change record instead, and read back +`get_configuration` after the restart to confirm. + +## Rollback is a designed capability + +Classify the change before choosing a reversal path, because these have different +compatibility requirements, different authorities, and very different blast +radii: + +| Change type | Reversal | Watch out for | +| ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------- | +| Component release | Deploy the previous immutable reference | The previous artifact must still be addressable | +| Feature behavior | Deactivate the flag, no deploy needed | Fastest reversal available. Requires the flag to have been built in | +| Traffic exposure | Restore prior weights or set the node unavailable | Fastest containment. Does not undo anything already written | +| Configuration | `set_configuration` back, then restart | Needs a restart, so it is not instant | +| Harper runtime version | Version-specific procedure | Mixed versions change replication and deploy behavior | +| Schema or data | Forward repair or restore | See the warning below | + +Practices that make each of these real: + +- **Keep the previous package addressable and rehearse redeploying it** before + launch, not during an incident. +- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, + then remove the old shape in a separate change. Destructive schema behavior is + operation and version dependent, so each destructive change needs an exact + written procedure and verification on every node. +- **Use the same gates for rollback as for forward movement.** Availability, + journey synthetic, peer stability, data validation, traffic reconciliation. A + rollback is a deployment and can fail like one. + +:::warning +If a release changed the meaning of persisted data, code rollback alone will not +fix it, and restoring a database to undo application code is usually the wrong +move. A restore rolls back every write in the window, including all the valid +ones, so it can violate your RPO in order to fix a code bug. Define forward repair +or replay for these cases instead. See +[Backup and Recovery](./backup-and-recovery.mdx). +::: + +### Prove it + +Rehearse a stopped rollout end to end on a non-production cluster: + +1. Declare a threshold before you start, for example "stop if journey success on + the new cohort is more than 0.5 percent below the control cohort over five + minutes." +2. Deploy a component that fails that threshold deliberately, to a bounded cohort. +3. Detect it through your dashboards rather than because you know what you did. +4. Reverse it, and time from decision to restored traffic. That number is your + release RTO, and it belongs in your reliability plan. +5. Verify uniform state afterwards: `get_components` on every node, and + `list_deployments` showing the reversal. + +The step people fail is 5. A partially reversed cluster looks fine on a dashboard +because the healthy majority dominates the average. + +## Operational notes + +- **Version parity is an operating requirement.** Mixed Harper versions in a + cluster change replication and deployment behavior, so a rollout that stalls + halfway is a state you want to detect and exit, not sit in. +- **Keep deploy credentials in your delivery platform's secret store**, use TLS + and least privilege, and retain the operation result as change evidence. See + [secrets](/reference/v5/security/secrets). +- **Component deployment can replicate, so a deploy is a cluster event.** Keep + isolated single-node deployment available for validation, because if the only + deployment path you have is replicated then you have no way to test anything on + one node. +- **Flag cleanup is part of the release.** A feature flag with no owner and no + removal date becomes permanent configuration that nobody understands, and it + will eventually be the thing nobody can explain during an incident. +- **Record who decided, not only what happened.** Named decision authority is what + makes a stop gate function under pressure, and it costs nothing to write down in + advance. + +## Readiness checklist + +- [ ] Artifacts are immutable and referenced by pinned version, never by branch +- [ ] Deploy and expose are separate actions in your pipeline +- [ ] The previous known-good artifact is addressable, and redeploying it has been + rehearsed +- [ ] Cohort ladder defined, from smallest useful sample to full exposure +- [ ] Stop thresholds and hold times declared before the rollout starts +- [ ] A named decision owner for advance, hold, stop, and roll back +- [ ] Pipeline polls `get_deployment` and the restart job separately, rather than + treating the deploy response as completion +- [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately +- [ ] Configuration changes go through the same change record as code +- [ ] `get_configuration` read back after every configuration change +- [ ] Schema changes follow expand-then-contract, with destructive operations + documented per node +- [ ] Forward repair defined for changes that alter persisted meaning +- [ ] Measured release RTO recorded from a rehearsed reversal + +## Additional Resources + +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the + pipeline mechanics and deploy credentials +- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) + for every parameter including `urlPath`, `host`, and replication controls +- [Applications reference](/reference/v5/components/applications) for component + structure and the full component operation set +- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) + for `list_deployments` and `get_deployment` +- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for + `urlPath` and `host` activation +- [Configuration operations](/reference/v5/configuration/operations) for + `set_configuration` and restart requirements +- [Database schema](/reference/v5/database/schema) for schema evolution +- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) + for running more than one component side by side diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx new file mode 100644 index 000000000..b06a86948 --- /dev/null +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -0,0 +1,261 @@ +--- +title: Sizing a Harper Cluster +sidebar_position: 3 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Most clusters get sized by dividing expected peak throughput by measured per-node +throughput and rounding up. That arithmetic produces a cluster that is exactly +large enough to handle a good day, which means the first node failure during peak +traffic becomes a user-visible outage. + +Sizing Harper starts from a different question. Not "how much traffic do we +have," but "how many nodes are we willing to lose at once, and can the survivors +carry the load." This guide gives you the capacity rule that follows from that, +what to include when you measure a node, and how to turn an availability target +into an operating budget you can actually spend. + +## What You Will Learn + +- The capacity invariant that sizes for the failure state, with a worked example +- What to include in a per-node throughput measurement, and the three things + people leave out +- Why maintenance is a planned failure, and what that means for change windows +- How to choose a topology from your objectives rather than from a diagram +- How to convert an availability SLO into minutes per month, and what each tier + demands of your automation + +## Prerequisites + +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your + service boundary inventory +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), + because capacity headroom is meaningless if traffic cannot be moved off a node +- A load generation tool and a non-production cluster you can push to saturation +- A stated peak throughput requirement for your critical journeys + +## Size for the failure state + +The rule: + +```text +(N - F) x tested per-node throughput x target utilization >= required peak throughput +``` + +Where `N` is your node count, `F` is the largest simultaneous node loss the design +tolerates, and target utilization is the fraction of measured capacity you are +willing to run at, which should leave room for latency to stay acceptable rather +than merely for requests to complete. + +Worked example. Suppose your critical journey needs 12,000 requests per second at +peak, a node sustains 5,000 requests per second in your own test with acceptable +latency, you will run at 70 percent utilization, and you want to survive losing +one node: + +```text +Required surviving capacity = 12,000 / 0.7 = 17,143 req/s +Surviving nodes needed = 17,143 / 5,000 = 3.43 -> 4 +N = 4 + F (1) = 5 nodes +``` + +Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded +up to 3, and that cluster degrades the moment anything goes wrong. + +Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not +double the cluster, so tolerating a second simultaneous failure is often cheaper +than people expect at this size. And the utilization factor is doing as much work +as `F` is: sizing to 100 percent of measured capacity means your "surviving" +nodes are at saturation, where latency degrades long before throughput does. + +## Measure a node honestly + +The per-node number in that formula is the one most likely to be wrong, because +benchmark conditions are kinder than production. Include all of this: + +- **Representative application code, data shape, and query mix.** A Harper node + runs your component logic in the same process as the data access, so your code + is part of the capacity measurement in a way it would not be for a standalone + database. A synthetic key-value benchmark tells you very little about the node's + capacity to serve your journey. +- **Realistic downstream latency.** If your resource calls an upstream pricing + service, its latency occupies worker capacity on the node. +- **Replication catch-up load.** A node that is feeding a recovering peer is doing + work that does not appear in its own request metrics. Measure with a peer + synchronizing, because that is precisely the state you will be in when you are + already down a node. +- **Traffic imbalance.** Configured weights express intent. Actual request + distribution is what consumes capacity, and it is rarely even. + +Measure each node separately rather than dividing a cluster total by node count. +An average conceals the one node that is about to tip. + + + + +```bash +curl -s -X POST https://my-node.example.com:9925/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"system_information","attributes":["cpu","memory","threads","harperdb_processes"]}' +``` + + + + +```javascript +await fetch('https://my-node.example.com:9925/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ + operation: 'system_information', + attributes: ['cpu', 'memory', 'threads', 'harperdb_processes'], + }), +}); +``` + + + + +Watch CPU, memory, event loop and worker pressure from the `threads` attribute, +disk, and network, and pair them with request rate and latency from +[analytics](/reference/v5/analytics/overview). The number you want is not the +point where requests start failing. It is the point where your journey's latency +leaves your SLO, which arrives earlier. + +:::tip +Record the conditions alongside the number: Harper version, node size, data +volume, query mix, and whether a peer was synchronizing. A per-node capacity +figure without its conditions is not reusable, and six months later nobody will +remember whether the test included replication load. +::: + +## Maintenance is a planned failure + +Draining a node for a deploy, an upgrade, or an investigation consumes exactly the +same headroom that a node failure does. So a cluster sized for `F = 1` is running +at `F = 0` for the duration of every maintenance window, with no tolerance left. + +The operating rule that follows: if draining one node means the service can no +longer tolerate its declared `F`, pause the change. Either wait for lower traffic, +or add capacity for the window, or accept and document the reduced tolerance for a +bounded period. What you should not do is treat the maintenance window as free +because nothing is technically broken. + +This is also the argument for sizing `F` at 2 in a cluster that deploys +frequently. It is not paranoia about correlated hardware failure. It is that you +want to be able to deploy during business hours and still survive an incident. + +## Choose a topology from objectives + +Pick the pattern that matches the failures you actually need to survive, and know +what you have to prove for it to count. + +| Pattern | When it fits | What you must prove | +| ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| Single region, multiple peers | Node and host failures dominate, and low inter-node latency matters | `N - F` capacity under load, peer catch-up time, traffic removal, backup independence | +| Multiple failure domains, one region | Zone or rack isolation is available from your provider | No shared power or network fault, acceptable replication latency across domains, balanced traffic | +| Multiple regions | Regional impairment or user locality is a real requirement | Data authority per region, route propagation time, convergence, degraded-mode behavior | +| Edge or on-premises peers | Data locality or residency constrains where data can live | Replication scope per location, identity provisioning, backup path from each site | + +The right-hand column is the useful one. A topology diagram is a claim, and the +claim is only true once you have exercised it. "Multiple failure domains" means +nothing if both domains draw from the same power feed, and you will not discover +that from your provider's documentation. + +Note that adding a node is a data movement event as well as a capacity event, +since a Harper node carries data along with request handling. A node whose +databases have never synchronized downloads them in full before it is useful, so +scaling out is not instantaneous and cannot be your response to an unexpected +traffic spike. Size ahead of demand. + +## Turn the SLO into a budget + +An availability target is a quantity of unavailability you are permitted to spend +per month. Written that way it becomes an operating constraint rather than an +aspiration. + +| Monthly SLO | Approximate maximum unavailability | What it demands | +| ----------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 99.9% | 43 min 50 sec | Manual diagnosis can fit inside some incidents, but one untested maintenance window can consume the month | +| 99.95% | 21 min 55 sec | Automated node removal and a rehearsed release reversal become necessary | +| 99.99% | 4 min 23 sec | Node failure must be close to transparent. Detection, traffic removal, and validation have to be automated, because no human response fits in the budget | + +Figures assume a 30.44 day month. Your contractual definition, exclusions, and +measurement boundary may differ, and the boundary matters more than the number: +availability measured at your CDN edge and availability measured at the Harper +node are different quantities. + +The practical consequence is that the SLO tier determines how much automation you +need, not how many nodes. At three nines a person can be paged, look at a +dashboard, and drain a node. At four nines that same sequence has already spent +the entire month's budget. + +### Prove it + +Run the failure state rather than calculating it. On a cluster carrying +representative load at your expected peak: + +1. Record per-node request rate, latency, and saturation as a baseline. +2. Drain one node using the sequence from + [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +3. Measure what the survivors do: request rate, latency at your SLO percentile, + CPU and worker pressure, and whether any user-visible errors appeared during + the transition. +4. While still down a node, bring the drained node back and let it synchronize, so + the survivors are carrying peak traffic and feeding a recovering peer at the + same time. This is the real worst case and it is the one nobody tests. +5. Compare measured surviving capacity against your invariant. If step 4 pushed + latency out of SLO, your target utilization is too high or your `N` is too low. + +## Operational notes + +- **Utilization targets belong per journey, not per cluster.** A write-heavy + journey and a cached read journey consume very different resources on the same + node, so a single cluster-wide utilization figure will be wrong for both. +- **Re-measure after a version upgrade.** Per-node capacity is a property of a + Harper version, your component code, and your data volume. All three change. +- **Watch measured distribution, not configured weights.** Per-node request counts + from analytics are the ground truth. A misconfigured weight, a sticky session + policy, or DNS caching can leave one node doing far more work than the topology + claims. +- **Storage growth is a capacity dimension too.** Disk headroom, compaction + behavior, and backup space all scale with data volume rather than with request + rate, so they need their own thresholds. See + [compaction](/reference/v5/database/compaction). +- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are + still choosing `N` and living with `F`. + +## Readiness checklist + +- [ ] `F` declared explicitly, and written down where change operators will see it +- [ ] Per-node throughput measured with your own application code and data shape +- [ ] Measurement taken with a peer synchronizing, not on an idle cluster +- [ ] Target utilization set from latency at your SLO percentile, not from request + success +- [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers +- [ ] Measurement conditions recorded alongside the capacity figure +- [ ] Maintenance policy states what happens when a drain would breach `F` +- [ ] Topology's "what you must prove" column exercised, not assumed +- [ ] SLO converted to minutes per month, with the measurement boundary named +- [ ] Peak-load drain and rejoin exercise completed and dated + +## Additional Resources + +- [Analytics overview](/reference/v5/analytics/overview) and + [analytics operations](/reference/v5/analytics/operations) for per-node request + and latency metrics +- [Grafana integration](/fabric/grafana-integration) for dashboards on Fabric +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) + for self-managed metrics scraping +- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and + throughput trade-offs available per database +- [Compaction](/reference/v5/database/compaction) for storage growth behavior +- [Replication overview](/reference/v5/replication/overview) for peer + synchronization behavior when adding or returning a node +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for + scenario-level targets built on this capacity model diff --git a/redirects.ts b/redirects.ts index c6eb20368..7e06c8f3b 100644 --- a/redirects.ts +++ b/redirects.ts @@ -358,6 +358,7 @@ const currentRedirects: RedirectRule[] = [ // ── Learn ───────────────────────────────────────────────────────────────── { from: '/learn/developers/coming-soon', to: '/learn' }, + { from: '/learn/administration/coming-soon', to: '/learn/administration/how-harper-runs-in-production' }, // ── Fabric ──────────────────────────────────────────────────────────────── { from: '/fabric/rest-api', to: '/fabric' }, From 2404a11f40001412a6adcce870965b2c46b972fa Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 08:58:55 -0600 Subject: [PATCH 2/7] docs: correct Administration guides against harper/harper-pro source Technical verification pass over the ten Administration guides against harper and harper-pro at origin/main (v5.2.12 current). Most of the track held up; these are the claims that did not, plus the reference fixes they turned up. Guides: - Rewrite "What restart actually does" for released 5.2 behavior. The awaited `restart: true` is v5.3.0 only, so the section now describes the fire-and-forget behavior operators actually have, with the wait moved to its own subsection badged `changed in v5.3.0`. - Warn that a replicated deploy with `restart: true` restarts every node at once, and prefer `rolling` on a cluster. Matches the guidance already in the shipped CI/CD guide. - Document that `replicated` is opt-out: omitting it replicates. - Correct the new-node row. A joining node requests a full copy from every peer it has no resume cursor for, and established peers full-copy from it in turn, so a join into N peers is N inbound plus N outbound transfers. - `system` is in replication scope by default, so users and roles already propagate unless someone narrowed it. - `drop_database` and `drop_table` replicate by default; `drop_attribute` does not. - `get_backup target=` writes a file; it does not clone a database. - Blob snapshots are hard-linked, so they are near-free on the default layout and a full copy only across filesystems. The transaction-log snapshot is the part that genuinely costs disk every backup. - `verify_backup` checks that a blob snapshot is present, not that blob contents are intact. - The availability flag is already `replicate: false` in both mechanisms, so the guidance is to preserve that rather than to go establish it. - `/status` returns a body as well as a status code, and the component entry belongs in the root harper-config.yaml, not an application's config.yaml, where it would replace the default component config. - Readiness sample: a missing record resolves rather than throws, so check the result; import from 'harper'; and note the route is unauthenticated because a static handler bypasses the authorization gate. - Logging sample: there is no node-name environment variable, so use server.hostname. `context.requestId` needs http.logging.id, which is off by default. Harper's logger renders text via Node's Console, not JSON. - Badge `write-transaction-queue-depth` v5.2.0, and note the Prometheus exporter does not cover the saturation, commit-time, or storage metrics. Reference: - State the `replicated` default across the component operations; it was the only optional parameter whose default went unstated. - `drop_database`/`drop_table` replicate by default; only `drop_attribute` is per-node. - Users and roles do propagate by default, which contradicted both the code and this file's own statement that all databases replicate by default. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 46 +++++--- .../health-checks-and-traffic-admission.mdx | 67 ++++++++--- .../administration/monitoring-and-triage.mdx | 61 +++++++++- .../administration/operating-replication.mdx | 77 ++++++++++--- .../production-readiness-checklist.mdx | 3 +- .../safe-deployments-and-rollback.mdx | 108 +++++++++++++----- reference/components/applications.md | 10 +- reference/operations-api/operations.md | 6 +- reference/replication/overview.md | 9 +- 9 files changed, 298 insertions(+), 89 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 54bfa9f88..8e0a286ad 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -123,12 +123,22 @@ references. :::warning The incremental behavior applies to the RocksDB data files only. **The -transaction-log snapshot, and the blob snapshot for a database with file-backed -blobs, are copied in full on every backup.** With a large audit-retention window -or many blobs, frequent backups cost considerably more disk than the -data-only view suggests. Pass `exclude_blobs: true` to skip blobs when that is -appropriate, and size your backup volume against the full cost rather than the -incremental one. +transaction-log snapshot is copied in full on every backup.** With a large +audit-retention window that is real, recurring disk cost that the data-only view +hides, so size your backup volume against it rather than against the incremental +figure. + +Blobs behave differently and are easy to over-budget. Each backup captures a full +set of blobs, but the files are hard-linked rather than copied when the backup +directory and the blob storage share a filesystem, which is the default layout. In +that case the extra space is near zero. They become genuine full copies only when +the two live on different filesystems, so putting backups on a separate volume is +a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip +blobs when that is appropriate. + +One consequence for disaster recovery: because the blob snapshot is hard-linked, +copying a backup repository with an ordinary recursive copy materializes every +blob as a full, separate file at the destination. ::: Do not use `list_backups` sizes for capacity planning. The `size` and @@ -162,11 +172,16 @@ retained managed backups off the node: # Pull a snapshot of the current state from a running node harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz -# Or pull from another node, which also clones that node's database onto this one +# Or pull a snapshot of another node, writing it to a file here harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` -Note that `get_backup` always streams the current state. It cannot download a +`target=` only changes which node the snapshot is read from. It writes a file and +nothing else: it does not restore, import, or clone that node's database onto the +local one. Turning the file into a live database is a separate, deliberate restore +step, performed with Harper stopped. + +Note also that `get_backup` always streams the current state. It cannot download a historical managed backup, so it is a way to take a fresh off-host copy, not a way to export your retention history. @@ -189,8 +204,11 @@ nothing else. `verify_checksum` is `true`, which is slower, and the framing of the transaction-log snapshot, which is always checked. -**The blob snapshot is not verified.** If your database has file-backed blobs, -verification does not tell you they are intact, and only a real restore does. +**Blob contents are not verified.** Verification confirms that a backup which +recorded blobs still has its blob snapshot present, and fails the backup as corrupt +if that snapshot is missing. It does not check the individual blob files for size, +checksum, or readability. So verification can pass on a backup whose blobs are +damaged, and only a real restore proves they are intact. More generally, a verified backup is a well-formed backup, not a proven recovery. The only evidence that your recovery works is a restore you have actually @@ -302,15 +320,17 @@ that it is the target than a promise you cannot keep. - [ ] No database in scope uses per-table storage paths, or its exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database -- [ ] Backup volume sized against the full cost, including non-incremental - transaction-log and blob snapshots +- [ ] Backup volume sized against the non-incremental transaction-log snapshot, + and against full blob copies if the backup and blob paths are on different + filesystems - [ ] An off-host copy exists in a destination that does not share a failure domain - [ ] Managed repository copies take the whole `/` directory, quiesced or from an atomic snapshot - [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least periodically -- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms + only that the blob snapshot is present - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index 249e9ddf6..fcd9524b3 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -67,21 +67,36 @@ other component: } ``` -Or declare it in your application's `config.yaml`, which is the better option -because it makes the availability contract part of your deployed artifact rather -than a manual step someone has to remember on a new node: +Or declare it in the root `harper-config.yaml`, so the component is part of the +node's configuration rather than something an operator has to remember to deploy: ```yaml status-check: package: '@harperdb/status-check' ``` -Once deployed, the route answers with a status code rather than a body: +:::note +That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, +typically `~/hdb`), not in an application's own `config.yaml`. The two files look +alike but behave differently: in the root config the entry name is free-form, while +in a component's `config.yaml` it must match a `package.json` dependency. A +component `config.yaml` also **replaces** Harper's default component +configuration outright instead of merging with it, so a file containing only this +entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and +`fastifyRoutes` defaults your application relies on. See +[applications](/reference/v5/components/applications). +::: + +Once deployed, the route's contract is its status code, which is what lets a load +balancer consume it without parsing anything: - `GET /status` returns `200` when the node is available, `404` when it is not - `POST /status` marks the node available (authenticated) - `DELETE /status` marks the node unavailable (authenticated) +It does also return a body, which is useful when you are checking by hand: a short +message on `200`, and an RFC 9457 problem-details document on `404`. + @@ -118,11 +133,16 @@ Operations API and not at your application's root. A `404` is the node telling t traffic layer to stop sending work, and it will keep saying so until something sets it back. -:::warning -Whatever mechanism you use to persist the availability flag, keep it out of -replication scope. A flag that replicates would let one node's maintenance state -propagate to its peers, which turns a routine drain into an outage. If you are -scoping replication by hand, confirm this explicitly rather than assuming it. See +:::note +The availability flag is node-local out of the box. The component stores it in a +table declared `replicate: false`, and the Operations API's own status values are +stored the same way, so neither propagates to peers. + +That is the property you want, and it is worth knowing why: a flag that replicated +would let one node's maintenance state reach its peers, turning a routine drain +into a cluster-wide outage. So the rule is to preserve it rather than to establish +it. If you fork the component or persist the flag some other way, keep +`replicate: false` on whatever holds it. See [Operating Replication](./operating-replication.mdx). ::: @@ -154,15 +174,19 @@ converted a partial degradation into a full outage yourself. Add a resource to your application's `resources.js`: ```javascript +import { Resource, tables } from 'harper'; + export class Readiness extends Resource { static async get() { const checks = {}; // A bounded read against the table this journey serves. // Keep it to a single primary-key lookup, never a scan. + // A missing record resolves to undefined rather than throwing, + // so check the result as well as catching a storage fault. try { - await tables.Product.get('readiness-probe-sentinel'); - checks.data = 'ok'; + const sentinel = await tables.Product.get('readiness-probe-sentinel'); + checks.data = sentinel ? 'ok' : 'failed'; } catch (error) { checks.data = 'failed'; } @@ -187,11 +211,25 @@ export class Readiness extends Resource { } ``` -Enable the `jsResource` plugin in `config.yaml` if it is not already, and the -route is served at `/Readiness` on the application port. See +The `jsResource` plugin is enabled by default, so the route is served at +`/Readiness` on the application port as soon as the class is exported. See [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for the resource and export mechanics. +:::note +Because the handler is a `static` method, it replaces Harper's built-in dispatch +along with the authorization check that lives inside it, so `GET /Readiness` is +unauthenticated. That is what a load balancer probe needs, and it is why the +endpoint must not return anything you would not publish. Keep the response to +check names and outcomes, never connection strings, credentials, or internal +hostnames. + +If you rewrite this as an instance `get()` instead, authorization comes back and +defaults to `super_user` only, which will make your probe start failing with an +authorization error rather than a readiness one. In that form you need +`allowRead() { return true; }` to keep it reachable. +::: + Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version @@ -288,7 +326,8 @@ than people assume, because it includes convergence rather than just startup. - [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than deployed by hand - [ ] Traffic layer health check points at `GET /status` on `9926` -- [ ] Availability flag confirmed to be outside replication scope +- [ ] Availability flag storage still declares `replicate: false`, if you forked the + component or persist it yourself - [ ] An application readiness route exists, scoped to one journey's dependencies - [ ] Every downstream call in the readiness route has a timeout - [ ] Readiness response includes the component version diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index f25621455..7549ccc58 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -65,7 +65,7 @@ dashboard. This much belongs on a pager: | Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | | Replication convergence lag | `replication-latency`, or the receive-time gap in `cluster_status` | A node serving stale data looks healthy on every other signal | | Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | -| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | +| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | | Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | | Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | | Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | @@ -102,10 +102,30 @@ and scrape it into whatever you already run: "operation": "deploy_component", "project": "prometheus-exporter", "package": "@harperdb/prometheus-exporter", - "restart": true + "restart": "rolling" } ``` +Once deployed, the scrape endpoint is `//metrics` on the application +port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is +authorized, so give your scraper a credential rather than expecting it to be open. + +:::warning +**The exporter does not cover every metric in the alert table above.** It +translates the request and replication metrics, including `success`, `duration`, +and `replication-latency`, but it does not currently export +`main-thread-utilization` and its `taskQueueLatency` attribute, +`transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, +`storage-volume`, or `table-size`. The `utilization` it does expose is a thread +utilization figure drawn from `system_information`, which is not the same as the +analytics `utilization` metric. + +For the saturation, commit-time, and storage rows you will need to query +`hdb_analytics` directly or collect `system_information` on your own schedule. +Check what your version actually exports before you build a dashboard on the +assumption that everything above arrives in Prometheus. +::: + @@ -138,14 +158,15 @@ to discover it after building panels on the wrong assumption. ## Logs you can use during an incident -Harper's `logger` global writes structured entries from component code: +Harper's `logger` global takes a message plus a context object from component +code: ```javascript logger.info('order submitted', { component: 'orders-api', - version: process.env.APP_VERSION, - node: process.env.HARPER_NODE_NAME, - requestId: context.requestId, + version: process.env.APP_VERSION, // whatever your build injects + node: server.hostname, + requestId: context.requestId, // requires http.logging.id, see below operation: 'submit', durationMs: elapsed, outcome: 'ok', @@ -158,6 +179,34 @@ is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident. +Three details in that example need care: + +- **The node name comes from `server.hostname`**, not an environment variable. + Harper does not set a node-name variable in the process environment, and + `server.hostname` is the same identity analytics uses, so it is what correlates + with your metrics. +- **`context.requestId` is only populated when `http.logging.id` is enabled**, and + HTTP request logging is off by default. Without it the field is `undefined` and + you silently lose your correlation id. Either enable it in + [logging configuration](/reference/v5/logging/configuration) or generate an id + in your own code. +- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your + deployment so the log line can name the build. + +:::warning +Harper's logger is built on Node's `Console`, so this renders as a formatted text +line, not JSON. The context object is inspected into the message rather than +emitted as separate fields: + +```text +[main/3] [info]: order submitted { component: 'orders-api', version: '2.4.1', ... } +``` + +That is fine for reading during an incident, but a log pipeline cannot key on +`component` or `outcome` without parsing the line. If you need queryable fields, +serialize the context yourself and log a single JSON string. +::: + Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged regardless of level. Choose levels deliberately: a production log at `debug` is a log nobody can read, and one diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 617f5c53f..93baa86c7 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -55,7 +55,7 @@ replication: ```graphql type LocalTableForNode @table(replicate: false) { - id: ID! + id: ID! @primaryKey name: String! } ``` @@ -101,29 +101,32 @@ directional configuration mistakes survive into production. :::warning Whether the `system` database is in your replication scope is the highest-stakes scoping question in a Harper cluster, because `system` holds `hdb_user`, -`hdb_role`, and `hdb_nodes`. Users and roles propagate only when `system` is -replicated, and a node that receives `system` must be trusted with its contents, -including encrypted secret rows. Confirm your own answer from `cluster_status` and -your configuration, and read +`hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default +replication scope is every database, so unless someone has narrowed it your users +and roles already propagate and every node that receives `system` must be trusted +with its contents, including encrypted secret rows. That is usually what you want, +but it should be a decision rather than a surprise. Confirm your own answer from +`cluster_status` and your configuration, and read [replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) before changing it. ::: ## What moves automatically, and what does not -| Object or change | Behavior | What it means for you | -| --------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | -| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | -| A brand new node | Downloads the full database from the first node it connects to, then enters incremental replication | Budget the transfer, and expect load on the source node while it happens | -| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | -| Component deployment | `deploy_component` supports `"replicated": true` for cluster-wide propagation | Useful once an artifact is trusted. Keep single-node deployment available for validation | -| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | -| Users and roles | Propagate only when the `system` database is in replication scope | If `system` is out of scope, identity must be provisioned on every node by your own automation | -| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | -| Destructive schema changes | Behavior is operation and version dependent | Treat as high risk. Write an exact procedure, and verify the result on every node | - -The two rows most likely to bite you are the returning node and the node registry. +| Object or change | Behavior | What it means for you | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | +| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | +| A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | +| Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | +| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | +| Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | +| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | +| Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | + +The three rows most likely to bite you are the returning node, the new node, and +the node registry. On the returning node: it is common to see a routine restart budgeted as though it were a full resynchronization, which makes teams avoid restarts they should be @@ -131,6 +134,21 @@ comfortable with. A node whose databases have never synced does download them in full. A node that was briefly offline catches up on what it missed. Measure both on your own data volume once, and use the right number for the right situation. +On the new node, the cost is larger than "one copy from one peer," because the +full-copy decision is made per peer and per database rather than once against a +bootstrap source. A joining node requests a full copy from every peer it has no +resume cursor for. And because replication is bidirectional, each established peer +independently decides it has no cursor for the newcomer and requests a full copy +_from_ it as well. Joining a cluster of N peers is therefore N inbound transfers +plus N outbound transfers of the new node's own (empty) databases, not a single +stream from one source. + +The practical consequences are worth planning around. Load lands on every existing +peer at once rather than on one, so a join during peak traffic is a capacity event +for the whole cluster. If you want a single designated source instead, `add_node` +accepts `isLeader: true`, which tells the joining node to request its full copy +from that peer alone. + On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads @@ -195,6 +213,21 @@ it is the one that tells you a returning node is not ready for traffic yet. `sendingMessage` appears while a transaction is actively being sent and is absent when the socket is idle, so its absence is not a fault. +Two things will mislead you if you automate on that pair without knowing them. +The gap only means "behind" while transactions are actually arriving: on an idle +source both stamps freeze, and the gap sits at whatever constant it last reached +rather than signalling lag. And because the two stamps come from two different +clocks, skew between peers is added to the gap directly, so calibrate your +threshold against a healthy baseline rather than treating the raw number as +replication delay. + +:::tip +While a database is taking a full copy, these timing fields render as the literal +string `"Copying"` rather than a date. Anything that parses them as timestamps +will fail on a joining node, which is exactly the node you most want to be +watching. Handle that value explicitly. +::: + Inventory `system.hdb_nodes` alongside this and compare it to your intended topology. Configuration intent and live peer state should agree, and the node's own row is in there too, not just its peers. @@ -206,7 +239,7 @@ own row is in there too, not just its peers. - Sustained growth in `latency`, judged against your own baseline - `lastCommitConfirmed` not advancing on a peer while writes are occurring - A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your - admission budget + admission budget, evaluated only while writes are flowing - Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy - Version or configuration drift between peers @@ -242,6 +275,12 @@ continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard. +Note that `replication-latency` is not recorded for the `system` database. Since +`system` is in replication scope by default and carries your users and roles, a +dashboard built only on this metric will show nothing for the database that +propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, +if you need to watch `system` convergence. + ## Consistency belongs in the application design Replication makes data available on every peer. It does not make every peer agree diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 29cabc5aa..0cbeac7f7 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -72,7 +72,8 @@ From - [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than by hand - [ ] Traffic layer health check points at `GET /status` on the application port -- [ ] Availability flag confirmed to be outside replication scope +- [ ] Availability flag storage still declares `replicate: false`, if you forked the + component or persist it yourself - [ ] Readiness route scoped to one journey's dependencies, with a timeout on every downstream call - [ ] Readiness response includes the component version diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 41310686c..92e62b3e4 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -47,7 +47,7 @@ Most bad deployments come from collapsing these into one action. | Decision | The question | Harper control point | | ------------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | | **Build** | What immutable artifact exists? | A pinned package reference, a versioned tarball, a checksum, a dependency lock | -| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `replicated` true or false | +| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `"replicated": false` to hold it to one node | | **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | | **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | @@ -57,6 +57,19 @@ route and expose it only to internal traffic. And when something is wrong you ca reverse exposure in seconds without touching what is installed, which is the fastest containment action available to you. +:::danger +**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no +`replicated` field replicates to every peer. So does `add_component`, +`drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and +the destructive `drop_database` and `drop_table`. Only `"replicated": false` +changes anything; `"replicated": true` is the default spelled out. + +This is the opposite of what most operators assume, and it is why the single-node +validation pattern below has to say `false` explicitly. If you intend to touch one +node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) +is the deliberate exception: it replicates only when you ask.) +::: + `urlPath` mounts a component at an HTTP path. `host` serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed @@ -137,10 +150,13 @@ against it directly, then admit a bounded cohort. } ``` -Same immutable reference, now cluster-wide. Poll the returned `restartJobId` with -[`get_job`](/reference/v5/operations-api/operations#get_job), and poll -[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) until -the deployment reports success rather than treating the response as completion. +Same immutable reference, now cluster-wide. The deploy itself is finished when the +call returns, so the thing still outstanding is the restart: poll the returned +`restartJobId` with +[`get_job`](/reference/v5/operations-api/operations#get_job). Read +[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the +per-peer detail the response summarizes away, including which peers received the +artifact and which failed. @@ -149,29 +165,66 @@ Prefer a pinned version or an immutable tarball over a moving branch reference. A branch moves, so you can neither audit what was running yesterday nor redeploy it. +:::tip +A pinned registry version such as `@my-org/orders-api@2.4.1` is the +best-travelled form. If you pin a tarball URL instead, note that a bare `https://` +URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git +clone rather than a download, so a GitHub release asset URL will not behave the +way the example above does. Host release tarballs somewhere neutral, or use a +registry version. +::: + ## What restart actually does This is worth reading carefully, because the naming invites a wrong assumption. -**`"restart": true`** restarts the HTTP worker threads on the node handling the -call, and waits for that restart to finish before responding. A successful -response therefore means every worker on that node is serving the new code. Until -a worker has been replaced it is still running the previous code, and on platforms -where replacements share a listening port it keeps accepting connections during -the changeover. The wait follows the restart's own progress rather than a fixed -timeout, so the response can take tens of seconds on a slow install with many -worker threads. - -**`"restart": "rolling"`** does not restart inline. It starts a replicated -`restart_service` job and returns a `restartJobId` for you to poll. Use this when -your caller has a short request timeout. - -Two consequences for your pipeline. A caller with an aggressive HTTP timeout -should use `"rolling"` and poll, because giving up on `true` does not stop the -restart, it just leaves you without the result. And a failed restart does not fail -the deploy: the component is installed and replicated either way, so your pipeline -needs to check both outcomes separately rather than assuming one implies the -other. +**`"restart": true`** starts a restart of the HTTP worker threads on the node +handling the call and returns immediately, without waiting for it. A `200` means +the deploy succeeded and a restart has been requested. It does not mean the new +code is serving yet. Until a worker has been replaced it is still running the +previous code, and on platforms where replacements share a listening port it keeps +accepting connections during the changeover. + +**`"restart": "rolling"`** does not restart inline either, but it is observable. +It starts a replicated `restart_service` job and returns a `restartJobId` you can +poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job +walks the cluster **one node at a time**, waiting for each node to come back +before starting the next, so the cluster keeps serving throughout. + +:::warning +**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to +every peer, and peers are dispatched in parallel, so a replicated deploy with +`"restart": true` restarts every node in the cluster at roughly the same moment. +That is a cluster-wide availability event, not a node-local one. Reserve +`"restart": true` for a single-node or development instance, and use `"rolling"` +anywhere you care about staying up. See +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). +::: + +Two more consequences for your pipeline. Neither restart mode reports completion +in the deploy response, so if you need to know the new code is live, poll the +rolling restart job or probe the nodes themselves. And a failed restart does not +fail the deploy: the component is installed and replicated either way, so your +pipeline needs to check both outcomes separately rather than assuming one implies +the other. + +### Waiting for the restart + + + +From v5.3.0, `"restart": true` waits for the worker restart to finish before +responding, rather than returning as soon as it has been requested. The wait +follows the restart's own progress rather than a fixed timeout, so the call can +take tens of seconds on a slow install with many worker threads, with a hard +ceiling of ten minutes. + +Two things do not change, and both matter more than the wait itself. A caller that +gives up early does not stop the restart, it only loses the result. And the +response still does not carry the restart's outcome: a restart that stalls, times +out, or leaves workers on the old code is reported in the node's log, not to you. +So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and +confirm the version actually serving through your readiness route or journey +synthetic rather than through the deploy response. Two parameters worth setting deliberately on replicated deploys: @@ -299,8 +352,11 @@ because the healthy majority dominates the average. - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before the rollout starts - [ ] A named decision owner for advance, hold, stop, and roll back -- [ ] Pipeline polls `get_deployment` and the restart job separately, rather than - treating the deploy response as completion +- [ ] Pipeline polls the restart job rather than treating the deploy response as + proof the new code is serving +- [ ] `"replicated": false` used deliberately wherever a change is meant to reach + one node only, since replication is the default +- [ ] Cluster deploys use `"restart": "rolling"`, not `"restart": true` - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change diff --git a/reference/components/applications.md b/reference/components/applications.md index 9c1f7477e..a376b10de 100644 --- a/reference/components/applications.md +++ b/reference/components/applications.md @@ -327,7 +327,7 @@ Creates a new component project in the component root directory using a template - `install_command` _(optional)_ — Install command. Defaults to `npm install` - `install_timeout` _(optional)_ — Install timeout in milliseconds. Defaults to `300000` (5 minutes) - `install_allow_scripts` _(optional)_ — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only ```json { @@ -345,7 +345,7 @@ Deploys a component using a package reference or a base64-encoded `.tar` payload - `payload` _(optional)_ — Base64-encoded `.tar` file content - `force` _(optional)_ — Allow deploying over protected core components. Defaults to `false` - `restart` _(optional)_ — `true` for immediate restart, `'rolling'` for sequential cluster restart -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only - `install_command` _(optional)_ — Install command override - `install_timeout` _(optional)_ — Install timeout override in milliseconds - `install_allow_scripts` _(optional)_ — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). @@ -366,7 +366,7 @@ Deletes a component project or a specific file within it. - `project` _(required)_ — Project name - `file` _(optional)_ — Path relative to project folder. If omitted, deletes the entire project -- `replicated` _(optional)_ — Replicate deletion to all cluster nodes +- `replicated` _(optional)_ — Replicate the deletion to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only - `restart` _(optional)_ — Restart Harper after dropping ```json @@ -425,7 +425,7 @@ Creates or updates a file within a component project. - `file` _(required)_ — Path relative to project folder - `payload` _(required)_ — File content to write - `encoding` _(optional)_ — File encoding. Defaults to `utf8` -- `replicated` _(optional)_ — Replicate update to all cluster nodes +- `replicated` _(optional)_ — Replicate the update to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only ```json { @@ -447,7 +447,7 @@ For deploying from private repositories, SSH keys must be registered on the Harp - `host` _(required)_ — Host alias for SSH config (used in `package` URL) - `hostname` _(required)_ — Actual domain (e.g., `github.com`) - `known_hosts` _(optional)_ — Public SSH keys of the host. Auto-retrieved for `github.com` -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only ```json { diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index ecacbc948..f7d54459f 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -116,7 +116,7 @@ Creates a new database. ### `drop_database` -Drops a database and all its tables/records. Supports `"replicated": true` to propagate to all cluster nodes. +Drops a database and all its tables/records. **Replicates to all cluster nodes by default** — pass `"replicated": false` to drop it on this node only. ```json { "operation": "drop_database", "database": "dev" } @@ -137,7 +137,7 @@ Creates a new table. Optional fields: `database` (defaults to `data`), `attribut ### `drop_table` -Drops a table and all associated records. Supports `"replicated": true`. +Drops a table and all associated records. **Replicates to all cluster nodes by default** — pass `"replicated": false` to drop it on this node only. ```json { "operation": "drop_table", "database": "dev", "table": "dog" } @@ -863,7 +863,7 @@ Detailed documentation: [Components Overview](../components/overview.md) ### `deploy_component` -Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`. +Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Replicates to all cluster nodes by default; pass `"replicated": false` to deploy to this node only. Supports `"restart": true` or `"restart": "rolling"`. Additional parameters: diff --git a/reference/replication/overview.md b/reference/replication/overview.md index bbe091ba6..1015ec7c2 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -360,9 +360,14 @@ The following data operations are replicated across the cluster: - Delete - Bulk loads (CSV data load, CSV file load, CSV URL load, import from S3) -**Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. +**Destructive schema operations differ from each other, so check before you run one:** -Users and roles are not replicated across the cluster by default. They do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication; as of v5.2 this no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). +- `drop_database` and `drop_table` **replicate to the whole cluster by default**, like the other operations that accept a `replicated` flag. Pass `"replicated": false` to drop on one node only. +- `drop_attribute` is not replicated and must be run on each node independently. + +Note that `replicated` is opt-out rather than opt-in across the operations that support it: omitting the flag replicates. [`set_configuration`](../configuration/operations.md#set-configuration) is the deliberate exception, replicating only when you pass `"replicated": true`, because configuration bodies routinely carry node-local values. + +Users and roles **do** propagate by default. They live in the `system` database, and the default replication scope is every database, so `system` is in scope unless you have narrowed it. As of v5.2 replicating `system` no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). If you narrow the scope to exclude `system`, users and roles stop propagating and must be provisioned on every node by your own automation. Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. From eb355357fe13206d8cd1daee49e03173ea573fd5 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 09:18:46 -0600 Subject: [PATCH 3/7] docs(reference): correct six claims contradicted by harper source Follow-on from the Administration verification pass. Each of these was checked against harper/harper-pro at origin/main and is wrong on the live site today. - `set_status` / `get_status` / `clear_status` are not in-memory. They persist to the `hdb_status` table in the `system` database, which is declared `replicate: false`, so values survive a restart and stay node-local. Adds what `restartRequired` on a bare `get_status` actually tracks. - `system_information` no longer advertises a `replication` attribute, which does not exist. Lists the ten real attribute names and notes that unrecognized names are dropped silently rather than rejected, so a typo returns a smaller response instead of an error. - The `cluster_status` sample showed `lastSendTime`, a field present in neither repo, and `thread_id` where the code emits `threadId`. Also notes that the timing fields hold the literal string "Copying" during an initial full copy, which breaks anything parsing them as dates. - `logging.console` defaults to `false`, not `true`. The two reference pages disagreed; the runtime default is `logOptions.console ?? false`. Core's own config schema description carries the same stale "Default: true" and should be fixed separately. - `get_backup target=` reads a snapshot from another node into a local file. It does not clone that node's database onto the current one. - `verify_backup` confirms a recorded blob snapshot is present and fails the backup as corrupt when it is missing, but does not check individual blob files, so verification can pass on damaged blob contents. Co-Authored-By: Claude Opus 5 --- reference/backups/operations.md | 2 +- reference/backups/overview.md | 2 +- reference/configuration/options.md | 2 +- reference/operations-api/operations.md | 28 +++++++++++++++++--------- reference/replication/clustering.md | 6 ++++-- 5 files changed, 25 insertions(+), 15 deletions(-) diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 2e04b33db..48bbd27fe 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -68,7 +68,7 @@ harper list_backups database=data -Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). The blob snapshot is not verified. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. +Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). For a backup that recorded blobs, verification confirms the blob snapshot is still present and fails the backup as corrupt if it is missing, but it does not check individual blob files for size, checksum, or readability — so verification can pass on a backup whose blob contents are damaged. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. ```json { "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 3a0ab21dd..4db448213 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -93,7 +93,7 @@ For a RocksDB database the stream is a `tar` archive, gzipped by default. It con ```bash harper get_backup database=data out=./data.tar.gz -# or pull from another node — this clones that node's database onto the current one +# or read the snapshot from another node, still writing it to a file here harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 61adfbf68..adb717edd 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -212,7 +212,7 @@ logging: - `root` — Log directory; _Default_: `/log` - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` -- `console` — Include `console.*` output; _Default_: `true` +- `console` — Include `console.*` output in the log file; _Default_: `false` - `auditLog` — Enable table transaction audit logging; _Default_: `false` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index f7d54459f..8e85a499f 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1338,14 +1338,14 @@ Detailed documentation: [WAF Operations and Rule Schema](../web-application-fire Operations for restarting Harper and managing system state. -| Operation | Description | Role Required | -| -------------------- | ----------------------------------------------------- | ------------- | -| `restart` | Restarts the Harper instance | super_user | -| `restart_service` | Restarts a specific Harper service | super_user | -| `system_information` | Returns detailed host system metrics | super_user | -| `set_status` | Sets an application-specific status value (in-memory) | super_user | -| `get_status` | Returns a previously set status value | super_user | -| `clear_status` | Removes a status entry | super_user | +| Operation | Description | Role Required | +| -------------------- | ----------------------------------------- | ------------- | +| `restart` | Restarts the Harper instance | super_user | +| `restart_service` | Restarts a specific Harper service | super_user | +| `system_information` | Returns detailed host system metrics | super_user | +| `set_status` | Sets an application-specific status value | super_user | +| `get_status` | Returns a previously set status value | super_user | +| `clear_status` | Removes a status entry | super_user | ### `restart` @@ -1365,7 +1365,11 @@ Restarts a specific service. `service` must be one of: `http`, `http_workers`, ` ### `system_information` -Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by `attributes` array (e.g., `["cpu", "memory", "replication"]`). +Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by an `attributes` array (e.g., `["cpu", "memory", "threads"]`). + +Valid attribute names are `system`, `time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, `metrics`, and `threads`. Omitting `attributes` returns all of them. + +> **Unrecognized attribute names are silently ignored**, not rejected. A typo produces a smaller response rather than an error, so if a section you asked for is missing, check the spelling before you check the node. ```json { "operation": "system_information" } @@ -1373,7 +1377,11 @@ Returns system metrics including CPU, memory, disk, network, and Harper process ### `set_status` / `get_status` / `clear_status` -Manage in-memory application status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). Status is not persisted across restarts. +Manage application-defined status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). + +Values are **persisted**, stored in the `hdb_status` table in the `system` database, so they survive a restart. That table is declared `replicate: false`, so a status set on one node stays on that node and does not propagate to peers — which is what makes these values usable as node-local coordination markers. + +These are a coordination primitive for your own automation: Harper stores and returns them but does not change its own behavior based on them, so they are not a substitute for a real health check. A `get_status` call with no `id` additionally returns Harper-derived state, including a `restartRequired` flag that tracks pending component and code restarts (not configuration changes). ```json { "operation": "set_status", "id": "primary", "status": "active" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 8baeaa11c..b3826dc45 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -171,12 +171,12 @@ Returns an array of status objects from the cluster, including active WebSocket "database": "data", "connected": true, "latency": 0.7, - "thread_id": 1, + "threadId": 1, "nodes": ["server-2.domain.com"], "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", "lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT", - "lastSendTime": "Wed, 12 Feb 2025 16:50:59 GMT" + "sendingMessage": "Wed, 12 Feb 2025 16:50:59 GMT" } ] } @@ -195,6 +195,8 @@ Returns an array of status objects from the cluster, including active WebSocket | `lastReceivedLocalTime` | Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up | | `sendingMessage` | Timestamp of the transaction actively being sent. Absent when waiting for the next transaction | +While a database is taking its initial full copy, these timing fields hold the literal string `"Copying"` rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching. + --- ### Configure Cluster From cf107183b03602f251fc420374dc075f99e1292c Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 09:32:14 -0600 Subject: [PATCH 4/7] docs(learn): close coverage gaps against the original field guide Read Jeff's "Operating Reliable Harper Applications" doc and compared it against the ten guides. Coverage was broadly good, but three documented capabilities were dropped entirely and one CLI claim was wrong. - `get_backup` has no offline CLI form. Guide 7 told readers every backup operation works with the server stopped; `get_backup` is absent from the offline command switch and fails as an unknown command. That matters because a stopped node is exactly when an operator reaches for it. - `storage.writeAsync` was missing from the RPO guide. It defaults off, and turning it on disables fsync so acknowledged commits can be lost on power loss or an OS crash. Nothing else in the track surfaces it: backup timestamps still look right and no alert fires, so a stated recovery point can be quietly false. Now framed as an RPO decision with the cases where enabling it is defensible. - `threads.preload` and `threads.preloadRequire` were missing from the observability guide. They are how an APM or tracing agent loads ahead of the code it instruments. Includes the split-entry case, which fails confusingly: with only the loader hooks preloaded a tracer emits no-op spans and exports nothing, so instrumentation looks installed while the collector stays empty. - `HARPER_SAFE_MODE` was missing from the backup guide. Guide 7 explains that a component holding a database open blocks an online restore but never mentioned the escape hatch. Safe mode starts Harper without loading components, so the restore can run, and it is also how you reach a node a broken component stopped. Also corrects the safe mode example in the configuration reference, which invoked `harperdb` rather than `harper`. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 25 +++++++++++ .../engineering-rpo-rto-and-uptime.mdx | 29 ++++++++++++ .../administration/monitoring-and-triage.mdx | 44 +++++++++++++++++++ reference/configuration/overview.md | 2 +- 4 files changed, 99 insertions(+), 1 deletion(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 8e0a286ad..cb13c67aa 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -113,6 +113,12 @@ Every backup operation runs from the CLI under the same name. With the server running the CLI forwards to the server; with it stopped the command operates directly on the files. +`get_backup` is the exception. It streams from a running server and has no offline +form, so with Harper stopped it fails as an unknown command. If you need a copy +off a node that is down, start it, or take the backup directory or a volume +snapshot instead. Worth knowing before an incident, because a stopped node is +exactly when you reach for it. + @@ -231,6 +237,23 @@ open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. +:::tip +That last constraint has an escape hatch worth knowing before you need it. Setting +`HARPER_SAFE_MODE` to any value starts Harper without loading any applications or +components, while the database, Operations API, and HTTP server come up normally: + +```bash +HARPER_SAFE_MODE=1 harper +``` + +Because no component loads, nothing holds a user database open, so a restore that +would fail inside its job can run online. It is also how you get back in when a +broken component is what stopped Harper in the first place, since the Operations +API is available to inspect, repair, or remove it. Traffic should stay off the +node either way. See +[safe mode](/reference/v5/configuration/overview#harper_safe_mode). +::: + Two more constraints worth writing into your procedure: - **`target_database` requires the server stopped.** Restoring into a separate @@ -334,6 +357,8 @@ that it is the target than a promise you cannot keep. - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged +- [ ] Safe mode known as the way to restore a database a component holds open, + and to reach a node a broken component stopped - [ ] Restore drill completed and dated, with measured recovery time and recovery point - [ ] `system` database restore rehearsed offline diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index 958c625d5..b1d7043e2 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -21,6 +21,7 @@ during an incident that the answer was the target all along. - How to convert an availability target into a monthly budget, and what each tier demands of your automation - How to reconcile stated targets against the numbers measured in earlier guides +- Why one storage setting can silently invalidate your stated recovery point - Why declaring your degraded modes is as important as declaring your targets ## Prerequisites @@ -56,6 +57,32 @@ availability measured at a Harper node are different quantities, and the difference is exactly the part of the stack you may not control. Whichever you choose, say so in the same sentence as the number. +### One storage setting can override your recovery point + +Your recovery point objective is a statement about how much acknowledged data you +can afford to lose. One configuration option can quietly make that statement +false. + +[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) +defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, +so a write returns as soon as it is queued to the operating system's page cache. +Throughput on write-heavy workloads improves substantially. The cost is that a +power loss or OS crash between the commit and the flush to disk loses the most +recent transactions, even though your application was told they succeeded. The +database stays structurally consistent; only the newest writes are gone. + +The trap is that this is invisible to everything else in this guide. Backup +timestamps look right, replication looks healthy, and no alert fires. You discover +it the first time a node loses power, which is also the first time the gap between +your stated recovery point and your real one matters. + +So treat it as an RPO decision rather than a performance one. Enabling it is +defensible when the data is reproducible from an upstream source, when the +workload is bulk ingest that can simply be re-run, or when peers acknowledge +writes before the loss window opens. Outside those cases, leave it off. If it is +on anywhere, record which databases and say so next to the recovery point in your +plan, because the number no longer means what it appears to. + ## Map scenarios to mechanisms For each scenario, know the mechanism, and know what must be proven for the @@ -199,6 +226,8 @@ would have hurt you. - [ ] Availability target converted to minutes per month - [ ] Error budget policy states what changes when the budget runs low - [ ] Backup cadence reconciled against recovery point +- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded + alongside their recovery point - [ ] Restore time plus detection time reconciled against recovery time - [ ] Convergence time reconciled against the traffic admission gate - [ ] Every unreconciled gap has an owner and a date diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index 7549ccc58..ef33fffa0 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -22,6 +22,7 @@ short enough to run under pressure. - Which Harper metrics are worth an alert, by name, and which are only worth a dashboard - How to get metrics out of Harper, on Fabric and self-managed +- How to preload an APM or tracing agent so it can instrument your application - How to write log entries that are still useful during an incident - A triage sequence that narrows a Harper incident in about five minutes @@ -156,6 +157,47 @@ what a cluster-wide query means, and it is easier to decide that deliberately th to discover it after building panels on the wrong assumption. ::: +## Run an APM or tracing agent + +Harper's own metrics tell you how the node is behaving. They do not give you +distributed traces across your application code and its downstream calls, which is +what you want when a journey is slow and you need to know where the time went. +That comes from an instrumentation agent, and an agent has to load before the code +it instruments. + +Two configuration keys put a module on each worker thread's startup, ahead of +Harper's own modules and yours: + +```yaml +threads: + preloadRequire: dd-trace/init # the entry that calls init() + preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation +``` + +[`threads.preload`](/reference/v5/configuration/options#threads) + + loads a module via Node's `--import`, which is how an agent installs the loader hooks +that let it instrument modules imported later. [`threads.preloadRequire`](/reference/v5/configuration/options#threads) + uses `--require`, which runs the module body and is typically how an agent's +initialization entry actually starts it. + +Installing loader hooks and starting the agent are two different jobs, and which +of your agent's entry points does which is specific to that agent. Some ship one +entry that does both; others split them, in which case set both keys. The example +above is the split-entry case, and it is the one that produces the most confusing +failure: with only the hooks loaded, a tracer will hand out spans with plausible +trace ids that are no-ops and export nothing, so your instrumentation looks +installed and your collector stays empty. + +Two constraints worth knowing before you plan around this. Both keys apply to +worker threads only, and neither works under Bun. And bare specifiers resolve +against the `node_modules` of your installed components, so an agent can ship as a +dependency of a deployed component rather than as a host-level install. + +Whatever agent you use, follow its own worker-thread documentation, and finish by +confirming spans actually arrive at your collector rather than assuming the +configuration took. + ## Logs you can use during an incident Harper's `logger` global takes a message plus a context object from component @@ -316,6 +358,8 @@ misleading signal during a real incident costs more than a missing one. - [ ] Alerts on storage headroom and backup age - [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed +- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` + set as that agent requires, with spans confirmed at the collector - [ ] `analytics.replicate` setting known and deliberate - [ ] Structured logging includes component, version, node, request id, operation, duration, and outcome diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 112327b24..51284de3f 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -299,7 +299,7 @@ logging: Setting `HARPER_SAFE_MODE` to any value starts Harper without loading user applications or components. Harper's core services (database, operations API, HTTP server) start normally, but no applications from the components directory are loaded and no package-based extensions are initialized. ```bash -HARPER_SAFE_MODE=1 harperdb +HARPER_SAFE_MODE=1 harper ``` This is useful when a broken or misbehaving component prevents Harper from starting. Safe mode lets you access the operations API to inspect, repair, or remove the problematic component without needing to manually edit files on disk. From c77e17c06c49109cbdf160da3a58fcd738558c67 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 13:23:45 -0600 Subject: [PATCH 5/7] docs: address Gemini review feedback Three of the seven review comments were worth taking. - Move `` out of prose and list items in guide 6, per CONTRIBUTING's "reserve VersionBadge for standalone placement after headings". Uses the repo's own inline form, `(Added in: vX.Y.Z)`, rather than the bare `(v5.2.0)` the review suggested. Also applies it to the two list-item badges the review did not flag, so the guide is internally consistent. Table-cell badges are left alone, which is established practice across the reference. - Promote the two managed-backup limitations to an admonition, since a database on per-table storage paths silently has no backup strategy at all. Uses `:::warning` rather than `:::caution` to match the dominant type in this repo. The other four were declined; see the PR threads for reasoning. In short: the snake_case suggestion would have made the sentinel example inconsistent with every shipped Learn guide, and the three remaining suggestions wrapped multi-paragraph body prose in admonitions, one of them quoting text that had already been corrected. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 3 +++ learn/administration/safe-deployments-and-rollback.mdx | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index cb13c67aa..ab87c0a08 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -73,6 +73,7 @@ A backup is a whole-database copy: all tables, the transaction log, and any file-backed blobs. So a restored database keeps its `read_audit_log` history as of the backup point. +:::warning Two limitations to check against your own deployment now rather than later: - **Managed backups require RocksDB.** For LMDB databases your options are @@ -83,6 +84,8 @@ Two limitations to check against your own deployment now rather than later: configured per-table paths, your backup strategy for that database does not exist yet and you need to know that before launch. +::: + ## Create and retain managed backups diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 92e62b3e4..baf18bffc 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -71,7 +71,7 @@ is the deliberate exception: it replicates only when you ask.) ::: `urlPath` mounts a component at an HTTP path. -`host` serves it on a virtual hostname. Both are +`host` (Added in: v5.2.0) serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). @@ -228,10 +228,10 @@ synthetic rather than through the deploy response. Two parameters worth setting deliberately on replicated deploys: -- `deployment_timeout` is how long a peer waits +- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. -- `ignore_replication_errors` treats a peer that +- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it, because @@ -243,7 +243,7 @@ Two parameters worth setting deliberately on replicated deploys: A configuration change carries the same risk as a code change and gets less ceremony, which is backwards. -`set_configuration` supports `"replicated": true` +`set_configuration` supports `"replicated": true` (Added in: v5.2.0) to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. From 7668c148f3874d566986e5f5140faf34d4707778 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 14:42:08 -0600 Subject: [PATCH 6/7] docs: fixes from an independent Gemini review pass Ran the PR diff through Gemini (gemini-3.1-pro-high) file by file. Of 52 findings, 12 held up against the harper/harper-pro sources; the rest were contradicted by code I had already verified. The ones that held up cluster almost entirely in internal consistency, which is what a diff-only reviewer can actually check. Bugs this pass caught, several of my own making: - `HARPER_SAFE_MODE=1 harper` does not start the server. Bare `harper` falls through to the CLI-operations branch; starting it needs `harper run`. I introduced this when correcting the reference's `harperdb`. - The drain sequence checked peer capacity at step 3, after the node was already drained and its traffic had moved. Moved the check before the drain and added the abort path, which was missing entirely. - The readiness sample could never pass. After I changed it to test the sentinel record, nothing told the reader to create one, so the probe would return 503 forever. Added the seeding step. - Guide 8 contradicted itself on recovery time: one place measured from detection, another counted a detection gap inside the total. Settled on recovery time starting when the journey breaks, detection included. - The availability flag does not stop replication, so the restore sequence implied an isolation it never provided. An online restore on a drained but still-connected node exposes rolled-back data to peers before validation. Isolation is now its own step with a danger note. - `system_information` reports node and npm versions, not Harper's. The service inventory now points at `registration_info`. - The sizing sample requested neither disk nor network while the prose told the reader to watch both. - A Prettier wrap left a link orphaned as its own paragraph, stranding the sentence that followed it. - Smaller: "point-in-time restore" implied continuous PITR rather than restoring a snapshot; two checklist items still named `config.yaml` after the prose moved to the root `harper-config.yaml`; the status types are fixed while only their values are application-defined; and one leftover opt-in phrasing sat next to the corrected opt-out default. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 38 +++++++++++--- .../engineering-rpo-rto-and-uptime.mdx | 51 ++++++++++--------- .../health-checks-and-traffic-admission.mdx | 36 ++++++++++--- .../how-harper-runs-in-production.mdx | 5 +- .../administration/monitoring-and-triage.mdx | 12 ++--- .../production-readiness-checklist.mdx | 6 ++- .../sizing-a-harper-cluster.mdx | 4 +- reference/configuration/overview.md | 2 +- reference/operations-api/operations.md | 2 +- reference/replication/overview.md | 2 +- 10 files changed, 104 insertions(+), 54 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index ab87c0a08..9b8419730 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -246,7 +246,7 @@ That last constraint has an escape hatch worth knowing before you need it. Setti components, while the database, Operations API, and HTTP server come up normally: ```bash -HARPER_SAFE_MODE=1 harper +HARPER_SAFE_MODE=1 harper run ``` Because no component loads, nothing holds a user database open, so a restore that @@ -275,15 +275,35 @@ then has to rejoin peers that never rolled back, so sequence it deliberately: 1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. -2. Decide what should happen to replication for the restored database, and do that - deliberately. Whether the restored data should propagate, or be overwritten by - peers, is a decision with two very different outcomes, and it is not a decision - to improvise. -3. Perform the restore, online or offline according to the table above. -4. Verify data expectations locally before any peer sees the node. -5. Re-admit the node through the full return sequence, including convergence +2. **Isolate it from its peers.** This is a separate action from step 1, and the + step most likely to be skipped. See the warning below. +3. Decide what should happen to the restored data: whether it should propagate to + peers, or be overwritten by them. Those are two very different outcomes and it + is not a decision to improvise once the node is back on the mesh. +4. Perform the restore, online or offline according to the table above. +5. Verify data expectations locally before any peer sees the node. +6. Re-admit the node through the full return sequence, including convergence verification. +:::danger +**The availability flag does not stop replication.** It tells your traffic layer +to stop sending user requests. Replication is a separate path: peer connections +stay up, and the node keeps exchanging transactions the whole time it is "out of +rotation." + +So a node that is drained but still connected is not isolated. Restore it in place +and the rolled-back database is live on the mesh immediately, before you have +validated anything. Two things can then happen, both bad: peers overwrite your +restored data with newer transactions, or your rolled-back state propagates +outward to peers that were never affected. + +Isolation is its own step. Stopping Harper and restoring offline is the +unambiguous way to get it, which is a further argument for the offline path in a +cluster even where the table above permits an online restore. If you restore +online, you must have removed the node from the replication topology first, and +put it back deliberately in step 6. +::: + ## What backups cannot fix If a release changed the meaning of persisted data, restoring the database is @@ -360,6 +380,8 @@ that it is the target than a promise you cannot keep. - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged +- [ ] Restore procedure isolates the node from replication, not just from user + traffic, since the availability flag does not stop peer exchange - [ ] Safe mode known as the way to restore a database a component holds open, and to reach a node a broken component stopped - [ ] Restore drill completed and dated, with measured recovery time and recovery diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index b1d7043e2..af0a44dd7 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -35,11 +35,11 @@ during an incident that the answer was the target all along. ## State the three per journey -| Term | The question it answers | How to state it so it is testable | -| ------------------------ | ------------------------------------ | --------------------------------------------------------------------------- | -| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | -| Recovery time objective | How long until service is restored | Per journey, measured from detection to verified service, not from decision | -| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | +| Term | The question it answers | How to state it so it is testable | +| ------------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------- | +| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | +| Recovery time objective | How long until service is restored | Per journey, measured from the moment the journey starts failing, not from the moment someone acts | +| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | Three details make the difference between a testable statement and a slogan. @@ -47,10 +47,12 @@ Three details make the difference between a testable statement and a slogan. differently, matter differently, and recover differently. One cluster-wide number overcommits on the cheap journey and undercommits on the expensive one. -**Recovery time starts at detection, not at decision.** The gap between something -breaking and someone knowing is part of the outage, and it is often the largest -part. Measuring from the moment a human decided to act produces a number that -flatters your automation and misleads your planning. +**Recovery time starts when the journey breaks, not when someone acts on it.** +Detection is inside the clock, not before it, and the gap between something +breaking and someone knowing is often the largest part of an outage. Starting the +count at the moment a human decided to act, or even at the moment an alert fired, +produces a number that flatters your automation and misleads your planning. Your +users were already down for all of it. **Name the measurement boundary.** Availability measured at your CDN edge and availability measured at a Harper node are different quantities, and the @@ -88,20 +90,20 @@ plan, because the number no longer means what it appears to. For each scenario, know the mechanism, and know what must be proven for the mechanism to count. The right-hand column is the whole exercise. -| Scenario | Primary mechanism | What must be proven | -| ------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | -| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | -| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | -| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | -| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | -| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | -| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | -| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | -| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | -| Corruption propagated by replication | Point-in-time restore plus a replication decision | A rehearsed sequence that includes what happens to peers | -| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | -| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | +| Scenario | Primary mechanism | What must be proven | +| ------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | +| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | +| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | +| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | +| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | +| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | +| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | +| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | +| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | +| Corruption propagated by replication | Restore the last good backup, plus a replication decision | A rehearsed sequence that includes what happens to peers | +| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | +| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | Two rows in that table are the ones most often missing. Node replacement is budgeted as though it were a restart, when it is a full synchronization and can be @@ -217,7 +219,8 @@ would have hurt you. ## Readiness checklist - [ ] Recovery point stated per database, in time units -- [ ] Recovery time stated per journey, measured from detection +- [ ] Recovery time stated per journey, measured from the start of failure and + including detection time - [ ] Availability stated per journey, with the measurement boundary named - [ ] Exclusions documented - [ ] Every scenario in the map has a named mechanism diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index fcd9524b3..da0339e74 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -230,6 +230,21 @@ authorization error rather than a readiness one. In that form you need `allowRead() { return true; }` to keep it reachable. ::: +The sentinel is a record you create once, on purpose, and leave alone. Seed it on +every node before you point anything at this route, or readiness will report +`failed` forever and you will have built a probe that never passes: + +```bash +curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"name":"readiness probe sentinel","description":"do not delete"}' +``` + +Give it a name that says what it is, because the next person to find it will be +deciding whether it is safe to delete. If your table replicates, seeding it once +is enough; if it does not, seed it per node. + Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version @@ -244,17 +259,22 @@ skip. -1. **Declare it unavailable.** `DELETE /status` on the target node. -2. **Verify traffic actually stopped.** Watch request volume on the target fall to +1. **Confirm the peers can carry it, before you drain anything.** The remaining + nodes have to stay inside their capacity budget with this node gone. If they + cannot, stop here and do not drain: see + [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after + the traffic has already moved is how a maintenance window becomes an incident. +2. **Declare it unavailable.** `DELETE /status` on the target node. +3. **Verify traffic actually stopped.** Watch request volume on the target fall to zero and rise on its peers. Do not trust the configured weight; check the measured request count from [analytics](/reference/v5/analytics/overview) or your traffic layer's own metrics. Health check intervals, DNS TTLs, and client-side connection reuse all add delay here, and the delay is yours to measure. -3. **Confirm the peers can carry it.** The remaining nodes have to be inside their - capacity budget with this node gone. If they are not, stop: see - [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). -4. **Do the work.** Restart, upgrade, reconfigure, or investigate. +4. **Watch the peers absorb it.** If they are going outside budget despite step 1, + abort: `POST /status` to put this node back in rotation, and re-plan the change + for a lower-traffic window or with added capacity. +5. **Do the work.** Restart, upgrade, reconfigure, or investigate. @@ -323,8 +343,8 @@ than people assume, because it includes convergence rather than just startup. ## Readiness checklist -- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than - deployed by hand +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` + rather than deployed by hand - [ ] Traffic layer health check points at `GET /status` on `9926` - [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index f9690246e..81e6160df 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -198,13 +198,16 @@ Then: what is deployed, including which extensions are present - [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) for what changed and when +- [`registration_info`](/reference/v5/operations-api/operations#registration_info) + for the Harper version, which `system_information` does not report (it returns + the node and npm versions, not Harper's) - `cluster_status` for the peers and databases actually connected Fill in a boundary record you can keep: | Item | Value for your deployment | | ----------------------- | ----------------------------------------------------- | -| Harper version | From `system_information` | +| Harper version | From `registration_info` | | Node count and roles | From `cluster_status` and your topology intent | | Ports in use | From `get_configuration` | | Databases | Which exist, and which replicate | diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index ef33fffa0..d51d8b89b 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -174,12 +174,12 @@ threads: preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation ``` -[`threads.preload`](/reference/v5/configuration/options#threads) - - loads a module via Node's `--import`, which is how an agent installs the loader hooks -that let it instrument modules imported later. [`threads.preloadRequire`](/reference/v5/configuration/options#threads) - uses `--require`, which runs the module body and is typically how an agent's -initialization entry actually starts it. +`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is +how an agent installs the loader hooks that let it instrument modules imported +later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the +module body and is typically how an agent's initialization entry actually starts +it. Both are documented under +[threads configuration](/reference/v5/configuration/options#threads). Installing loader hooks and starting the agent are two different jobs, and which of your agent's entry points does which is specific to that agent. Some ship one diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 0cbeac7f7..45d1165ba 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -69,8 +69,8 @@ From - [ ] Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals -- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than by - hand +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` + rather than by hand - [ ] Traffic layer health check points at `GET /status` on the application port - [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself @@ -161,6 +161,8 @@ From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged +- [ ] Restore procedure isolates the node from replication, not just from user + traffic - [ ] Restore drill completed and dated, with measured recovery time and actual data loss - [ ] `system` database restore rehearsed offline diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx index b06a86948..9a226224f 100644 --- a/learn/administration/sizing-a-harper-cluster.mdx +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -98,7 +98,7 @@ An average conceals the one node that is about to tip. curl -s -X POST https://my-node.example.com:9925/ \ -H 'Content-Type: application/json' \ -u 'admin:password' \ - -d '{"operation":"system_information","attributes":["cpu","memory","threads","harperdb_processes"]}' + -d '{"operation":"system_information","attributes":["cpu","memory","disk","network","threads","harperdb_processes"]}' ``` @@ -113,7 +113,7 @@ await fetch('https://my-node.example.com:9925/', { }, body: JSON.stringify({ operation: 'system_information', - attributes: ['cpu', 'memory', 'threads', 'harperdb_processes'], + attributes: ['cpu', 'memory', 'disk', 'network', 'threads', 'harperdb_processes'], }), }); ``` diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 51284de3f..c2fc1e65b 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -299,7 +299,7 @@ logging: Setting `HARPER_SAFE_MODE` to any value starts Harper without loading user applications or components. Harper's core services (database, operations API, HTTP server) start normally, but no applications from the components directory are loaded and no package-based extensions are initialized. ```bash -HARPER_SAFE_MODE=1 harper +HARPER_SAFE_MODE=1 harper run ``` This is useful when a broken or misbehaving component prevents Harper from starting. Safe mode lets you access the operations API to inspect, repair, or remove the problematic component without needing to manually edit files on disk. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 8e85a499f..17a077be8 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1377,7 +1377,7 @@ Valid attribute names are `system`, `time`, `cpu`, `memory`, `disk`, `network`, ### `set_status` / `get_status` / `clear_status` -Manage application-defined status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). +Manage application status values. The status types are fixed — `primary`, `maintenance`, `availability` — while the value you store against each is yours to define (availability only accepts `'Available'` or `'Unavailable'`). Values are **persisted**, stored in the `hdb_status` table in the `system` database, so they survive a restart. That table is declared `replicate: false`, so a status set on one node stays on that node and does not propagate to peers — which is what makes these values usable as node-local coordination markers. diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 1015ec7c2..37bc6f847 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -369,7 +369,7 @@ Note that `replicated` is opt-out rather than opt-in across the operations that Users and roles **do** propagate by default. They live in the `system` database, and the default replication scope is every database, so `system` is in scope unless you have narrowed it. As of v5.2 replicating `system` no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). If you narrow the scope to exclude `system`, users and roles stop propagating and must be provisioned on every node by your own automation. -Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. +Certain management operations — including component deployment and rolling restarts — are replicated across the cluster as well, following the same opt-out rule described above. ## Inspecting Cluster Configuration From a5b20e0dad74599a66b6a8764fa868bfbdf58ac6 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 15:46:26 -0600 Subject: [PATCH 7/7] docs(learn): unwrap hard-wrapped prose in the Administration guides The ten guides arrived hard-wrapped at 80 columns, which nothing else in the repo does: existing Learn and reference pages run one line per paragraph, with prose lines routinely past 500 characters. Prettier is configured without proseWrap, so it preserves whatever it is given and never enforced either style. Wrapping is undone across all ten. Paragraphs and list items are now one logical line each. Code fences, tables, frontmatter, JSX, admonitions, headings, blockquotes, and import lines are untouched, and a content fingerprint was compared before and after each rewrite to confirm nothing but line breaks changed. Verified per file: identical structural element counts, identical token streams, and a clean build. Also carries an edit to the opening paragraph of how-harper-runs-in-production.mdx that was made by hand in the worktree, not by the rewrap. It is the one token-level change in the diff. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 342 +++++------------- .../engineering-rpo-rto-and-uptime.mdx | 181 +++------ .../health-checks-and-traffic-admission.mdx | 238 +++--------- .../how-harper-runs-in-production.mdx | 173 +++------ .../administration/monitoring-and-triage.mdx | 261 ++++--------- .../administration/operating-replication.mdx | 282 ++++----------- .../production-readiness-checklist.mdx | 117 ++---- .../reliability-plan-template.mdx | 39 +- .../safe-deployments-and-rollback.mdx | 277 ++++---------- .../sizing-a-harper-cluster.mdx | 178 +++------ 10 files changed, 523 insertions(+), 1565 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 9b8419730..164222592 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -6,24 +6,15 @@ sidebar_position: 7 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Replication gives you availability. It does not give you recovery, and the reason -is worth stating plainly: replication faithfully propagates whatever you write, -including the bad write. A destructive operation, a schema mistake, or a release -that corrupted data reaches every peer at the speed of your convergence time. +Replication gives you availability. It does not give you recovery, and the reason is worth stating plainly: replication faithfully propagates whatever you write, including the bad write. A destructive operation, a schema mistake, or a release that corrupted data reaches every peer at the speed of your convergence time. -Backups cover the failure classes replication cannot. This guide is about -deciding what you need before you pick an operation, getting a copy somewhere the -node's failure cannot reach, and knowing your restore constraints before you are -in an incident rather than during one. +Backups cover the failure classes replication cannot. This guide is about deciding what you need before you pick an operation, getting a copy somewhere the node's failure cannot reach, and knowing your restore constraints before you are in an incident rather than during one. ## What You Will Learn -- The five questions that determine your backup design, answered before any - command -- Which backup mechanism you actually have, since it depends on your storage - engine -- Why a copy on the node that created it is not a backup, and the specific way - copying one wrong breaks it +- The five questions that determine your backup design, answered before any command +- Which backup mechanism you actually have, since it depends on your storage engine +- Why a copy on the node that created it is not a backup, and the specific way copying one wrong breaks it - What `verify_backup` does and does not check - Which databases can be restored with the server running, and which cannot - Why restoring a database to undo a code bug is usually the wrong move @@ -31,36 +22,19 @@ in an incident rather than during one. ## Prerequisites - A cluster with a `super_user` credential, and CLI access to at least one node -- Knowledge of your storage engine per database, from your service boundary - inventory in - [How Harper Runs in Production](./how-harper-runs-in-production.mdx) -- [Operating Replication](./operating-replication.mdx), because a restore in a - replicated cluster is a replication event as much as a storage one +- Knowledge of your storage engine per database, from your service boundary inventory in [How Harper Runs in Production](./how-harper-runs-in-production.mdx) +- [Operating Replication](./operating-replication.mdx), because a restore in a replicated cluster is a replication event as much as a storage one - A non-production database you are willing to destroy ## Decide before you choose a command -Answer these first. Every operation below is easy, and every one of them is the -wrong choice for some of these answers. - -1. **Which failure classes must you survive?** Node loss, storage loss, a bad - deployment, an accidental destructive operation, a corrupting application bug, - a site or region event, and a control-plane or credential loss are seven - different problems. Replication addresses the first one well and the sixth one - partially. It addresses none of the rest. -2. **What is the recovery point and recovery time per database?** Not per cluster. - A catalog table and an order ledger rarely deserve the same answer, and paying - ledger-grade backup cadence for catalog data is how backup cost becomes a - reason to reduce frequency. -3. **What granularity do you need?** Harper backs up and restores whole databases. - There is no per-table restore. If your recovery story requires restoring one - table, that requirement has to be met by database layout or by forward repair, - and it is far cheaper to learn that now. -4. **Where must copies live?** How many independent locations, under what - retention, and with what access control. -5. **Who is authorized to execute a restore, and how is that logged?** A restore - destroys current data by design. It deserves a named authority and an audit - trail. +Answer these first. Every operation below is easy, and every one of them is the wrong choice for some of these answers. + +1. **Which failure classes must you survive?** Node loss, storage loss, a bad deployment, an accidental destructive operation, a corrupting application bug, a site or region event, and a control-plane or credential loss are seven different problems. Replication addresses the first one well and the sixth one partially. It addresses none of the rest. +2. **What is the recovery point and recovery time per database?** Not per cluster. A catalog table and an order ledger rarely deserve the same answer, and paying ledger-grade backup cadence for catalog data is how backup cost becomes a reason to reduce frequency. +3. **What granularity do you need?** Harper backs up and restores whole databases. There is no per-table restore. If your recovery story requires restoring one table, that requirement has to be met by database layout or by forward repair, and it is far cheaper to learn that now. +4. **Where must copies live?** How many independent locations, under what retention, and with what access control. +5. **Who is authorized to execute a restore, and how is that logged?** A restore destroys current data by design. It deserves a named authority and an audit trail. ## Know which mechanism you have @@ -69,20 +43,13 @@ wrong choice for some of these answers. | **Managed backups** | | Incremental, verifiable backups in a server-side repository under `storage.backupPath` | | **Snapshot download** | | [`get_backup`](/reference/v5/backups/operations#get_backup) streams a snapshot over HTTP, with no server-side artifact | -A backup is a whole-database copy: all tables, the transaction log, and any -file-backed blobs. So a restored database keeps its `read_audit_log` history as of -the backup point. +A backup is a whole-database copy: all tables, the transaction log, and any file-backed blobs. So a restored database keeps its `read_audit_log` history as of the backup point. :::warning Two limitations to check against your own deployment now rather than later: -- **Managed backups require RocksDB.** For LMDB databases your options are - `get_backup` or volume snapshots. -- **One storage root per database.** A database whose tables use per-table `path` - storage configs spans multiple root stores and cannot be backed up with these - operations at all. Database-level custom storage paths are fine. If someone has - configured per-table paths, your backup strategy for that database does not - exist yet and you need to know that before launch. +- **Managed backups require RocksDB.** For LMDB databases your options are `get_backup` or volume snapshots. +- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations at all. Database-level custom storage paths are fine. If someone has configured per-table paths, your backup strategy for that database does not exist yet and you need to know that before launch. ::: @@ -98,10 +65,7 @@ Two limitations to check against your own deployment now rather than later: } ``` -Through a running server this returns a `job_id` immediately. Poll -[`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, -which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result -as the completion signal, not the original response. +Through a running server this returns a `job_id` immediately. Poll [`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result as the completion signal, not the original response. @@ -112,70 +76,37 @@ harper list_backups database=data harper purge_backups database=data keep_count=7 ``` -Every backup operation runs from the CLI under the same name. With the server -running the CLI forwards to the server; with it stopped the command operates -directly on the files. +Every backup operation runs from the CLI under the same name. With the server running the CLI forwards to the server; with it stopped the command operates directly on the files. -`get_backup` is the exception. It streams from a running server and has no offline -form, so with Harper stopped it fails as an unknown command. If you need a copy -off a node that is down, start it, or take the backup directory or a volume -snapshot instead. Worth knowing before an incident, because a stopped node is -exactly when you reach for it. +`get_backup` is the exception. It streams from a running server and has no offline form, so with Harper stopped it fails as an unknown command. If you need a copy off a node that is down, start it, or take the backup directory or a volume snapshot instead. Worth knowing before an incident, because a stopped node is exactly when you reach for it. -Backups of the same database share unchanged RocksDB data files, so the first one -copies everything and later ones copy only what changed. Shared files are -reference-counted, so deleting a backup removes only files no remaining backup -references. +Backups of the same database share unchanged RocksDB data files, so the first one copies everything and later ones copy only what changed. Shared files are reference-counted, so deleting a backup removes only files no remaining backup references. :::warning -The incremental behavior applies to the RocksDB data files only. **The -transaction-log snapshot is copied in full on every backup.** With a large -audit-retention window that is real, recurring disk cost that the data-only view -hides, so size your backup volume against it rather than against the incremental -figure. - -Blobs behave differently and are easy to over-budget. Each backup captures a full -set of blobs, but the files are hard-linked rather than copied when the backup -directory and the blob storage share a filesystem, which is the default layout. In -that case the extra space is near zero. They become genuine full copies only when -the two live on different filesystems, so putting backups on a separate volume is -a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip -blobs when that is appropriate. - -One consequence for disaster recovery: because the blob snapshot is hard-linked, -copying a backup repository with an ordinary recursive copy materializes every -blob as a full, separate file at the destination. +The incremental behavior applies to the RocksDB data files only. **The transaction-log snapshot is copied in full on every backup.** With a large audit-retention window that is real, recurring disk cost that the data-only view hides, so size your backup volume against it rather than against the incremental figure. + +Blobs behave differently and are easy to over-budget. Each backup captures a full set of blobs, but the files are hard-linked rather than copied when the backup directory and the blob storage share a filesystem, which is the default layout. In that case the extra space is near zero. They become genuine full copies only when the two live on different filesystems, so putting backups on a separate volume is a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip blobs when that is appropriate. + +One consequence for disaster recovery: because the blob snapshot is hard-linked, copying a backup repository with an ordinary recursive copy materializes every blob as a full, separate file at the destination. ::: -Do not use `list_backups` sizes for capacity planning. The `size` and -`file_count` fields come from the RocksDB backup engine and exclude the -transaction-log and blob snapshots, so each entry undercounts, while the shared -files between entries mean summing them overcounts. Measure the repository -directory instead. +Do not use `list_backups` sizes for capacity planning. The `size` and `file_count` fields come from the RocksDB backup engine and exclude the transaction-log and blob snapshots, so each entry undercounts, while the shared files between entries mean summing them overcounts. Measure the repository directory instead. ## Get a copy off the node -This is the step most likely to be missing, and the one where doing it slightly -wrong produces a copy that cannot be restored. +This is the step most likely to be missing, and the one where doing it slightly wrong produces a copy that cannot be restored. -**Managed backups live on the node that created them.** The repository is a local -directory, and RocksDB shares files across backup IDs, so **a backup ID is not a -self-contained folder.** Two consequences: +**Managed backups live on the node that created them.** The repository is a local directory, and RocksDB shares files across backup IDs, so **a backup ID is not a self-contained folder.** Two consequences: -- A disaster-recovery copy has to take the entire per-database repository, - `/`, not an individual backup. -- It has to do that while no backup operation is running, or from an atomic - filesystem snapshot. A live recursive copy can race `create_backup`, - `delete_backup`, or `purge_backups` and produce an unrestorable copy. +- A disaster-recovery copy has to take the entire per-database repository, `/`, not an individual backup. +- It has to do that while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`, `delete_backup`, or `purge_backups` and produce an unrestorable copy. -An unrestorable copy is worse than no copy, because it will pass a "backups exist" -check and fail during an incident. +An unrestorable copy is worse than no copy, because it will pass a "backups exist" check and fail during an incident. -The simpler off-host path, and the one to prefer unless you specifically need -retained managed backups off the node: +The simpler off-host path, and the one to prefer unless you specifically need retained managed backups off the node: ```bash # Pull a snapshot of the current state from a running node @@ -185,18 +116,11 @@ harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` -`target=` only changes which node the snapshot is read from. It writes a file and -nothing else: it does not restore, import, or clone that node's database onto the -local one. Turning the file into a live database is a separate, deliberate restore -step, performed with Harper stopped. +`target=` only changes which node the snapshot is read from. It writes a file and nothing else: it does not restore, import, or clone that node's database onto the local one. Turning the file into a live database is a separate, deliberate restore step, performed with Harper stopped. -Note also that `get_backup` always streams the current state. It cannot download a -historical managed backup, so it is a way to take a fresh off-host copy, not a way -to export your retention history. +Note also that `get_backup` always streams the current state. It cannot download a historical managed backup, so it is a way to take a fresh off-host copy, not a way to export your retention history. -Whichever path you choose, the destination must not share a failure domain with -the source. A second directory on the same volume survives a deleted file and -nothing else. +Whichever path you choose, the destination must not share a failure domain with the source. A second directory on the same volume survives a deleted file and nothing else. ## Verification is not optional @@ -209,25 +133,15 @@ nothing else. } ``` -`verify_backup` checks the RocksDB file sizes, their checksums when -`verify_checksum` is `true`, which is slower, and the framing of the -transaction-log snapshot, which is always checked. +`verify_backup` checks the RocksDB file sizes, their checksums when `verify_checksum` is `true`, which is slower, and the framing of the transaction-log snapshot, which is always checked. -**Blob contents are not verified.** Verification confirms that a backup which -recorded blobs still has its blob snapshot present, and fails the backup as corrupt -if that snapshot is missing. It does not check the individual blob files for size, -checksum, or readability. So verification can pass on a backup whose blobs are -damaged, and only a real restore proves they are intact. +**Blob contents are not verified.** Verification confirms that a backup which recorded blobs still has its blob snapshot present, and fails the backup as corrupt if that snapshot is missing. It does not check the individual blob files for size, checksum, or readability. So verification can pass on a backup whose blobs are damaged, and only a real restore proves they are intact. -More generally, a verified backup is a well-formed backup, not a proven recovery. -The only evidence that your recovery works is a restore you have actually -performed, which is why the drill at the end of this guide is the point of it. +More generally, a verified backup is a well-formed backup, not a proven recovery. The only evidence that your recovery works is a restore you have actually performed, which is why the drill at the end of this guide is the point of it. ## Know your restore constraints before the incident -RocksDB is single-writer, so an in-place restore requires the database to be fully -closed first. That produces hard constraints you cannot negotiate during an -outage: +RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. That produces hard constraints you cannot negotiate during an outage: | Database | Online restore, server running | Offline restore, server stopped | | ---------------------------------------------- | ------------------------------------ | ------------------------------- | @@ -235,176 +149,96 @@ outage: | A user database a loaded component holds open | No, the job ends in `ERROR` | Yes | | The `system` database | No, rejected before a job is created | Yes | -Because the restore runs as a background job, a component holding the database -open does not fail your request. It fails inside the job, so you find out from -`get_job` rather than from the response. Harper does not track which component -uses which database, so it cannot selectively stop one. +Because the restore runs as a background job, a component holding the database open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. :::tip -That last constraint has an escape hatch worth knowing before you need it. Setting -`HARPER_SAFE_MODE` to any value starts Harper without loading any applications or -components, while the database, Operations API, and HTTP server come up normally: +That last constraint has an escape hatch worth knowing before you need it. Setting `HARPER_SAFE_MODE` to any value starts Harper without loading any applications or components, while the database, Operations API, and HTTP server come up normally: ```bash HARPER_SAFE_MODE=1 harper run ``` -Because no component loads, nothing holds a user database open, so a restore that -would fail inside its job can run online. It is also how you get back in when a -broken component is what stopped Harper in the first place, since the Operations -API is available to inspect, repair, or remove it. Traffic should stay off the -node either way. See -[safe mode](/reference/v5/configuration/overview#harper_safe_mode). +Because no component loads, nothing holds a user database open, so a restore that would fail inside its job can run online. It is also how you get back in when a broken component is what stopped Harper in the first place, since the Operations API is available to inspect, repair, or remove it. Traffic should stay off the node either way. See [safe mode](/reference/v5/configuration/overview#harper_safe_mode). ::: Two more constraints worth writing into your procedure: -- **`target_database` requires the server stopped.** Restoring into a separate - database rather than overwriting the source is CLI-only with Harper down. The - target must not already exist or must be an empty directory. -- **An interrupted restore leaves the database unloadable.** On a crash or power - loss mid-restore, Harper marks the database as incompletely restored and refuses - to load it on the next start. Recover by rerunning `restore_backup` for the same - database and `backup_id`. Do not try to load or hand-repair the directory. +- **`target_database` requires the server stopped.** Restoring into a separate database rather than overwriting the source is CLI-only with Harper down. The target must not already exist or must be an empty directory. +- **An interrupted restore leaves the database unloadable.** On a crash or power loss mid-restore, Harper marks the database as incompletely restored and refuses to load it on the next start. Recover by rerunning `restore_backup` for the same database and `backup_id`. Do not try to load or hand-repair the directory. ## Restoring in a replicated cluster -A restore is a point-in-time rollback of one node's data. In a cluster, that node -then has to rejoin peers that never rolled back, so sequence it deliberately: - -1. Take the node out of rotation with the availability flag, per - [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). - Never restore a node that is serving traffic. -2. **Isolate it from its peers.** This is a separate action from step 1, and the - step most likely to be skipped. See the warning below. -3. Decide what should happen to the restored data: whether it should propagate to - peers, or be overwritten by them. Those are two very different outcomes and it - is not a decision to improvise once the node is back on the mesh. +A restore is a point-in-time rollback of one node's data. In a cluster, that node then has to rejoin peers that never rolled back, so sequence it deliberately: + +1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. +2. **Isolate it from its peers.** This is a separate action from step 1, and the step most likely to be skipped. See the warning below. +3. Decide what should happen to the restored data: whether it should propagate to peers, or be overwritten by them. Those are two very different outcomes and it is not a decision to improvise once the node is back on the mesh. 4. Perform the restore, online or offline according to the table above. 5. Verify data expectations locally before any peer sees the node. -6. Re-admit the node through the full return sequence, including convergence - verification. +6. Re-admit the node through the full return sequence, including convergence verification. :::danger -**The availability flag does not stop replication.** It tells your traffic layer -to stop sending user requests. Replication is a separate path: peer connections -stay up, and the node keeps exchanging transactions the whole time it is "out of -rotation." - -So a node that is drained but still connected is not isolated. Restore it in place -and the rolled-back database is live on the mesh immediately, before you have -validated anything. Two things can then happen, both bad: peers overwrite your -restored data with newer transactions, or your rolled-back state propagates -outward to peers that were never affected. - -Isolation is its own step. Stopping Harper and restoring offline is the -unambiguous way to get it, which is a further argument for the offline path in a -cluster even where the table above permits an online restore. If you restore -online, you must have removed the node from the replication topology first, and -put it back deliberately in step 6. +**The availability flag does not stop replication.** It tells your traffic layer to stop sending user requests. Replication is a separate path: peer connections stay up, and the node keeps exchanging transactions the whole time it is "out of rotation." + +So a node that is drained but still connected is not isolated. Restore it in place and the rolled-back database is live on the mesh immediately, before you have validated anything. Two things can then happen, both bad: peers overwrite your restored data with newer transactions, or your rolled-back state propagates outward to peers that were never affected. + +Isolation is its own step. Stopping Harper and restoring offline is the unambiguous way to get it, which is a further argument for the offline path in a cluster even where the table above permits an online restore. If you restore online, you must have removed the node from the replication topology first, and put it back deliberately in step 6. ::: ## What backups cannot fix -If a release changed the meaning of persisted data, restoring the database is -usually the wrong response. A restore rolls back every write in the window, -including all the correct ones from the same period, so you can violate your -recovery point objective in the course of fixing a code bug. +If a release changed the meaning of persisted data, restoring the database is usually the wrong response. A restore rolls back every write in the window, including all the correct ones from the same period, so you can violate your recovery point objective in the course of fixing a code bug. -For that failure class, define forward repair instead: a targeted correction, or a -replay from the transaction log, that fixes the affected records and leaves the -rest alone. Decide which of your failure classes get restore and which get forward -repair while you are calm, and record the decision. See -[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this -interacts with release reversal. +For that failure class, define forward repair instead: a targeted correction, or a replay from the transaction log, that fixes the affected records and leaves the rest alone. Decide which of your failure classes get restore and which get forward repair while you are calm, and record the decision. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this interacts with release reversal. ### Prove it -A backup you have never restored is a hypothesis. Run this on a non-production -cluster and record the numbers: +A backup you have never restored is a hypothesis. Run this on a non-production cluster and record the numbers: 1. Create a managed backup, then verify it with `verify_checksum: true`. -2. Take an off-host copy using the correct procedure for your mechanism, quiesced - or from an atomic snapshot if you are copying a managed repository. +2. Take an off-host copy using the correct procedure for your mechanism, quiesced or from an atomic snapshot if you are copying a managed repository. 3. Destroy the source database. -4. Restore it, taking the node out of rotation first, and time from decision to - restored service. That is your measured recovery time. -5. Determine how much data was actually lost against the backup timestamp. That is - your measured recovery point. -6. Validate correctness through the application, not just at the storage layer. - Row counts agreeing is not the same as the journey working. -7. Repeat for the `system` database specifically, offline, since it has different - constraints and it is the one people never rehearse. - -Both measured numbers feed -[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they -do not meet your stated targets, one of the two has to change, and it is better -that it is the target than a promise you cannot keep. +4. Restore it, taking the node out of rotation first, and time from decision to restored service. That is your measured recovery time. +5. Determine how much data was actually lost against the backup timestamp. That is your measured recovery point. +6. Validate correctness through the application, not just at the storage layer. Row counts agreeing is not the same as the journey working. +7. Repeat for the `system` database specifically, offline, since it has different constraints and it is the one people never rehearse. + +Both measured numbers feed [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they do not meet your stated targets, one of the two has to change, and it is better that it is the target than a promise you cannot keep. ## Operational notes -- **Retention is a policy, not a side effect of disk space.** Use - `purge_backups` with an explicit `keep_count` that matches a written retention - decision. -- **Backup operations are `super_user` through the server, and filesystem - permissions offline.** An operator with shell access on a node can restore - without an API credential, so protect the host accordingly. -- **Schedule backups per database, matched to that database's recovery point.** - A single cluster-wide cadence overspends on some databases and underspends on - the ones that matter. -- **Alert on backup age, and on job failure.** A `create_backup` job that fails - silently produces a gap you will find at the worst possible time. Backup age - exceeding your recovery point objective is a pageable condition. -- **Record the storage engine per database in your inventory** and re-check after - migrations, since your entire mechanism choice depends on it. +- **Retention is a policy, not a side effect of disk space.** Use `purge_backups` with an explicit `keep_count` that matches a written retention decision. +- **Backup operations are `super_user` through the server, and filesystem permissions offline.** An operator with shell access on a node can restore without an API credential, so protect the host accordingly. +- **Schedule backups per database, matched to that database's recovery point.** A single cluster-wide cadence overspends on some databases and underspends on the ones that matter. +- **Alert on backup age, and on job failure.** A `create_backup` job that fails silently produces a gap you will find at the worst possible time. Backup age exceeding your recovery point objective is a pageable condition. +- **Record the storage engine per database in your inventory** and re-check after migrations, since your entire mechanism choice depends on it. ## Readiness checklist - [ ] Failure classes enumerated, with the mechanism that addresses each - [ ] Recovery point and recovery time stated per database, not per cluster - [ ] Storage engine confirmed per database, and the mechanism chosen accordingly -- [ ] No database in scope uses per-table storage paths, or its exclusion is known - and accepted +- [ ] No database in scope uses per-table storage paths, or its exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database -- [ ] Backup volume sized against the non-incremental transaction-log snapshot, - and against full blob copies if the backup and blob paths are on different - filesystems -- [ ] An off-host copy exists in a destination that does not share a failure - domain -- [ ] Managed repository copies take the whole `/` directory, - quiesced or from an atomic snapshot -- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least - periodically -- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms - only that the blob snapshot is present -- [ ] Restore constraints documented for user databases, component-held databases, - and `system` +- [ ] Backup volume sized against the non-incremental transaction-log snapshot, and against full blob copies if the backup and blob paths are on different filesystems +- [ ] An off-host copy exists in a destination that does not share a failure domain +- [ ] Managed repository copies take the whole `/` directory, quiesced or from an atomic snapshot +- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least periodically +- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms only that the blob snapshot is present +- [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged -- [ ] Restore procedure isolates the node from replication, not just from user - traffic, since the availability flag does not stop peer exchange -- [ ] Safe mode known as the way to restore a database a component holds open, - and to reach a node a broken component stopped -- [ ] Restore drill completed and dated, with measured recovery time and recovery - point +- [ ] Restore procedure isolates the node from replication, not just from user traffic, since the availability flag does not stop peer exchange +- [ ] Safe mode known as the way to restore a database a component holds open, and to reach a node a broken component stopped +- [ ] Restore drill completed and dated, with measured recovery time and recovery point - [ ] `system` database restore rehearsed offline - [ ] Forward repair defined for corruption caused by application code ## Additional Resources -- [Backups overview](/reference/v5/backups/overview) for how managed backups work, - the full limitation list, and manual restore examples -- [Backup operations](/reference/v5/backups/operations) for every parameter of - `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, - `purge_backups`, `restore_backup`, and `get_backup` -- [Storage configuration](/reference/v5/configuration/options#storage) for - `storage.backupPath`, `storage.path`, and `storage.blobPaths` -- [Jobs](/reference/v5/operations-api/operations#jobs) and - [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking - long-running backup operations -- [CLI operations](/reference/v5/cli/operations-api-commands) and - [remote operations](/reference/v5/cli/overview#remote-operations) for the - offline and `target=` forms -- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for - transaction history usable in forward repair -- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for - turning the measured numbers into commitments +- [Backups overview](/reference/v5/backups/overview) for how managed backups work, the full limitation list, and manual restore examples +- [Backup operations](/reference/v5/backups/operations) for every parameter of `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, `restore_backup`, and `get_backup` +- [Storage configuration](/reference/v5/configuration/options#storage) for `storage.backupPath`, `storage.path`, and `storage.blobPaths` +- [Jobs](/reference/v5/operations-api/operations#jobs) and [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking long-running backup operations +- [CLI operations](/reference/v5/cli/operations-api-commands) and [remote operations](/reference/v5/cli/overview#remote-operations) for the offline and `target=` forms +- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for transaction history usable in forward repair +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for turning the measured numbers into commitments diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index af0a44dd7..6b1cc11d0 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -3,35 +3,25 @@ title: Engineering RPO, RTO, and Uptime sidebar_position: 8 --- -Recovery point, recovery time, and availability are usually written down once, in -a document nobody consults, as three numbers that were never checked against a -mechanism. This guide is about making them real: stating them per journey, -mapping each failure scenario to the mechanism that addresses it, and reconciling -the targets against numbers you have actually measured. +Recovery point, recovery time, and availability are usually written down once, in a document nobody consults, as three numbers that were never checked against a mechanism. This guide is about making them real: stating them per journey, mapping each failure scenario to the mechanism that addresses it, and reconciling the targets against numbers you have actually measured. -If a target and a measurement disagree, one of them has to change. The point of -doing this deliberately is that you get to choose which, rather than finding out -during an incident that the answer was the target all along. +If a target and a measurement disagree, one of them has to change. The point of doing this deliberately is that you get to choose which, rather than finding out during an incident that the answer was the target all along. ## What You Will Learn -- How to state recovery point, recovery time, and availability so they are - testable rather than aspirational +- How to state recovery point, recovery time, and availability so they are testable rather than aspirational - A scenario-to-mechanism map, with what has to be proven for each to count -- How to convert an availability target into a monthly budget, and what each tier - demands of your automation +- How to convert an availability target into a monthly budget, and what each tier demands of your automation - How to reconcile stated targets against the numbers measured in earlier guides - Why one storage setting can silently invalidate your stated recovery point - Why declaring your degraded modes is as important as declaring your targets ## Prerequisites -- Measured numbers from the earlier guides: per-node capacity, convergence time, - node re-entry time, release reversal time, and restore time +- Measured numbers from the earlier guides: per-node capacity, convergence time, node re-entry time, release reversal time, and restore time - [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity model - [Backup and Recovery](./backup-and-recovery.mdx) for the recovery mechanisms -- A named owner who can approve or reject a target, because this exercise produces - decisions rather than data +- A named owner who can approve or reject a target, because this exercise produces decisions rather than data ## State the three per journey @@ -43,52 +33,25 @@ during an incident that the answer was the target all along. Three details make the difference between a testable statement and a slogan. -**Per journey, not per cluster.** A catalog browse and an order submission fail -differently, matter differently, and recover differently. One cluster-wide number -overcommits on the cheap journey and undercommits on the expensive one. +**Per journey, not per cluster.** A catalog browse and an order submission fail differently, matter differently, and recover differently. One cluster-wide number overcommits on the cheap journey and undercommits on the expensive one. -**Recovery time starts when the journey breaks, not when someone acts on it.** -Detection is inside the clock, not before it, and the gap between something -breaking and someone knowing is often the largest part of an outage. Starting the -count at the moment a human decided to act, or even at the moment an alert fired, -produces a number that flatters your automation and misleads your planning. Your -users were already down for all of it. +**Recovery time starts when the journey breaks, not when someone acts on it.** Detection is inside the clock, not before it, and the gap between something breaking and someone knowing is often the largest part of an outage. Starting the count at the moment a human decided to act, or even at the moment an alert fired, produces a number that flatters your automation and misleads your planning. Your users were already down for all of it. -**Name the measurement boundary.** Availability measured at your CDN edge and -availability measured at a Harper node are different quantities, and the -difference is exactly the part of the stack you may not control. Whichever you -choose, say so in the same sentence as the number. +**Name the measurement boundary.** Availability measured at your CDN edge and availability measured at a Harper node are different quantities, and the difference is exactly the part of the stack you may not control. Whichever you choose, say so in the same sentence as the number. ### One storage setting can override your recovery point -Your recovery point objective is a statement about how much acknowledged data you -can afford to lose. One configuration option can quietly make that statement -false. - -[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) -defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, -so a write returns as soon as it is queued to the operating system's page cache. -Throughput on write-heavy workloads improves substantially. The cost is that a -power loss or OS crash between the commit and the flush to disk loses the most -recent transactions, even though your application was told they succeeded. The -database stays structurally consistent; only the newest writes are gone. - -The trap is that this is invisible to everything else in this guide. Backup -timestamps look right, replication looks healthy, and no alert fires. You discover -it the first time a node loses power, which is also the first time the gap between -your stated recovery point and your real one matters. - -So treat it as an RPO decision rather than a performance one. Enabling it is -defensible when the data is reproducible from an upstream source, when the -workload is bulk ingest that can simply be re-run, or when peers acknowledge -writes before the loss window opens. Outside those cases, leave it off. If it is -on anywhere, record which databases and say so next to the recovery point in your -plan, because the number no longer means what it appears to. +Your recovery point objective is a statement about how much acknowledged data you can afford to lose. One configuration option can quietly make that statement false. + +[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, so a write returns as soon as it is queued to the operating system's page cache. Throughput on write-heavy workloads improves substantially. The cost is that a power loss or OS crash between the commit and the flush to disk loses the most recent transactions, even though your application was told they succeeded. The database stays structurally consistent; only the newest writes are gone. + +The trap is that this is invisible to everything else in this guide. Backup timestamps look right, replication looks healthy, and no alert fires. You discover it the first time a node loses power, which is also the first time the gap between your stated recovery point and your real one matters. + +So treat it as an RPO decision rather than a performance one. Enabling it is defensible when the data is reproducible from an upstream source, when the workload is bulk ingest that can simply be re-run, or when peers acknowledge writes before the loss window opens. Outside those cases, leave it off. If it is on anywhere, record which databases and say so next to the recovery point in your plan, because the number no longer means what it appears to. ## Map scenarios to mechanisms -For each scenario, know the mechanism, and know what must be proven for the -mechanism to count. The right-hand column is the whole exercise. +For each scenario, know the mechanism, and know what must be proven for the mechanism to count. The right-hand column is the whole exercise. | Scenario | Primary mechanism | What must be proven | | ------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | @@ -105,10 +68,7 @@ mechanism to count. The right-hand column is the whole exercise. | Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | | Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | -Two rows in that table are the ones most often missing. Node replacement is -budgeted as though it were a restart, when it is a full synchronization and can be -much slower. And the downstream dependency row usually has no answer at all, which -means the answer is whatever the code happens to do. +Two rows in that table are the ones most often missing. Node replacement is budgeted as though it were a restart, when it is a full synchronization and can be much slower. And the downstream dependency row usually has no answer at all, which means the answer is whatever the code happens to do. ## Convert availability into a budget @@ -120,23 +80,15 @@ An availability target is a quantity of unavailability you may spend per month. | 99.95% | 21 min 55 sec | Automated traffic removal and a rehearsed release reversal become necessary rather than nice | | 99.99% | 4 min 23 sec | Node failure has to be close to transparent. Detection, traffic removal, and validation must be automated, since no human response fits | -Figures assume a 30.44 day month. Your contractual definition, exclusions, and -measurement boundary may differ. +Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ. -Read down that right-hand column rather than the middle one. The tier you choose -determines how much automation you are committing to build, not how many nodes you -buy. Adding nodes without automating detection and traffic removal moves you -across the table's rows without moving you down its column. +Read down that right-hand column rather than the middle one. The tier you choose determines how much automation you are committing to build, not how many nodes you buy. Adding nodes without automating detection and traffic removal moves you across the table's rows without moving you down its column. -Treat the budget as spendable. If the month's remaining budget is small, that is -an argument for pausing discretionary change, and if it is large, that is -permission to ship. An error budget that never changes anyone's behavior is -just a number in a document. +Treat the budget as spendable. If the month's remaining budget is small, that is an argument for pausing discretionary change, and if it is large, that is permission to ship. An error budget that never changes anyone's behavior is just a number in a document. ## Reconcile targets against measurements -This is the step that turns the exercise into engineering. Fill in both columns -and look for rows where they disagree. +This is the step that turns the exercise into engineering. Fill in both columns and look for rows where they disagree. | Commitment | Measured from | Your target | Your measurement | | --------------------------- | ---------------------------------------------------------------------------------------- | ----------- | ---------------- | @@ -152,85 +104,50 @@ and look for rows where they disagree. Three reconciliations catch most problems: -- **Backup cadence against recovery point.** If you back up every six hours, your - recovery point cannot be one hour, no matter what the document says. -- **Restore time plus detection time against recovery time.** Recovery time - includes noticing. A twenty minute restore behind a forty minute detection gap - is a one hour recovery. -- **Convergence time against your admission gate.** If your gate admits a node - faster than it converges, you are serving stale data on purpose and calling it - availability. +- **Backup cadence against recovery point.** If you back up every six hours, your recovery point cannot be one hour, no matter what the document says. +- **Restore time plus detection time against recovery time.** Recovery time includes noticing. A twenty minute restore behind a forty minute detection gap is a one hour recovery. +- **Convergence time against your admission gate.** If your gate admits a node faster than it converges, you are serving stale data on purpose and calling it availability. -Where a target and a measurement disagree, the resolution is one of three things: -invest in the mechanism, relax the target, or accept the gap explicitly with an -owner and a date. All three are legitimate. Leaving it unreconciled is not. +Where a target and a measurement disagree, the resolution is one of three things: invest in the mechanism, relax the target, or accept the gap explicitly with an owner and a date. All three are legitimate. Leaving it unreconciled is not. ## Declare your degraded modes -Between fully working and fully down there is a range of states, and if you have -not decided what they should be, the code has decided for you. +Between fully working and fully down there is a range of states, and if you have not decided what they should be, the code has decided for you. -For each critical journey, write down what happens when a dependency is -unavailable, when a node is behind on replication, and when the cluster is below -its capacity budget. Reads served from slightly stale data may be entirely -acceptable for a catalog and entirely unacceptable for a balance check, and the -right answer differs by journey rather than by system. +For each critical journey, write down what happens when a dependency is unavailable, when a node is behind on replication, and when the cluster is below its capacity budget. Reads served from slightly stale data may be entirely acceptable for a catalog and entirely unacceptable for a balance check, and the right answer differs by journey rather than by system. -Then decide who can declare a degraded mode, and whether the declaration is -manual or automatic. A degraded mode nobody is authorized to invoke is not a -degraded mode. +Then decide who can declare a degraded mode, and whether the declaration is manual or automatic. A degraded mode nobody is authorized to invoke is not a degraded mode. ### Prove it Two passes, and both are needed. -**Tabletop the whole scenario table.** For each row, walk through who detects it, -what they do, what mechanism carries the recovery, and what evidence confirms -success. Rows where the group cannot answer without speculating are the gaps, and -finding them costs an hour rather than an outage. +**Tabletop the whole scenario table.** For each row, walk through who detects it, what they do, what mechanism carries the recovery, and what evidence confirms success. Rows where the group cannot answer without speculating are the gaps, and finding them costs an hour rather than an outage. -**Then run one real drill per scenario class**, spread over a quarter rather than -attempted in a day. The classes are: node loss, node return, node replacement, -release reversal, restore, and dependency failure. Record the date, the measured -numbers, what surprised you, and what you changed as a result. +**Then run one real drill per scenario class**, spread over a quarter rather than attempted in a day. The classes are: node loss, node return, node replacement, release reversal, restore, and dependency failure. Record the date, the measured numbers, what surprised you, and what you changed as a result. -The surprises are the deliverable. A drill that goes exactly as expected has -confirmed your documentation. A drill that does not has found the thing that -would have hurt you. +The surprises are the deliverable. A drill that goes exactly as expected has confirmed your documentation. A drill that does not has found the thing that would have hurt you. ## Operational notes -- **Recovery numbers expire.** Every measurement here is a property of a Harper - version, a data volume, a topology, and a set of components. Re-measure after - version upgrades and significant data growth, and date every number you record. -- **Uptime and recovery targets belong in the same document as the mechanism.** A - target stored separately from its supporting evidence drifts from reality within - a quarter. -- **Do not let a single incident rewrite your targets.** Adjust targets from - measurement and business need, not from the last thing that went wrong. -- **Exclusions matter as much as the number.** Planned maintenance, third-party - dependency failures, and client-side problems are usually excluded from an - availability calculation. Whether yours excludes them changes the number - substantially, so it belongs in writing. -- **A target you cannot measure is not a target.** If nothing in your monitoring - produces the number in your commitment, the first investment is measurement, not - more nodes. +- **Recovery numbers expire.** Every measurement here is a property of a Harper version, a data volume, a topology, and a set of components. Re-measure after version upgrades and significant data growth, and date every number you record. +- **Uptime and recovery targets belong in the same document as the mechanism.** A target stored separately from its supporting evidence drifts from reality within a quarter. +- **Do not let a single incident rewrite your targets.** Adjust targets from measurement and business need, not from the last thing that went wrong. +- **Exclusions matter as much as the number.** Planned maintenance, third-party dependency failures, and client-side problems are usually excluded from an availability calculation. Whether yours excludes them changes the number substantially, so it belongs in writing. +- **A target you cannot measure is not a target.** If nothing in your monitoring produces the number in your commitment, the first investment is measurement, not more nodes. ## Readiness checklist - [ ] Recovery point stated per database, in time units -- [ ] Recovery time stated per journey, measured from the start of failure and - including detection time +- [ ] Recovery time stated per journey, measured from the start of failure and including detection time - [ ] Availability stated per journey, with the measurement boundary named - [ ] Exclusions documented - [ ] Every scenario in the map has a named mechanism -- [ ] Every mechanism has its "what must be proven" satisfied or explicitly - outstanding +- [ ] Every mechanism has its "what must be proven" satisfied or explicitly outstanding - [ ] Availability target converted to minutes per month - [ ] Error budget policy states what changes when the budget runs low - [ ] Backup cadence reconciled against recovery point -- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded - alongside their recovery point +- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded alongside their recovery point - [ ] Restore time plus detection time reconciled against recovery time - [ ] Convergence time reconciled against the traffic admission gate - [ ] Every unreconciled gap has an owner and a date @@ -240,17 +157,11 @@ would have hurt you. ## Additional Resources -- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity - invariant behind the surviving-capacity commitment -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) - for drain, admission, and failback timing +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity invariant behind the surviving-capacity commitment +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) for drain, admission, and failback timing - [Operating Replication](./operating-replication.mdx) for convergence measurement -- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release - reversal timing -- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and - mechanism limits +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release reversal timing +- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and mechanism limits - [Monitoring and Triage](./monitoring-and-triage.mdx) for time to detection -- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the - gate this feeds -- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record - the results +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the gate this feeds +- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record the results diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index da0339e74..a4d48196e 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -6,36 +6,24 @@ sidebar_position: 2 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -The default health check on most load balancers asks one question: did something -answer on this port? A Harper node answers that question correctly while still -being the wrong place to send a user request, because a node can be running before -its databases have synchronized, while it is catching up on missed transactions, or -while a dependency it needs for one particular journey is down. +The default health check on most load balancers asks one question: did something answer on this port? A Harper node answers that question correctly while still being the wrong place to send a user request, because a node can be running before its databases have synchronized, while it is catching up on missed transactions, or while a dependency it needs for one particular journey is down. -This guide builds four separate signals so your traffic layer can tell the -difference between "the process is alive" and "this node will serve this journey -correctly," and then uses them to drain and return a node safely. +This guide builds four separate signals so your traffic layer can tell the difference between "the process is alive" and "this node will serve this journey correctly," and then uses them to drain and return a node safely. ## What You Will Learn -- Why liveness, availability, readiness, and journey verification are four signals - rather than one, and which layer consumes each -- How to install and drive the `@harperdb/status-check` component to control - whether a node advertises itself as available -- How to write an application readiness route that checks only what the routed - journey needs +- Why liveness, availability, readiness, and journey verification are four signals rather than one, and which layer consumes each +- How to install and drive the `@harperdb/status-check` component to control whether a node advertises itself as available +- How to write an application readiness route that checks only what the routed journey needs - The order of operations for draining a node and returning it to service - What makes a readiness endpoint safe under load, and what makes one dangerous ## Prerequisites -- A Harper cluster with at least two nodes, so draining one leaves a service - behind ([Fabric](/fabric) or self-managed) +- A Harper cluster with at least two nodes, so draining one leaves a service behind ([Fabric](/fabric) or self-managed) - A `super_user` credential for the Operations API -- A traffic layer you can configure health checks on: a load balancer, a CDN - origin group, or a service mesh -- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your - ports written down +- A traffic layer you can configure health checks on: a load balancer, a CDN origin group, or a service mesh +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your ports written down ## Four signals, not one @@ -46,17 +34,11 @@ correctly," and then uses them to drain and return a node safely. | Application readiness | A component route you write on `9926` | This node can serve this journey right now | Traffic layer, or your own gating | | Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | -The distinction that does the most work is the second one. The availability flag is -not a measurement, it is a declaration. It exists so that you, or your automation, -can take a node out of rotation deliberately, before doing something to it, and -put it back afterwards. Nothing else in this list can be set by an operator, and -nothing else is safe to use as the primary routing signal. +The distinction that does the most work is the second one. The availability flag is not a measurement, it is a declaration. It exists so that you, or your automation, can take a node out of rotation deliberately, before doing something to it, and put it back afterwards. Nothing else in this list can be set by an operator, and nothing else is safe to use as the primary routing signal. ## Install the availability flag -[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper -component that adds a `/status` route on the application port. Deploy it like any -other component: +[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper component that adds a `/status` route on the application port. Deploy it like any other component: ```json { @@ -67,8 +49,7 @@ other component: } ``` -Or declare it in the root `harper-config.yaml`, so the component is part of the -node's configuration rather than something an operator has to remember to deploy: +Or declare it in the root `harper-config.yaml`, so the component is part of the node's configuration rather than something an operator has to remember to deploy: ```yaml status-check: @@ -76,26 +57,16 @@ status-check: ``` :::note -That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, -typically `~/hdb`), not in an application's own `config.yaml`. The two files look -alike but behave differently: in the root config the entry name is free-form, while -in a component's `config.yaml` it must match a `package.json` dependency. A -component `config.yaml` also **replaces** Harper's default component -configuration outright instead of merging with it, so a file containing only this -entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and -`fastifyRoutes` defaults your application relies on. See -[applications](/reference/v5/components/applications). +That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, typically `~/hdb`), not in an application's own `config.yaml`. The two files look alike but behave differently: in the root config the entry name is free-form, while in a component's `config.yaml` it must match a `package.json` dependency. A component `config.yaml` also **replaces** Harper's default component configuration outright instead of merging with it, so a file containing only this entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and `fastifyRoutes` defaults your application relies on. See [applications](/reference/v5/components/applications). ::: -Once deployed, the route's contract is its status code, which is what lets a load -balancer consume it without parsing anything: +Once deployed, the route's contract is its status code, which is what lets a load balancer consume it without parsing anything: - `GET /status` returns `200` when the node is available, `404` when it is not - `POST /status` marks the node available (authenticated) - `DELETE /status` marks the node unavailable (authenticated) -It does also return a body, which is useful when you are checking by hand: a short -message on `200`, and an RFC 9457 problem-details document on `404`. +It does also return a body, which is useful when you are checking by hand: a short message on `200`, and an RFC 9457 problem-details document on `404`. @@ -128,48 +99,25 @@ await fetch(base, { method: 'POST', headers: auth }); // back in -Point your traffic layer's health check at `GET /status` on `9926`, not at the -Operations API and not at your application's root. A `404` is the node telling the -traffic layer to stop sending work, and it will keep saying so until something -sets it back. +Point your traffic layer's health check at `GET /status` on `9926`, not at the Operations API and not at your application's root. A `404` is the node telling the traffic layer to stop sending work, and it will keep saying so until something sets it back. :::note -The availability flag is node-local out of the box. The component stores it in a -table declared `replicate: false`, and the Operations API's own status values are -stored the same way, so neither propagates to peers. - -That is the property you want, and it is worth knowing why: a flag that replicated -would let one node's maintenance state reach its peers, turning a routine drain -into a cluster-wide outage. So the rule is to preserve it rather than to establish -it. If you fork the component or persist the flag some other way, keep -`replicate: false` on whatever holds it. See -[Operating Replication](./operating-replication.mdx). +The availability flag is node-local out of the box. The component stores it in a table declared `replicate: false`, and the Operations API's own status values are stored the same way, so neither propagates to peers. + +That is the property you want, and it is worth knowing why: a flag that replicated would let one node's maintenance state reach its peers, turning a routine drain into a cluster-wide outage. So the rule is to preserve it rather than to establish it. If you fork the component or persist the flag some other way, keep `replicate: false` on whatever holds it. See [Operating Replication](./operating-replication.mdx). ::: ### A note on `set_status` -The Operations API also offers -[`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) -for application-defined status values, with types for primary, maintenance, and -availability. +The Operations API also offers [`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) for application-defined status values, with types for primary, maintenance, and availability. -These are a coordination primitive for your own automation, not a health report, -and not a substitute for a real readiness check. Nothing in Harper acts on a value -you set through them. Prefer the `status-check` component for traffic admission, -because its contract is an HTTP status code that a load balancer can consume -directly, and reach for `set_status` when you need to coordinate something between -your own scripts. +These are a coordination primitive for your own automation, not a health report, and not a substitute for a real readiness check. Nothing in Harper acts on a value you set through them. Prefer the `status-check` component for traffic admission, because its contract is an HTTP status code that a load balancer can consume directly, and reach for `set_status` when you need to coordinate something between your own scripts. ## Write an application readiness route -Liveness and the availability flag both answer questions about the node. -Readiness answers a question about the journey: if traffic arrives for this route -right now, will it succeed? +Liveness and the availability flag both answer questions about the node. Readiness answers a question about the journey: if traffic arrives for this route right now, will it succeed? -The rule that keeps this useful is to check only the dependencies the routed -journey actually needs. A readiness route that checks everything will report a -node unready because of a subsystem that route never touches, and you will have -converted a partial degradation into a full outage yourself. +The rule that keeps this useful is to check only the dependencies the routed journey actually needs. A readiness route that checks everything will report a node unready because of a subsystem that route never touches, and you will have converted a partial degradation into a full outage yourself. Add a resource to your application's `resources.js`: @@ -211,28 +159,15 @@ export class Readiness extends Resource { } ``` -The `jsResource` plugin is enabled by default, so the route is served at -`/Readiness` on the application port as soon as the class is exported. See -[Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for -the resource and export mechanics. +The `jsResource` plugin is enabled by default, so the route is served at `/Readiness` on the application port as soon as the class is exported. See [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for the resource and export mechanics. :::note -Because the handler is a `static` method, it replaces Harper's built-in dispatch -along with the authorization check that lives inside it, so `GET /Readiness` is -unauthenticated. That is what a load balancer probe needs, and it is why the -endpoint must not return anything you would not publish. Keep the response to -check names and outcomes, never connection strings, credentials, or internal -hostnames. - -If you rewrite this as an instance `get()` instead, authorization comes back and -defaults to `super_user` only, which will make your probe start failing with an -authorization error rather than a readiness one. In that form you need -`allowRead() { return true; }` to keep it reachable. +Because the handler is a `static` method, it replaces Harper's built-in dispatch along with the authorization check that lives inside it, so `GET /Readiness` is unauthenticated. That is what a load balancer probe needs, and it is why the endpoint must not return anything you would not publish. Keep the response to check names and outcomes, never connection strings, credentials, or internal hostnames. + +If you rewrite this as an instance `get()` instead, authorization comes back and defaults to `super_user` only, which will make your probe start failing with an authorization error rather than a readiness one. In that form you need `allowRead() { return true; }` to keep it reachable. ::: -The sentinel is a record you create once, on purpose, and leave alone. Seed it on -every node before you point anything at this route, or readiness will report -`failed` forever and you will have built a probe that never passes: +The sentinel is a record you create once, on purpose, and leave alone. Seed it on every node before you point anything at this route, or readiness will report `failed` forever and you will have built a probe that never passes: ```bash curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel \ @@ -241,133 +176,76 @@ curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel -d '{"name":"readiness probe sentinel","description":"do not delete"}' ``` -Give it a name that says what it is, because the next person to find it will be -deciding whether it is safe to delete. If your table replicates, seeding it once -is enough; if it does not, seed it per node. +Give it a name that says what it is, because the next person to find it will be deciding whether it is safe to delete. If your table replicates, seeding it once is enough; if it does not, seed it per node. -Two details in that example are the point of it. The timeout on the downstream -call means a slow dependency cannot make your readiness check hang, which would -make the node look dead to a probe rather than unready. And returning the version -means that when you are staring at a dashboard during a rollout, the readiness -response itself tells you which build answered. +Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version means that when you are staring at a dashboard during a rollout, the readiness response itself tells you which build answered. ## Drain a node and bring it back -The order matters, and the verification steps between them are the parts people -skip. +The order matters, and the verification steps between them are the parts people skip. -1. **Confirm the peers can carry it, before you drain anything.** The remaining - nodes have to stay inside their capacity budget with this node gone. If they - cannot, stop here and do not drain: see - [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after - the traffic has already moved is how a maintenance window becomes an incident. +1. **Confirm the peers can carry it, before you drain anything.** The remaining nodes have to stay inside their capacity budget with this node gone. If they cannot, stop here and do not drain: see [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after the traffic has already moved is how a maintenance window becomes an incident. 2. **Declare it unavailable.** `DELETE /status` on the target node. -3. **Verify traffic actually stopped.** Watch request volume on the target fall to - zero and rise on its peers. Do not trust the configured weight; check the - measured request count from - [analytics](/reference/v5/analytics/overview) or your traffic layer's own - metrics. Health check intervals, DNS TTLs, and client-side connection reuse all - add delay here, and the delay is yours to measure. -4. **Watch the peers absorb it.** If they are going outside budget despite step 1, - abort: `POST /status` to put this node back in rotation, and re-plan the change - for a lower-traffic window or with added capacity. +3. **Verify traffic actually stopped.** Watch request volume on the target fall to zero and rise on its peers. Do not trust the configured weight; check the measured request count from [analytics](/reference/v5/analytics/overview) or your traffic layer's own metrics. Health check intervals, DNS TTLs, and client-side connection reuse all add delay here, and the delay is yours to measure. +4. **Watch the peers absorb it.** If they are going outside budget despite step 1, abort: `POST /status` to put this node back in rotation, and re-plan the change for a lower-traffic window or with added capacity. 5. **Do the work.** Restart, upgrade, reconfigure, or investigate. -1. **Confirm the databases are current.** `cluster_status` should show the expected - peer sockets connected for every database this node serves, and convergence - should be complete rather than in progress. -2. **Confirm readiness passes locally.** Call your `/Readiness` route directly - against the node, bypassing the traffic layer. -3. **Run the journey synthetic against the node directly.** A real read, or a safe - write, through the same path a user would take. +1. **Confirm the databases are current.** `cluster_status` should show the expected peer sockets connected for every database this node serves, and convergence should be complete rather than in progress. +2. **Confirm readiness passes locally.** Call your `/Readiness` route directly against the node, bypassing the traffic layer. +3. **Run the journey synthetic against the node directly.** A real read, or a safe write, through the same path a user would take. 4. **Declare it available.** `POST /status`. -5. **Hold before restoring full weight.** Give it a stability window at partial - traffic and watch error rate and latency against its peers before treating it as - fully back. +5. **Hold before restoring full weight.** Give it a stability window at partial traffic and watch error rate and latency against its peers before treating it as fully back. -Step 5 on the return side is the one worth defending in a review. Failback is a -change like any other, and a node that has just synchronized under load is the -most likely one to surprise you. Design failback, do not just design failover. +Step 5 on the return side is the one worth defending in a review. Failback is a change like any other, and a node that has just synchronized under load is the most likely one to surprise you. Design failback, do not just design failover. ## Readiness hygiene -- **Keep the response cheap and bounded.** A readiness check that performs a broad - scan or writes data will amplify an incident, because it runs at probe frequency - across every node at exactly the moment the system is already struggling. -- **Never make it the only routing signal.** Liveness plus availability plus - readiness, consumed at the right layers. -- **Probe from more than one location** where your traffic layer supports it. A - single probe point cannot distinguish a network path problem from a node problem. -- **Make it observable.** Record response code, latency, the reason for a failure, - and the node and component version. A readiness check whose failures you cannot - explain after the fact is a check you will end up ignoring. -- **Version the contract.** If you change what readiness means, that is a change - to the traffic admission policy, and it deserves the same care as a code - release. +- **Keep the response cheap and bounded.** A readiness check that performs a broad scan or writes data will amplify an incident, because it runs at probe frequency across every node at exactly the moment the system is already struggling. +- **Never make it the only routing signal.** Liveness plus availability plus readiness, consumed at the right layers. +- **Probe from more than one location** where your traffic layer supports it. A single probe point cannot distinguish a network path problem from a node problem. +- **Make it observable.** Record response code, latency, the reason for a failure, and the node and component version. A readiness check whose failures you cannot explain after the fact is a check you will end up ignoring. +- **Version the contract.** If you change what readiness means, that is a change to the traffic admission policy, and it deserves the same care as a code release. ### Prove it -On a non-production cluster, restart one node under representative read and write -load, using the full drain and return sequence above. Record three numbers: how -long from `DELETE /status` until measured traffic on that node reaches zero, -whether any user-visible errors occurred during the transition, and how long from -process start until the node legitimately passed all three return gates. +On a non-production cluster, restart one node under representative read and write load, using the full drain and return sequence above. Record three numbers: how long from `DELETE /status` until measured traffic on that node reaches zero, whether any user-visible errors occurred during the transition, and how long from process start until the node legitimately passed all three return gates. -That third number is your real node re-entry time, and it is almost always longer -than people assume, because it includes convergence rather than just startup. +That third number is your real node re-entry time, and it is almost always longer than people assume, because it includes convergence rather than just startup. ## Operational notes -- **A returning node is not the same as a new node.** A node whose databases have - never synchronized downloads them in full. A node that was offline and comes back - catches up on the transactions it missed. Both need to finish before traffic - arrives, but they take very different amounts of time, so do not budget for the - first when you are planning a routine restart. -- **Set unavailable before recovery work, not after.** Any operation that touches - data on a node, including a restore, should happen with the node out of rotation. -- **Configuration changes need a restart to take effect**, so a node that has been - reconfigured but not restarted is running the old configuration while reporting - the new one. Sequence the restart into the same maintenance window. -- **Fabric provides its own cluster-level health and routing.** These signals still - matter, because the availability flag and your readiness route are what Fabric's - routing has to consult. +- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A node that was offline and comes back catches up on the transactions it missed. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart. +- **Set unavailable before recovery work, not after.** Any operation that touches data on a node, including a restore, should happen with the node out of rotation. +- **Configuration changes need a restart to take effect**, so a node that has been reconfigured but not restarted is running the old configuration while reporting the new one. Sequence the restart into the same maintenance window. +- **Fabric provides its own cluster-level health and routing.** These signals still matter, because the availability flag and your readiness route are what Fabric's routing has to consult. ## Readiness checklist -- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` - rather than deployed by hand +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` rather than deployed by hand - [ ] Traffic layer health check points at `GET /status` on `9926` -- [ ] Availability flag storage still declares `replicate: false`, if you forked the - component or persist it yourself +- [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself - [ ] An application readiness route exists, scoped to one journey's dependencies - [ ] Every downstream call in the readiness route has a timeout - [ ] Readiness response includes the component version -- [ ] A critical-journey synthetic runs against the public route, separately from - the liveness probe +- [ ] A critical-journey synthetic runs against the public route, separately from the liveness probe - [ ] Drain sequence documented, with measured time-to-zero-traffic - [ ] Return sequence documented, with a stability window before full weight - [ ] Measured node re-entry time recorded, including convergence ## Additional Resources -- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component - source and options -- [Operations API operations](/reference/v5/operations-api/operations) for - `cluster_status`, `system_information`, and the status operations -- [Components overview](/reference/v5/components/overview) for the full list of - Harper-maintained components, including the Prometheus exporter -- [Analytics overview](/reference/v5/analytics/overview) for per-node request and - latency data -- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) - for custom resources and the `jsResource` plugin -- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers - can absorb a drained node +- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component source and options +- [Operations API operations](/reference/v5/operations-api/operations) for `cluster_status`, `system_information`, and the status operations +- [Components overview](/reference/v5/components/overview) for the full list of Harper-maintained components, including the Prometheus exporter +- [Analytics overview](/reference/v5/analytics/overview) for per-node request and latency data +- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for custom resources and the `jsResource` plugin +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers can absorb a drained node diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index 81e6160df..f3c60cd12 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -6,40 +6,26 @@ sidebar_position: 1 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -You have built an application, deployed it from a pipeline, and now real users are -going to depend on it. Operating Harper is not the same job as operating an -application tier in front of a database tier, because Harper does not have those -two tiers. One process holds your component code, the HTTP stack, the local -database, and the replication client. That shape removes several problems you may -be used to solving, and it changes where the remaining ones live. +You have built and deployed an application, and now real users are going to depend on it. Operating Harper is not the same as operating separate application and database tiers. In Harper, one process holds your component code, the HTTP stack, the local database, and the replication client. That "collapsed stack" architecture reduces several problems you may be used to solving. -This guide is the map for the rest of the Administration track. It covers what a -single node actually contains, what that means for failure and scale, and how to -take an inventory of your own deployment before you design anything around it. +This guide is the map for the rest of the Administration track. It covers what a single node actually contains, what that means for failure and scale, and how to take an inventory of your own deployment before you design anything around it. ## What You Will Learn - What a Harper node contains, and which port carries which kind of traffic -- Why the node is your unit of failure and your unit of scale, and what that - removes from your operating burden as well as what it adds +- Why the node is your unit of failure and your unit of scale, and what that removes from your operating burden as well as what it adds - The three ways a Harper runbook differs from a two-tier runbook -- How to inventory your service boundary with the Operations API, so later guides - have something concrete to work from +- How to inventory your service boundary with the Operations API, so later guides have something concrete to work from ## Prerequisites -- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a - [local installation](../getting-started/install-and-connect-harper.mdx) +- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a [local installation](../getting-started/install-and-connect-harper.mdx) - A `super_user` credential for the Operations API -- An application deployed to it - ([Create your First Application](../getting-started/create-your-first-application.mdx)) +- An application deployed to it ([Create your First Application](../getting-started/create-your-first-application.mdx)) ## What one node contains -A Harper node is a single process running your components, an HTTP server, a local -storage engine, and peer replication. There is no network hop between your -application code and the data it reads, no separate cache tier to keep coherent, -and no connection pool to tune between tiers. +A Harper node is a single process running your components, an HTTP server, a local storage engine, and peer replication. There is no network hop between your application code and the data it reads, no separate cache tier to keep coherent, and no connection pool to tune between tiers. Three ports carry the traffic you will operate around: @@ -49,9 +35,7 @@ Three ports carry the traffic you will operate around: | `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | | `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | -These are documented defaults, not guarantees about your cluster. Confirm the live -values rather than assuming them, because a replication port in particular can be -inherited from other configuration: +These are documented defaults, not guarantees about your cluster. Confirm the live values rather than assuming them, because a replication port in particular can be inherited from other configuration: ```json { @@ -59,68 +43,40 @@ inherited from other configuration: } ``` -Read back `http.port`, `operationsApi.network.port`, and the `replication` block. -Record what you find. Later guides in this track assume you know these numbers for -your own deployment. +Read back `http.port`, `operationsApi.network.port`, and the `replication` block. Record what you find. Later guides in this track assume you know these numbers for your own deployment. :::tip -Keep `9925` off any public route. The Operations API can deploy components, read -logs, and read configuration, so it is an administrative surface, not an -application one. See [security overview](/reference/v5/security/overview). +Keep `9925` off any public route. The Operations API can deploy components, read logs, and read configuration, so it is an administrative surface, not an application one. See [security overview](/reference/v5/security/overview). ::: ## The node is your unit of failure and your unit of scale -Because one process holds the runtime and the local data together, there is no -internal application-to-database seam that can fail over independently. The node -is the practical unit of service failure. +Because one process holds the runtime and the local data together, there is no internal application-to-database seam that can fail over independently. The node is the practical unit of service failure. Start with what that removes, because it is the larger half of the trade: -- No cross-tier network latency on data access, and no tail latency from a - saturated connection pool between tiers -- No cache invalidation problem between an application cache and a database of - record, because they are the same thing -- No partial-outage state where the application tier is healthy and the data tier - is not, which is the failure mode that produces the most confusing incidents +- No cross-tier network latency on data access, and no tail latency from a saturated connection pool between tiers +- No cache invalidation problem between an application cache and a database of record, because they are the same thing +- No partial-outage state where the application tier is healthy and the data tier is not, which is the failure mode that produces the most confusing incidents - One capacity number to measure and one thing to size -Then the consequence: when you lose a node, you lose a whole slice of your -service, not one layer of it. This is why the first real design decision in -[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak -throughput. It is how many nodes you are willing to lose at once, and whether the -survivors can carry the load. +Then the consequence: when you lose a node, you lose a whole slice of your service, not one layer of it. This is why the first real design decision in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak throughput. It is how many nodes you are willing to lose at once, and whether the survivors can carry the load. -Scale works on the same unit. Adding capacity means adding a node that carries -both request handling and data, so a scaling event is also a data movement event. -That is not a problem, but it is a thing with a duration, and traffic should not -arrive until it finishes. +Scale works on the same unit. Adding capacity means adding a node that carries both request handling and data, so a scaling event is also a data movement event. That is not a problem, but it is a thing with a duration, and traffic should not arrive until it finishes. ## Three differences that change your runbook ### A process that is up is not a node that should take traffic -A Harper process will answer a TCP connection and return an HTTP response before -it is a good place to send a user request. A new or replacement node has to -synchronize the databases it serves before its answers are correct. A returning -node has to catch up on transactions it missed. +A Harper process will answer a TCP connection and return an HTTP response before it is a good place to send a user request. A new or replacement node has to synchronize the databases it serves before its answers are correct. A returning node has to catch up on transactions it missed. -So liveness and admission are two different decisions, and a load balancer health -check that only proves liveness will route users to a node that is technically -running and functionally wrong. This is the whole subject of -[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), -and it is the single most common gap in a first production deployment. +So liveness and admission are two different decisions, and a load balancer health check that only proves liveness will route users to a node that is technically running and functionally wrong. This is the whole subject of [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), and it is the single most common gap in a first production deployment. ### Replication is peer-to-peer and scoped, so verify your own scope -Harper peers exchange data over WebSockets with mTLS on the secure replication -port and discover each other through configured routes. There is no primary. Data -mutations and transactions replicate; some things do not, and the scope is -configurable per database and per table. +Harper peers exchange data over WebSockets with mTLS on the secure replication port and discover each other through configured routes. There is no primary. Data mutations and transactions replicate; some things do not, and the scope is configurable per database and per table. -The important operating habit is not memorizing a default. It is checking what -your cluster actually replicates, because the answer depends on your -configuration, your version, and whether anyone has scoped it since: +The important operating habit is not memorizing a default. It is checking what your cluster actually replicates, because the answer depends on your configuration, your version, and whether anyone has scoped it since: ```json { @@ -128,31 +84,17 @@ configuration, your version, and whether anyone has scoped it since: } ``` -The response lists each peer connection and, within it, one socket per database -per peer. That tells you which databases are actually flowing, which is the -question that matters during an incident. What is in scope, what is deliberately -out of it, and how to prove convergence rather than just connection are covered in -[Operating Replication](./operating-replication.mdx). +The response lists each peer connection and, within it, one socket per database per peer. That tells you which databases are actually flowing, which is the question that matters during an incident. What is in scope, what is deliberately out of it, and how to prove convergence rather than just connection are covered in [Operating Replication](./operating-replication.mdx). ### Reversal is a redeploy, not an infrastructure event -Your application ships as a component, deployed with -[`deploy_component`](/reference/v5/operations-api/operations#deploy_component) -from an immutable reference. Rolling back means deploying the previous immutable -reference. There is no image to rebuild, no instance to replace, and no cluster to -rebuild to undo a bad release. +Your application ships as a component, deployed with [`deploy_component`](/reference/v5/operations-api/operations#deploy_component) from an immutable reference. Rolling back means deploying the previous immutable reference. There is no image to rebuild, no instance to replace, and no cluster to rebuild to undo a bad release. -That makes reversal fast enough to be a real option under pressure, which in turn -makes it worth designing for deliberately rather than improvising. It also means -code rollback, configuration rollback, and data recovery are three separate -actions with three different blast radii, and conflating them during an incident -is how a bad release becomes a data loss event. See -[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). +That makes reversal fast enough to be a real option under pressure, which in turn makes it worth designing for deliberately rather than improvising. It also means code rollback, configuration rollback, and data recovery are three separate actions with three different blast radii, and conflating them during an incident is how a bad release becomes a data loss event. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). ## Inventory your service boundary -Everything else in this track builds on knowing what you have. Run these four -operations against each node and write down the answers. +Everything else in this track builds on knowing what you have. Run these four operations against each node and write down the answers. @@ -185,22 +127,14 @@ await fetch('https://my-node.example.com:9925/', { :::warning -The `attributes` array silently drops names it does not recognize, so a typo -returns a smaller response rather than an error. The valid values are `system`, -`time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, -`metrics`, and `threads`. If a response is missing a section you asked for, check -the spelling before you check the node. +The `attributes` array silently drops names it does not recognize, so a typo returns a smaller response rather than an error. The valid values are `system`, `time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, `metrics`, and `threads`. If a response is missing a section you asked for, check the spelling before you check the node. ::: Then: -- [`get_components`](/reference/v5/components/applications#get_components) for - what is deployed, including which extensions are present -- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) - for what changed and when -- [`registration_info`](/reference/v5/operations-api/operations#registration_info) - for the Harper version, which `system_information` does not report (it returns - the node and npm versions, not Harper's) +- [`get_components`](/reference/v5/components/applications#get_components) for what is deployed, including which extensions are present +- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) for what changed and when +- [`registration_info`](/reference/v5/operations-api/operations#registration_info) for the Harper version, which `system_information` does not report (it returns the node and npm versions, not Harper's) - `cluster_status` for the peers and databases actually connected Fill in a boundary record you can keep: @@ -216,36 +150,18 @@ Fill in a boundary record you can keep: | Critical journeys | The user-facing paths that must work, named | | Downstream dependencies | Anything Harper calls that can fail independently | -The last two rows are the ones people skip and the ones that matter most. A -healthy Harper process cannot compensate for a failed downstream dependency or for -application logic returning wrong answers, so an operating model that only watches -Harper will miss the incidents your users actually notice. +The last two rows are the ones people skip and the ones that matter most. A healthy Harper process cannot compensate for a failed downstream dependency or for application logic returning wrong answers, so an operating model that only watches Harper will miss the incidents your users actually notice. ### Prove it -Before moving on, confirm the picture is real rather than assumed. On a -non-production cluster, stop one node and watch what happens to the others: -whether peers keep serving, how long the remaining nodes take to show the change -in `cluster_status`, and what your traffic layer does about it. You are not -measuring anything precisely yet. You are checking that the boundary you wrote -down matches the system you have. +Before moving on, confirm the picture is real rather than assumed. On a non-production cluster, stop one node and watch what happens to the others: whether peers keep serving, how long the remaining nodes take to show the change in `cluster_status`, and what your traffic layer does about it. You are not measuring anything precisely yet. You are checking that the boundary you wrote down matches the system you have. ## Operational notes -- **Fabric and self-managed differ in what you own, not in how Harper behaves.** - On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline - are managed for you. The failure unit, the replication model, and the admission - problem are identical. -- **Version parity across nodes is an operating requirement, not a nicety.** - Mixed versions in a cluster change replication and deployment behavior. Record - the version per node in your boundary inventory and alert on drift. -- **Configuration changes made through the API take effect on restart.** A - `set_configuration` call that has not been followed by a restart or - `restart_service` leaves a node running something other than its stated - configuration. Track pending changes in your change record. -- **`get_status` reports a `restartRequired` flag, but it tracks component and - code restarts rather than configuration changes.** Do not rely on it to tell you - a configuration change is still pending. +- **Fabric and self-managed differ in what you own, not in how Harper behaves.** On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline are managed for you. The failure unit, the replication model, and the admission problem are identical. +- **Version parity across nodes is an operating requirement, not a nicety.** Mixed versions in a cluster change replication and deployment behavior. Record the version per node in your boundary inventory and alert on drift. +- **Configuration changes made through the API take effect on restart.** A `set_configuration` call that has not been followed by a restart or `restart_service` leaves a node running something other than its stated configuration. Track pending changes in your change record. +- **`get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes.** Do not rely on it to tell you a configuration change is still pending. ## Readiness checklist @@ -260,17 +176,10 @@ down matches the system you have. ## Additional Resources -- [HTTP server reference](/reference/v5/http/overview) for the application port and - server architecture -- [Operations API overview](/reference/v5/operations-api/overview) and the - [full operation list](/reference/v5/operations-api/operations) -- [Replication overview](/reference/v5/replication/overview) for the peer model, - mTLS, routes, and scope -- [Components overview](/reference/v5/components/overview) for the component model - and the available extensions -- [Database overview](/reference/v5/database/overview) for storage engines and - transaction boundaries -- [Configuration overview](/reference/v5/configuration/overview) for - `harper-config.yaml` and restart requirements -- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how - application code reaches these nodes +- [HTTP server reference](/reference/v5/http/overview) for the application port and server architecture +- [Operations API overview](/reference/v5/operations-api/overview) and the [full operation list](/reference/v5/operations-api/operations) +- [Replication overview](/reference/v5/replication/overview) for the peer model, mTLS, routes, and scope +- [Components overview](/reference/v5/components/overview) for the component model and the available extensions +- [Database overview](/reference/v5/database/overview) for storage engines and transaction boundaries +- [Configuration overview](/reference/v5/configuration/overview) for `harper-config.yaml` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how application code reaches these nodes diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index d51d8b89b..035e8356d 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -6,21 +6,14 @@ sidebar_position: 5 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -A dashboard is useful when it lets an operator place a symptom. Users are seeing -errors: is it the edge, the traffic layer, one node, the replication path, a -downstream dependency, or the change someone shipped twenty minutes ago? A -dashboard that shows CPU across the cluster cannot answer that, which is why it -gets ignored during incidents. +A dashboard is useful when it lets an operator place a symptom. Users are seeing errors: is it the edge, the traffic layer, one node, the replication path, a downstream dependency, or the change someone shipped twenty minutes ago? A dashboard that shows CPU across the cluster cannot answer that, which is why it gets ignored during incidents. -This guide builds monitoring around the boundaries you can actually act on, names -the specific Harper metrics worth alerting on, and gives you a triage sequence -short enough to run under pressure. +This guide builds monitoring around the boundaries you can actually act on, names the specific Harper metrics worth alerting on, and gives you a triage sequence short enough to run under pressure. ## What You Will Learn - The layers a Harper symptom can live in, and the minimum signal for each -- Which Harper metrics are worth an alert, by name, and which are only worth a - dashboard +- Which Harper metrics are worth an alert, by name, and which are only worth a dashboard - How to get metrics out of Harper, on Fabric and self-managed - How to preload an APM or tracing agent so it can instrument your application - How to write log entries that are still useful during an incident @@ -30,10 +23,8 @@ short enough to run under pressure. - A cluster with your application deployed and taking traffic - A `super_user` credential for the Operations API -- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or - a Prometheus-compatible system for self-managed -- [Operating Replication](./operating-replication.mdx), since replication signals - are half of what you will watch +- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or a Prometheus-compatible system for self-managed +- [Operating Replication](./operating-replication.mdx), since replication signals are half of what you will watch ## Monitor at the boundaries you operate @@ -48,15 +39,11 @@ short enough to run under pressure. | Change | Component and config version, cohort, operator, start and end, outcome | `list_deployments`, `get_deployment`, `get_components`, `get_configuration` read-back | | Recovery | Backup age, job state, verification result, last restore drill | `list_backups`, `verify_backup`, `get_job` | -The Change row is the one teams leave out and the one that resolves incidents -fastest. Most production symptoms correlate with something a human did, so being -able to overlay deployments onto a latency graph is worth more than another -resource metric. +The Change row is the one teams leave out and the one that resolves incidents fastest. Most production symptoms correlate with something a human did, so being able to overlay deployments onto a latency graph is worth more than another resource metric. ## The signals worth an alert -Harper records a large standard metric set automatically. Most of it belongs on a -dashboard. This much belongs on a pager: +Harper records a large standard metric set automatically. Most of it belongs on a dashboard. This much belongs on a pager: | Alert on | Metric or source | Why this one | | -------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | @@ -71,32 +58,22 @@ dashboard. This much belongs on a pager: | Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | | Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | -Two notes on how to set these. Alert on the percentile that appears in your SLO, -because a mean latency graph will look fine through an incident that is failing -your slowest ten percent of users. And set saturation thresholds from the -per-node capacity work in -[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a -generic number, since the point where latency leaves your SLO is specific to your -application and your data. +Two notes on how to set these. Alert on the percentile that appears in your SLO, because a mean latency graph will look fine through an incident that is failing your slowest ten percent of users. And set saturation thresholds from the per-node capacity work in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a generic number, since the point where latency leaves your SLO is specific to your application and your data. ## Get the metrics out -Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into -`hdb_analytics`. You can query those directly, but for a real monitoring setup +Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into `hdb_analytics`. You can query those directly, but for a real monitoring setup export them. -Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards -over Harper's analytics without you building a pipeline, which is the fastest path -to the alert list above. +Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards over Harper's analytics without you building a pipeline, which is the fastest path to the alert list above. -Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) -and scrape it into whatever you already run: +Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) and scrape it into whatever you already run: ```json { @@ -107,31 +84,18 @@ and scrape it into whatever you already run: } ``` -Once deployed, the scrape endpoint is `//metrics` on the application -port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is -authorized, so give your scraper a credential rather than expecting it to be open. +Once deployed, the scrape endpoint is `//metrics` on the application port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is authorized, so give your scraper a credential rather than expecting it to be open. :::warning -**The exporter does not cover every metric in the alert table above.** It -translates the request and replication metrics, including `success`, `duration`, -and `replication-latency`, but it does not currently export -`main-thread-utilization` and its `taskQueueLatency` attribute, -`transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, -`storage-volume`, or `table-size`. The `utilization` it does expose is a thread -utilization figure drawn from `system_information`, which is not the same as the -analytics `utilization` metric. - -For the saturation, commit-time, and storage rows you will need to query -`hdb_analytics` directly or collect `system_information` on your own schedule. -Check what your version actually exports before you build a dashboard on the -assumption that everything above arrives in Prometheus. +**The exporter does not cover every metric in the alert table above.** It translates the request and replication metrics, including `success`, `duration`, and `replication-latency`, but it does not currently export `main-thread-utilization` and its `taskQueueLatency` attribute, `transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, `storage-volume`, or `table-size`. The `utilization` it does expose is a thread utilization figure drawn from `system_information`, which is not the same as the analytics `utilization` metric. + +For the saturation, commit-time, and storage rows you will need to query `hdb_analytics` directly or collect `system_information` on your own schedule. Check what your version actually exports before you build a dashboard on the assumption that everything above arrives in Prometheus. ::: -To find out what is actually available on your version rather than guessing from -documentation: +To find out what is actually available on your version rather than guessing from documentation: ```json { @@ -140,33 +104,19 @@ documentation: } ``` -Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for -the shape of any one of them. +Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for the shape of any one of them. -Your application can add its own metrics with -[`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), -which is how you get business-level signals such as checkout completion onto the -same dashboard as node saturation. That correlation is usually what tells you -whether a technical symptom matters. +Your application can add its own metrics with [`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), which is how you get business-level signals such as checkout completion onto the same dashboard as node saturation. That correlation is usually what tells you whether a technical symptom matters. :::tip -Check `analytics.replicate` in the -[analytics configuration](/reference/v5/analytics/overview#analytics-configuration) -before building dashboards. Whether metrics stay node-local or replicate changes -what a cluster-wide query means, and it is easier to decide that deliberately than -to discover it after building panels on the wrong assumption. +Check `analytics.replicate` in the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) before building dashboards. Whether metrics stay node-local or replicate changes what a cluster-wide query means, and it is easier to decide that deliberately than to discover it after building panels on the wrong assumption. ::: ## Run an APM or tracing agent -Harper's own metrics tell you how the node is behaving. They do not give you -distributed traces across your application code and its downstream calls, which is -what you want when a journey is slow and you need to know where the time went. -That comes from an instrumentation agent, and an agent has to load before the code -it instruments. +Harper's own metrics tell you how the node is behaving. They do not give you distributed traces across your application code and its downstream calls, which is what you want when a journey is slow and you need to know where the time went. That comes from an instrumentation agent, and an agent has to load before the code it instruments. -Two configuration keys put a module on each worker thread's startup, ahead of -Harper's own modules and yours: +Two configuration keys put a module on each worker thread's startup, ahead of Harper's own modules and yours: ```yaml threads: @@ -174,34 +124,17 @@ threads: preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation ``` -`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is -how an agent installs the loader hooks that let it instrument modules imported -later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the -module body and is typically how an agent's initialization entry actually starts -it. Both are documented under -[threads configuration](/reference/v5/configuration/options#threads). - -Installing loader hooks and starting the agent are two different jobs, and which -of your agent's entry points does which is specific to that agent. Some ship one -entry that does both; others split them, in which case set both keys. The example -above is the split-entry case, and it is the one that produces the most confusing -failure: with only the hooks loaded, a tracer will hand out spans with plausible -trace ids that are no-ops and export nothing, so your instrumentation looks -installed and your collector stays empty. - -Two constraints worth knowing before you plan around this. Both keys apply to -worker threads only, and neither works under Bun. And bare specifiers resolve -against the `node_modules` of your installed components, so an agent can ship as a -dependency of a deployed component rather than as a host-level install. - -Whatever agent you use, follow its own worker-thread documentation, and finish by -confirming spans actually arrive at your collector rather than assuming the -configuration took. +`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is how an agent installs the loader hooks that let it instrument modules imported later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under [threads configuration](/reference/v5/configuration/options#threads). + +Installing loader hooks and starting the agent are two different jobs, and which of your agent's entry points does which is specific to that agent. Some ship one entry that does both; others split them, in which case set both keys. The example above is the split-entry case, and it is the one that produces the most confusing failure: with only the hooks loaded, a tracer will hand out spans with plausible trace ids that are no-ops and export nothing, so your instrumentation looks installed and your collector stays empty. + +Two constraints worth knowing before you plan around this. Both keys apply to worker threads only, and neither works under Bun. And bare specifiers resolve against the `node_modules` of your installed components, so an agent can ship as a dependency of a deployed component rather than as a host-level install. + +Whatever agent you use, follow its own worker-thread documentation, and finish by confirming spans actually arrive at your collector rather than assuming the configuration took. ## Logs you can use during an incident -Harper's `logger` global takes a message plus a context object from component -code: +Harper's `logger` global takes a message plus a context object from component code: ```javascript logger.info('order submitted', { @@ -215,49 +148,27 @@ logger.info('order submitted', { }); ``` -Include the component and its version, the node, a request or trace identifier, -the operation, its duration, the outcome, and a safe error class. That field set -is what lets you answer "was this only the new version" and "was this only one -node" without guessing, and those are the first two questions in almost every -incident. +Include the component and its version, the node, a request or trace identifier, the operation, its duration, the outcome, and a safe error class. That field set is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident. Three details in that example need care: -- **The node name comes from `server.hostname`**, not an environment variable. - Harper does not set a node-name variable in the process environment, and - `server.hostname` is the same identity analytics uses, so it is what correlates - with your metrics. -- **`context.requestId` is only populated when `http.logging.id` is enabled**, and - HTTP request logging is off by default. Without it the field is `undefined` and - you silently lose your correlation id. Either enable it in - [logging configuration](/reference/v5/logging/configuration) or generate an id - in your own code. -- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your - deployment so the log line can name the build. +- **The node name comes from `server.hostname`**, not an environment variable. Harper does not set a node-name variable in the process environment, and `server.hostname` is the same identity analytics uses, so it is what correlates with your metrics. +- **`context.requestId` is only populated when `http.logging.id` is enabled**, and HTTP request logging is off by default. Without it the field is `undefined` and you silently lose your correlation id. Either enable it in [logging configuration](/reference/v5/logging/configuration) or generate an id in your own code. +- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your deployment so the log line can name the build. :::warning -Harper's logger is built on Node's `Console`, so this renders as a formatted text -line, not JSON. The context object is inspected into the message rather than -emitted as separate fields: +Harper's logger is built on Node's `Console`, so this renders as a formatted text line, not JSON. The context object is inspected into the message rather than emitted as separate fields: ```text [main/3] [info]: order submitted { component: 'orders-api', version: '2.4.1', ... } ``` -That is fine for reading during an incident, but a log pipeline cannot key on -`component` or `outcome` without parsing the line. If you need queryable fields, -serialize the context yourself and log a single JSON string. +That is fine for reading during an incident, but a log pipeline cannot key on `component` or `outcome` without parsing the line. If you need queryable fields, serialize the context yourself and log a single JSON string. ::: -Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The -default is `warn`, and `notify` is always logged regardless of level. Choose -levels deliberately: a production log at `debug` is a log nobody can read, and one -at `error` only has the incidents in it and none of the context. +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged regardless of level. Choose levels deliberately: a production log at `debug` is a log nobody can read, and one at `error` only has the incidents in it and none of the context. -`console.log` output does not reach the log files unless `logging.console` is -enabled, so unstructured console output is not a production record. Centralize -logs off the node, because the node you most need logs from is the one you are -about to restart. +`console.log` output does not reach the log files unless `logging.console` is enabled, so unstructured console output is not a production record. Centralize logs off the node, because the node you most need logs from is the one you are about to restart. Read logs through the API when you need them from a specific node: @@ -273,21 +184,13 @@ Read logs through the API when you need them from a specific node: Run these in order. The goal is not diagnosis, it is narrowing. -1. **Confirm and bound the impact.** What journey, starting exactly when, in which - geography or cohort, and which component version is live. Without a start time - you cannot correlate anything. +1. **Confirm and bound the impact.** What journey, starting exactly when, in which geography or cohort, and which component version is live. Without a start time you cannot correlate anything. -2. **Compare the public route against a direct node call.** Request the same - journey through your traffic layer, then directly against each node on `9926`. - If direct calls succeed and the public route fails, you are looking at the - traffic layer or the availability flag, not at Harper. +2. **Compare the public route against a direct node call.** Request the same journey through your traffic layer, then directly against each node on `9926`. If direct calls succeed and the public route fails, you are looking at the traffic layer or the availability flag, not at Harper. -3. **Check admission state on every node.** `GET /status` on `9926` and your - readiness route. A node advertising unavailable is a node deliberately or - accidentally out of rotation, and finding that here saves a lot of time. +3. **Check admission state on every node.** `GET /status` on `9926` and your readiness route. A node advertising unavailable is a node deliberately or accidentally out of rotation, and finding that here saves a lot of time. -4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then - diff: +4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then diff: ```json { @@ -296,73 +199,41 @@ Run these in order. The goal is not diagnosis, it is narrowing. } ``` - Differences between peers are more informative than absolute values, because - they tell you whether this is one node or the whole cluster. + Differences between peers are more informative than absolute values, because they tell you whether this is one node or the whole cluster. -5. **Check the replication path.** `cluster_status` for the databases this journey - needs. Look for a missing peer, `connected: false`, or a widening gap between - `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. +5. **Check the replication path.** `cluster_status` for the databases this journey needs. Look for a missing peer, `connected: false`, or a widening gap between `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. -6. **Correlate with the last change.** `list_deployments` for what shipped and - when, and `get_configuration` read back against what you believe is configured. - Remember that a configuration change applied without a restart leaves a node - running something other than its stated configuration. +6. **Correlate with the last change.** `list_deployments` for what shipped and when, and `get_configuration` read back against what you believe is configured. Remember that a configuration change applied without a restart leaves a node running something other than its stated configuration. -7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, - deactivate a feature, restore prior traffic weights, or isolate suspected data. - Record the decision, who made it, and the next decision deadline. +7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, deactivate a feature, restore prior traffic weights, or isolate suspected data. Record the decision, who made it, and the next decision deadline. -Two rules make this sequence work. Silence is a failed gate: if telemetry is -missing for the thing you are checking, treat that as a negative signal rather -than skipping the step. And containment comes before root cause. You can diagnose -after users are being served again. +Two rules make this sequence work. Silence is a failed gate: if telemetry is missing for the thing you are checking, treat that as a negative signal rather than skipping the step. And containment comes before root cause. You can diagnose after users are being served again. ### Prove it -Pick a fault and inject it on a non-production cluster, then time yourself. Good -candidates: saturate one node's workers, block the replication port on one peer, -make a downstream dependency return errors, or deploy a component that fails on -one route. +Pick a fault and inject it on a non-production cluster, then time yourself. Good candidates: saturate one node's workers, block the replication port on one peer, make a downstream dependency return errors, or deploy a component that fails on one route. -Measure three things. How long until an alert fired. How long until an operator -following the sequence above could name the layer. And whether any step gave a -misleading answer, which is the most valuable output of the drill, because a -misleading signal during a real incident costs more than a missing one. +Measure three things. How long until an alert fired. How long until an operator following the sequence above could name the layer. And whether any step gave a misleading answer, which is the most valuable output of the drill, because a misleading signal during a real incident costs more than a missing one. ## Operational notes -- **Node-level dashboards, not cluster averages.** A cluster average conceals the - single node doing twice the work, which is the most common cause of a latency - complaint that looks like nothing on a dashboard. -- **Watch measured request distribution, not configured weights.** Per-node - request counts are the ground truth. Sticky sessions, DNS caching, and - connection reuse all skew actual distribution away from intent. -- **Annotate deployments onto your graphs.** If your monitoring supports - annotations, feed `list_deployments` into them. This single change resolves more - incidents faster than any additional metric. -- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for - reconstructing what changed in a table. Know before an incident whether you have - it on, because turning it on afterwards does not help. -- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose - configuration and source detail, so they are `super_user` operations for a - reason. Restrict and log their use. +- **Node-level dashboards, not cluster averages.** A cluster average conceals the single node doing twice the work, which is the most common cause of a latency complaint that looks like nothing on a dashboard. +- **Watch measured request distribution, not configured weights.** Per-node request counts are the ground truth. Sticky sessions, DNS caching, and connection reuse all skew actual distribution away from intent. +- **Annotate deployments onto your graphs.** If your monitoring supports annotations, feed `list_deployments` into them. This single change resolves more incidents faster than any additional metric. +- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for reconstructing what changed in a table. Know before an incident whether you have it on, because turning it on afterwards does not help. +- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose configuration and source detail, so they are `super_user` operations for a reason. Restrict and log their use. ## Readiness checklist -- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, - replication, data, or change +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change - [ ] Alerts on journey success and latency at the SLO percentile, not the mean -- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived - from measured per-node capacity +- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived from measured per-node capacity - [ ] Alerts on replication convergence lag and missing peers - [ ] Alerts on storage headroom and backup age -- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus - exporter self-managed -- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` - set as that agent requires, with spans confirmed at the collector +- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed +- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` set as that agent requires, with spans confirmed at the collector - [ ] `analytics.replicate` setting known and deliberate -- [ ] Structured logging includes component, version, node, request id, operation, - duration, and outcome +- [ ] Structured logging includes component, version, node, request id, operation, duration, and outcome - [ ] Logs centralized off the node - [ ] Deployments annotated onto dashboards - [ ] Triage sequence written down where on-call can find it @@ -370,17 +241,11 @@ misleading signal during a real incident costs more than a missing one. ## Additional Resources -- [Analytics overview](/reference/v5/analytics/overview) for the full standard - metric catalog and configuration options -- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` - and `describe_metric` +- [Analytics overview](/reference/v5/analytics/overview) for the full standard metric catalog and configuration options +- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` and `describe_metric` - [Grafana integration](/fabric/grafana-integration) for Fabric dashboards -- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) - for self-managed metric scraping -- [Logging overview](/reference/v5/logging/overview), - [configuration](/reference/v5/logging/configuration), and - [operations](/reference/v5/logging/operations) +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) for self-managed metric scraping +- [Logging overview](/reference/v5/logging/overview), [configuration](/reference/v5/logging/configuration), and [operations](/reference/v5/logging/operations) - [HTTP API reference](/reference/v5/http/api) for `server.recordAnalytics()` - [Fabric logging](/fabric/logging) for log access on Fabric -- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the - gates these signals feed +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the gates these signals feed diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 93baa86c7..c18964755 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -6,45 +6,28 @@ sidebar_position: 4 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Replication is how a Harper cluster stays available: peers exchange data directly -over WebSockets, there is no primary, and a node that falls behind catches up on -its own. The [replication reference](/reference/v5/replication/overview) covers -how to configure that. This guide covers the part that only matters once real -users depend on it, which is how to tell whether replication is actually working. +Replication is how a Harper cluster stays available: peers exchange data directly over WebSockets, there is no primary, and a node that falls behind catches up on its own. The [replication reference](/reference/v5/replication/overview) covers how to configure that. This guide covers the part that only matters once real users depend on it, which is how to tell whether replication is actually working. -The distinction that runs through this guide is between a connection and -convergence. A connected socket proves two nodes can talk. It does not prove the -data a user is about to read is current. Those are different claims, and only one -of them is what your traffic admission decision depends on. +The distinction that runs through this guide is between a connection and convergence. A connected socket proves two nodes can talk. It does not prove the data a user is about to read is current. Those are different claims, and only one of them is what your traffic admission decision depends on. ## What You Will Learn -- How to determine what your cluster actually replicates, rather than assuming a - default -- What moves between peers automatically, what does not, and which of those will - surprise you -- How to read `cluster_status` as an operator, including which timing field - actually indicates a node is behind -- How to prove convergence with a sentinel rather than inferring it from - connection state -- Which application behaviors replication cannot make safe, and what to do about - them instead +- How to determine what your cluster actually replicates, rather than assuming a default +- What moves between peers automatically, what does not, and which of those will surprise you +- How to read `cluster_status` as an operator, including which timing field actually indicates a node is behind +- How to prove convergence with a sentinel rather than inferring it from connection state +- Which application behaviors replication cannot make safe, and what to do about them instead ## Prerequisites -- A cluster of at least three nodes, so you can interrupt one peer and still - observe the others +- A cluster of at least three nodes, so you can interrupt one peer and still observe the others - A `super_user` credential for the Operations API -- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your - service boundary inventory -- Familiarity with how your cluster was joined, either through - `harper-config.yaml` routes or the - [clustering operations](/reference/v5/replication/clustering) +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your service boundary inventory +- Familiarity with how your cluster was joined, either through `harper-config.yaml` routes or the [clustering operations](/reference/v5/replication/clustering) ## Know your own scope -By default Harper replicates all data in all databases. Scope can be narrowed two -ways: per database in configuration, and per table in the schema. +By default Harper replicates all data in all databases. Scope can be narrowed two ways: per database in configuration, and per table in the schema. ```yaml replication: @@ -60,10 +43,7 @@ type LocalTableForNode @table(replicate: false) { } ``` -All tables in a replicated database replicate unless the table opts out. So the -scope you are operating is the product of a config list, a set of schema -directives, and any directional routes someone added later. Do not reconstruct it -from memory. Read it off the running cluster: +All tables in a replicated database replicate unless the table opts out. So the scope you are operating is the product of a config list, a set of schema directives, and any directional routes someone added later. Do not reconstruct it from memory. Read it off the running cluster: @@ -92,23 +72,10 @@ await fetch('https://my-node.example.com:9925/', { -There is one socket per database per peer, so the sockets present in the response -are the ground truth about what is flowing. Run this from every node, not one. -Replication direction can be constrained per route, so node A's view of the -cluster is not necessarily node B's view, and a one-sided picture is how -directional configuration mistakes survive into production. +There is one socket per database per peer, so the sockets present in the response are the ground truth about what is flowing. Run this from every node, not one. Replication direction can be constrained per route, so node A's view of the cluster is not necessarily node B's view, and a one-sided picture is how directional configuration mistakes survive into production. :::warning -Whether the `system` database is in your replication scope is the highest-stakes -scoping question in a Harper cluster, because `system` holds `hdb_user`, -`hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default -replication scope is every database, so unless someone has narrowed it your users -and roles already propagate and every node that receives `system` must be trusted -with its contents, including encrypted secret rows. That is usually what you want, -but it should be a decision rather than a surprise. Confirm your own answer from -`cluster_status` and your configuration, and read -[replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) -before changing it. +Whether the `system` database is in your replication scope is the highest-stakes scoping question in a Harper cluster, because `system` holds `hdb_user`, `hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default replication scope is every database, so unless someone has narrowed it your users and roles already propagate and every node that receives `system` must be trusted with its contents, including encrypted secret rows. That is usually what you want, but it should be a decision rather than a surprise. Confirm your own answer from `cluster_status` and your configuration, and read [replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) before changing it. ::: ## What moves automatically, and what does not @@ -125,43 +92,18 @@ before changing it. | Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -The three rows most likely to bite you are the returning node, the new node, and -the node registry. - -On the returning node: it is common to see a routine restart budgeted as though it -were a full resynchronization, which makes teams avoid restarts they should be -comfortable with. A node whose databases have never synced does download them in -full. A node that was briefly offline catches up on what it missed. Measure both -on your own data volume once, and use the right number for the right situation. - -On the new node, the cost is larger than "one copy from one peer," because the -full-copy decision is made per peer and per database rather than once against a -bootstrap source. A joining node requests a full copy from every peer it has no -resume cursor for. And because replication is bidirectional, each established peer -independently decides it has no cursor for the newcomer and requests a full copy -_from_ it as well. Joining a cluster of N peers is therefore N inbound transfers -plus N outbound transfers of the new node's own (empty) databases, not a single -stream from one source. - -The practical consequences are worth planning around. Load lands on every existing -peer at once rather than on one, so a join during peak traffic is a capacity event -for the whole cluster. If you want a single designated source instead, `add_node` -accepts `isLeader: true`, which tells the joining node to request its full copy -from that peer alone. - -On the registry: a node's advertised record is derived from its configuration -file, and it replicates. That means a topology constraint you applied -imperatively is superseded the next time that node restarts or reloads -components, and the node quietly goes back to advertising itself more broadly -than you intended. +The three rows most likely to bite you are the returning node, the new node, and the node registry. + +On the returning node: it is common to see a routine restart budgeted as though it were a full resynchronization, which makes teams avoid restarts they should be comfortable with. A node whose databases have never synced does download them in full. A node that was briefly offline catches up on what it missed. Measure both on your own data volume once, and use the right number for the right situation. + +On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. + +The practical consequences are worth planning around. Load lands on every existing peer at once rather than on one, so a join during peak traffic is a capacity event for the whole cluster. If you want a single designated source instead, `add_node` accepts `isLeader: true`, which tells the joining node to request its full copy from that peer alone. + +On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads components, and the node quietly goes back to advertising itself more broadly than you intended. :::danger -When replicating configuration, only send cluster-appropriate parameters. -Replicating a node-local value such as a port, `node.hostname`, a file path, TLS -material, or `replication.hostname`, `url`, or `routes` overwrites every peer's -own local value. To apply a cluster-wide change safely, use -`set_configuration` with `"replicated": true` for the parameter, then -`restart_service` with `"replicated": true`, which restarts nodes one at a time. +When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. To apply a cluster-wide change safely, use `set_configuration` with `"replicated": true` for the parameter, then `restart_service` with `"replicated": true`, which restarts nodes one at a time. ::: ## Read `cluster_status` like an operator @@ -194,61 +136,34 @@ A trimmed response, with the fields that matter: What each field is telling you: -- **`connected`** is the liveness of this one database's socket to this one peer. - A missing peer, or a peer present with `connected: false`, is actionable before - users notice. -- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained - growth rather than on a single sample. -- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one - of your commits. If it stops advancing while you are still writing, your writes - are not landing on that peer. -- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest - transaction you have received. +- **`connected`** is the liveness of this one database's socket to this one peer. A missing peer, or a peer present with `connected: false`, is actionable before users notice. +- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained growth rather than on a single sample. +- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer. +- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. - **`lastReceivedLocalTime`** is your own clock when you received it. -The last two are the pair that matters. **A widening gap between -`lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and -working through a backlog.** That is the signal to alert on for convergence, and -it is the one that tells you a returning node is not ready for traffic yet. -`sendingMessage` appears while a transaction is actively being sent and is absent -when the socket is idle, so its absence is not a fault. - -Two things will mislead you if you automate on that pair without knowing them. -The gap only means "behind" while transactions are actually arriving: on an idle -source both stamps freeze, and the gap sits at whatever constant it last reached -rather than signalling lag. And because the two stamps come from two different -clocks, skew between peers is added to the gap directly, so calibrate your -threshold against a healthy baseline rather than treating the raw number as -replication delay. +The last two are the pair that matters. **A widening gap between `lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and working through a backlog.** That is the signal to alert on for convergence, and it is the one that tells you a returning node is not ready for traffic yet. `sendingMessage` appears while a transaction is actively being sent and is absent when the socket is idle, so its absence is not a fault. + +Two things will mislead you if you automate on that pair without knowing them. The gap only means "behind" while transactions are actually arriving: on an idle source both stamps freeze, and the gap sits at whatever constant it last reached rather than signalling lag. And because the two stamps come from two different clocks, skew between peers is added to the gap directly, so calibrate your threshold against a healthy baseline rather than treating the raw number as replication delay. :::tip -While a database is taking a full copy, these timing fields render as the literal -string `"Copying"` rather than a date. Anything that parses them as timestamps -will fail on a joining node, which is exactly the node you most want to be -watching. Handle that value explicitly. +While a database is taking a full copy, these timing fields render as the literal string `"Copying"` rather than a date. Anything that parses them as timestamps will fail on a joining node, which is exactly the node you most want to be watching. Handle that value explicitly. ::: -Inventory `system.hdb_nodes` alongside this and compare it to your intended -topology. Configuration intent and live peer state should agree, and the node's -own row is in there too, not just its peers. +Inventory `system.hdb_nodes` alongside this and compare it to your intended topology. Configuration intent and live peer state should agree, and the node's own row is in there too, not just its peers. ### What to alert on -- A peer missing entirely from `connections`, or `connected: false` on a database - that serves a critical journey +- A peer missing entirely from `connections`, or `connected: false` on a database that serves a critical journey - Sustained growth in `latency`, judged against your own baseline - `lastCommitConfirmed` not advancing on a peer while writes are occurring -- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your - admission budget, evaluated only while writes are flowing -- Repeated reconnects, which are visible in the logs even when a point-in-time - status check looks healthy +- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your admission budget, evaluated only while writes are flowing +- Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy - Version or configuration drift between peers ## Prove convergence, not connection -Socket state cannot tell you that a specific business record is current. For -anything where the answer matters, write a sentinel and read it back from the -peer. +Socket state cannot tell you that a specific business record is current. For anything where the answer matters, write a sentinel and read it back from the peer. ```javascript // On the source node: write a sentinel with a known value @@ -262,120 +177,63 @@ curl -s https://server-2.example.net:9926/OpsProbe/convergence-probe \ -u 'admin:password' ``` -The interval between the write and the moment the peer returns the new value is -your measured convergence time for that database, under whatever load the cluster -is carrying at the time. Run it under load, not on an idle cluster, and record the -result. That number is what your node admission gate should be compared against, -and it is an input to the RPO work in -[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). - -The `replication-latency` metric from -[analytics](/reference/v5/analytics/overview#replication-metrics) gives you the -continuous version of the same measurement: the difference between the source -commit timestamp and local time, reported per node, database, and table. Use the -sentinel for a definitive answer during a change, and the metric for a dashboard. - -Note that `replication-latency` is not recorded for the `system` database. Since -`system` is in replication scope by default and carries your users and roles, a -dashboard built only on this metric will show nothing for the database that -propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, -if you need to watch `system` convergence. +The interval between the write and the moment the peer returns the new value is your measured convergence time for that database, under whatever load the cluster is carrying at the time. Run it under load, not on an idle cluster, and record the result. That number is what your node admission gate should be compared against, and it is an input to the RPO work in [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +The `replication-latency` metric from [analytics](/reference/v5/analytics/overview#replication-metrics) gives you the continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard. + +Note that `replication-latency` is not recorded for the `system` database. Since `system` is in replication scope by default and carries your users and roles, a dashboard built only on this metric will show nothing for the database that propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, if you need to watch `system` convergence. ## Consistency belongs in the application design -Replication makes data available on every peer. It does not make every peer agree -at every instant, and a distributed decision made on a node that has not yet -converged can observe stale state and act on it. +Replication makes data available on every peer. It does not make every peer agree at every instant, and a distributed decision made on a node that has not yet converged can observe stale state and act on it. -This is not a Harper limitation to work around, it is the property that lets any -node serve any request without a coordinator. But it means a specific class of -operation is unsafe if you write it as a plain read followed by a write: +This is not a Harper limitation to work around, it is the property that lets any node serve any request without a coordinator. But it means a specific class of operation is unsafe if you write it as a plain read followed by a write: -- Claim-once actions: redeeming a code, assigning a unique handle, awarding a - one-per-customer offer -- Hard floors: inventory that must not go negative, a balance that must not - overdraw +- Claim-once actions: redeeming a code, assigning a unique handle, awarding a one-per-customer offer +- Hard floors: inventory that must not go negative, a balance that must not overdraw - Global limits: a rate limit or quota enforced across the whole cluster -- State machine transitions where two nodes could both believe they are making - the same transition +- State machine transitions where two nodes could both believe they are making the same transition -For each of these, choose one of three designs: route all decisions for a given -key to a single owner, use a serialization mechanism so the conflict is resolved -in one place, or delegate to an external coordinator. Then test your -read-after-write expectations through the actual public route rather than against -a single node, because a single node always looks consistent to itself. +For each of these, choose one of three designs: route all decisions for a given key to a single owner, use a serialization mechanism so the conflict is resolved in one place, or delegate to an external coordinator. Then test your read-after-write expectations through the actual public route rather than against a single node, because a single node always looks consistent to itself. ### Prove it On a non-production cluster under write load: 1. Record baseline `cluster_status` timing fields on every node. -2. Interrupt replication to one peer, by stopping the node or blocking the - replication port, while writes continue elsewhere. -3. Watch what your monitoring reports, and how long it takes to say anything. Note - whether `connected` flipped, whether latency alerted, and how long until a - human would have known. -4. Verify your application's actual behavior on the isolated node. Does the - critical journey fail, serve stale data, or serve correctly? All three are - possible and you should know which. -5. Restore the peer, then measure convergence with the sentinel until it is - current. -6. Compare the convergence time against the admission gate in - [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). - If your gate would have admitted the node before step 5 finished, the gate is - wrong. +2. Interrupt replication to one peer, by stopping the node or blocking the replication port, while writes continue elsewhere. +3. Watch what your monitoring reports, and how long it takes to say anything. Note whether `connected` flipped, whether latency alerted, and how long until a human would have known. +4. Verify your application's actual behavior on the isolated node. Does the critical journey fail, serve stale data, or serve correctly? All three are possible and you should know which. +5. Restore the peer, then measure convergence with the sentinel until it is current. +6. Compare the convergence time against the admission gate in [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). If your gate would have admitted the node before step 5 finished, the gate is wrong. ## Operational notes -- **mTLS is required on the secure replication port and cannot be disabled.** - Certificate expiry is therefore a replication outage, so treat certificate - lifetime as an operational deadline with its own alert. See - [certificate management](/reference/v5/security/certificate-management). -- **Gossip discovery means one route can join a whole cluster.** A node that - connects to one peer discovers the rest. That is convenient and it means an - accidental route can widen your topology more than you intended. -- **Sharding is a separate scope control.** If you use - [sharding](/reference/v5/replication/sharding), not every node holds every - record, so "this node is converged" and "this node can answer this query - locally" become different questions. -- **Analytics data has its own replication setting.** See `analytics.replicate` in - the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) - before you build dashboards that assume metrics are or are not cluster-wide. -- **Keep node-local operational state out of scope.** Availability flags, - maintenance markers, and anything else that describes one node rather than the - cluster should not replicate, or a routine drain propagates to peers. +- **mTLS is required on the secure replication port and cannot be disabled.** Certificate expiry is therefore a replication outage, so treat certificate lifetime as an operational deadline with its own alert. See [certificate management](/reference/v5/security/certificate-management). +- **Gossip discovery means one route can join a whole cluster.** A node that connects to one peer discovers the rest. That is convenient and it means an accidental route can widen your topology more than you intended. +- **Sharding is a separate scope control.** If you use [sharding](/reference/v5/replication/sharding), not every node holds every record, so "this node is converged" and "this node can answer this query locally" become different questions. +- **Analytics data has its own replication setting.** See `analytics.replicate` in the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) before you build dashboards that assume metrics are or are not cluster-wide. +- **Keep node-local operational state out of scope.** Availability flags, maintenance markers, and anything else that describes one node rather than the cluster should not replicate, or a routine drain propagates to peers. ## Readiness checklist -- [ ] Replication scope read off the running cluster, from every node, not - reconstructed from memory -- [ ] Whether the `system` database is in scope is a documented, deliberate - decision +- [ ] Replication scope read off the running cluster, from every node, not reconstructed from memory +- [ ] Whether the `system` database is in scope is a documented, deliberate decision - [ ] Identity provisioning procedure exists and matches that decision - [ ] Tables requiring one transaction boundary confirmed to be in one database -- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in - `add_node` calls +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls - [ ] Measured convergence time recorded under load, per critical database -- [ ] Alerts configured on missing peers, sustained latency growth, stalled - `lastCommitConfirmed`, and the remote-to-local receive gap -- [ ] Non-commutative operations identified and given an owner, a serialization - point, or an external coordinator +- [ ] Alerts configured on missing peers, sustained latency growth, stalled `lastCommitConfirmed`, and the remote-to-local receive gap +- [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator - [ ] Replication certificate expiry dates tracked with an alert - [ ] Replication interruption and recovery exercise completed and dated ## Additional Resources -- [Replication overview](/reference/v5/replication/overview) for routes, scope, - controlled flow, and securing connections -- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, - `set_node`, and the `cluster_status` response in full -- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes - hold a given record -- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for - `replication-latency` and byte counters -- [Certificate management](/reference/v5/security/certificate-management) and - [certificate verification](/reference/v5/security/certificate-verification) -- [Database schema](/reference/v5/database/schema) for the `replicate` table - directive -- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals - on a dashboard +- [Replication overview](/reference/v5/replication/overview) for routes, scope, controlled flow, and securing connections +- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, `set_node`, and the `cluster_status` response in full +- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes hold a given record +- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for `replication-latency` and byte counters +- [Certificate management](/reference/v5/security/certificate-management) and [certificate verification](/reference/v5/security/certificate-verification) +- [Database schema](/reference/v5/database/schema) for the `replicate` table directive +- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals on a dashboard diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 45d1165ba..882c39202 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -3,33 +3,21 @@ title: Production Readiness Checklist sidebar_position: 9 --- -This is the aggregated launch gate for the whole Administration track. Every item -appears in one of the earlier guides, which is where the reasoning lives. This -page exists so you have one thing to work down before a launch, and one thing to -hand to a reviewer who asks how you operate Harper. +This is the aggregated launch gate for the whole Administration track. Every item appears in one of the earlier guides, which is where the reasoning lives. This page exists so you have one thing to work down before a launch, and one thing to hand to a reviewer who asks how you operate Harper. -Copy it into your own runbook and adapt it. It is a starting point for your gate, -not a substitute for having one. +Copy it into your own runbook and adapt it. It is a starting point for your gate, not a substitute for having one. ## How to use this -Each item is a claim about your deployment. A claim counts when there is evidence: -a measured number with a date, a written procedure, a configured alert, or a -completed drill. "We know about that" is not evidence, and neither is a passing -intention. +Each item is a claim about your deployment. A claim counts when there is evidence: a measured number with a date, a written procedure, a configured alert, or a completed drill. "We know about that" is not evidence, and neither is a passing intention. -Some items will not apply to you, and that is fine. Mark them not applicable with -a reason rather than deleting them, so a reviewer can see the decision was made -rather than missed. +Some items will not apply to you, and that is fine. Mark them not applicable with a reason rather than deleting them, so a reviewer can see the decision was made rather than missed. Three things are worth deciding before you start: - **Who signs off.** A gate with no named approver is a document, not a gate. -- **What "outstanding" means.** Some items can be accepted as gaps with an owner - and a date. Decide in advance which ones cannot. -- **When you run it again.** This is not a one-time launch artifact. Re-run it - after a Harper version upgrade, a topology change, or significant data growth, - since most of the measured numbers expire. +- **What "outstanding" means.** Some items can be accepted as gaps with an owner and a date. Decide in advance which ones cannot. +- **When you run it again.** This is not a one-time launch artifact. Re-run it after a Harper version upgrade, a topology change, or significant data growth, since most of the measured numbers expire. ## Service definition @@ -49,37 +37,28 @@ From [How Harper Runs in Production](./how-harper-runs-in-production.mdx). From [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). -- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to - change operators -- [ ] Per-node throughput measured with your own application code, data shape, and - query mix +- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to change operators +- [ ] Per-node throughput measured with your own application code, data shape, and query mix - [ ] Measurement taken with a peer synchronizing, not on an idle cluster - [ ] Target utilization derived from latency at your SLO percentile - [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers - [ ] Measurement conditions recorded alongside the number, and dated -- [ ] Topology's "what must be proven" satisfied, including independence of failure - domains +- [ ] Topology's "what must be proven" satisfied, including independence of failure domains - [ ] Maintenance policy states what happens when a drain would breach `F` - [ ] Storage growth thresholds set, separately from request-rate thresholds ## Health and traffic admission -From -[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +From [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). -- [ ] Liveness, availability flag, application readiness, and journey synthetic all - exist as separate signals -- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` - rather than by hand +- [ ] Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` rather than by hand - [ ] Traffic layer health check points at `GET /status` on the application port -- [ ] Availability flag storage still declares `replicate: false`, if you forked the - component or persist it yourself -- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every - downstream call +- [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself +- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every downstream call - [ ] Readiness response includes the component version - [ ] Drain sequence documented, with measured time to zero traffic -- [ ] Return sequence documented, including convergence verification and a - stability window before full weight +- [ ] Return sequence documented, including convergence verification and a stability window before full weight - [ ] Measured node re-entry time recorded ## Replication @@ -87,17 +66,13 @@ From From [Operating Replication](./operating-replication.mdx). - [ ] Replication scope read off the running cluster, from every node -- [ ] Whether the `system` database is in scope is a deliberate, documented - decision +- [ ] Whether the `system` database is in scope is a deliberate, documented decision - [ ] Identity provisioning procedure matches that decision - [ ] Tables that must commit together confirmed to be in one database -- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in - `add_node` calls +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls - [ ] Measured convergence time recorded under load, per critical database -- [ ] Alerts on missing peers, sustained latency growth, stalled - `lastCommitConfirmed`, and the receive-time gap -- [ ] Non-commutative operations identified and given an owner, a serialization - point, or an external coordinator +- [ ] Alerts on missing peers, sustained latency growth, stalled `lastCommitConfirmed`, and the receive-time gap +- [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator - [ ] Replication certificate expiry tracked with an alert - [ ] Replication interruption and recovery exercise completed and dated @@ -116,8 +91,7 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change - [ ] Node-local parameters never replicated -- [ ] Schema changes follow expand then contract, with destructive operations - documented and verified per node +- [ ] Schema changes follow expand then contract, with destructive operations documented and verified per node - [ ] Forward repair defined for changes that alter persisted meaning - [ ] Measured release reversal time recorded from a rehearsal @@ -125,17 +99,14 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). From [Monitoring and Triage](./monitoring-and-triage.mdx). -- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, - replication, data, or change +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change - [ ] Alerts on journey success and latency at the SLO percentile, not the mean -- [ ] Alerts on worker utilization and task queue latency, with thresholds derived - from measured capacity +- [ ] Alerts on worker utilization and task queue latency, with thresholds derived from measured capacity - [ ] Alerts on replication convergence lag - [ ] Alerts on storage headroom and backup age - [ ] Metrics exported off the node - [ ] `analytics.replicate` setting known and deliberate -- [ ] Structured logs include component, version, node, request id, operation, - duration, and outcome +- [ ] Structured logs include component, version, node, request id, operation, duration, and outcome - [ ] Logs centralized off the node - [ ] Deployments annotated onto dashboards - [ ] Triage sequence written where on-call can find it @@ -148,31 +119,24 @@ From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Failure classes enumerated, each with the mechanism that addresses it - [ ] Recovery point and recovery time stated per database - [ ] Mechanism chosen per database according to its storage engine -- [ ] No database in scope uses per-table storage paths, or the exclusion is known - and accepted +- [ ] No database in scope uses per-table storage paths, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point -- [ ] Backup volume sized against the full cost, including the non-incremental - transaction-log and blob snapshots +- [ ] Backup volume sized against the full cost, including the non-incremental transaction-log and blob snapshots - [ ] Off-host copy exists in a destination that does not share a failure domain -- [ ] Managed repository copies take the whole per-database directory, quiesced or - from an atomic snapshot +- [ ] Managed repository copies take the whole per-database directory, quiesced or from an atomic snapshot - [ ] `verify_backup` runs on a schedule - [ ] Blob integrity understood to be unverified by `verify_backup` -- [ ] Restore constraints documented for user databases, component-held databases, - and `system` +- [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged -- [ ] Restore procedure isolates the node from replication, not just from user - traffic -- [ ] Restore drill completed and dated, with measured recovery time and actual - data loss +- [ ] Restore procedure isolates the node from replication, not just from user traffic +- [ ] Restore drill completed and dated, with measured recovery time and actual data loss - [ ] `system` database restore rehearsed offline ## Objectives From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). -- [ ] Recovery point, recovery time, and availability stated per journey with the - measurement boundary named +- [ ] Recovery point, recovery time, and availability stated per journey with the measurement boundary named - [ ] Exclusions documented - [ ] Every scenario in the scenario map has a named mechanism - [ ] Availability target converted to minutes per month @@ -188,18 +152,15 @@ From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). - [ ] Least privilege enforced for operators, applications, and automation - [ ] `super_user` credentials inventoried, with a rotation procedure - [ ] Operations API restricted to administrative networks -- [ ] TLS material inventoried, with expiry alerts for both application and - replication certificates +- [ ] TLS material inventoried, with expiry alerts for both application and replication certificates - [ ] Deploy credentials held in a secret store, not in pipeline configuration - [ ] Administrative access logged, including CLI access on the hosts -- [ ] Filesystem access on nodes understood to permit offline restore without an - API credential +- [ ] Filesystem access on nodes understood to permit offline restore without an API credential ## Ownership and runbooks - [ ] On-call rotation and escalation path defined -- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster - expansion +- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster expansion - [ ] Every runbook has been executed by someone other than its author - [ ] Change record location known, and used - [ ] Incident and postmortem process defined @@ -207,8 +168,7 @@ From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). ## Exercises completed -Record the date of the most recent run of each. An undated exercise is an -undocumented one. +Record the date of the most recent run of each. An undated exercise is an undocumented one. | Exercise | Guide | Last run | Result | | ----------------------------------------- | ------------------------------------------------------------------------ | -------- | ------ | @@ -224,10 +184,7 @@ undocumented one. ## Additional Resources -- [Reliability Plan Template](./reliability-plan-template.mdx) for the document - that holds the answers this checklist asks for +- [Reliability Plan Template](./reliability-plan-template.mdx) for the document that holds the answers this checklist asks for - Every guide in this section, linked per group above -- [Security overview](/reference/v5/security/overview) and - [certificate management](/reference/v5/security/certificate-management) -- [Configuration options](/reference/v5/configuration/options) for the settings - referenced throughout +- [Security overview](/reference/v5/security/overview) and [certificate management](/reference/v5/security/certificate-management) +- [Configuration options](/reference/v5/configuration/options) for the settings referenced throughout diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx index f580ef251..a563f7723 100644 --- a/learn/administration/reliability-plan-template.mdx +++ b/learn/administration/reliability-plan-template.mdx @@ -3,16 +3,11 @@ title: Reliability Plan Template sidebar_position: 10 --- -The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks -the questions. This is the document that holds the answers. +The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks the questions. This is the document that holds the answers. -Copy it, keep one per service, and keep it where your on-call can reach it during -an incident rather than in a wiki nobody remembers. It is deliberately short. -A reliability plan that takes a day to read is a reliability plan nobody consults -at 3am. +Copy it, keep one per service, and keep it where your on-call can reach it during an incident rather than in a wiki nobody remembers. It is deliberately short. A reliability plan that takes a day to read is a reliability plan nobody consults at 3am. -Example values are included in italics to show the intended level of specificity. -Replace them. +Example values are included in italics to show the intended level of specificity. Replace them. --- @@ -32,8 +27,7 @@ Replace them. ## 2. Critical journeys -The user-facing paths that must work. Everything else in this plan is stated per -journey, so this table defines the scope of all of it. +The user-facing paths that must work. Everything else in this plan is stated per journey, so this table defines the scope of all of it. | Journey | Description | Databases and tables involved | Downstream dependencies | Owner | | ---------------- | ----------------------------- | ----------------------------- | ----------------------- | ----- | @@ -48,8 +42,7 @@ journey, so this table defines the scope of all of it. Monthly unavailability budget: _21 min 55 sec at 99.95%_ -Error budget policy: _what changes when the remaining budget falls below a -threshold, and who decides_ +Error budget policy: _what changes when the remaining budget falls below a threshold, and who decides_ ## 4. Architecture and topology @@ -69,8 +62,7 @@ threshold, and who decides_ | Target utilization | | _derived from latency at the SLO percentile_ | | | Surviving capacity at `F` | | _measured in the drain drill, not calculated_ | | -Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept -with documented duration_ +Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept with documented duration_ ## 6. Traffic admission @@ -158,8 +150,7 @@ with documented duration_ ## 14. Open risks -Accepted gaps, each with an owner and a date. This section existing and being -honest is worth more than it being empty. +Accepted gaps, each with an owner and a date. This section existing and being honest is worth more than it being empty. | Risk | Impact if realized | Why accepted | Owner | Review date | | ---- | ------------------ | ------------ | ----- | ----------- | @@ -175,25 +166,19 @@ honest is worth more than it being empty. ## Keeping this current -Most of the numbers in this plan are properties of a Harper version, a data -volume, a topology, and a set of components. All four change, so the numbers -expire. Re-measure and revise after: +Most of the numbers in this plan are properties of a Harper version, a data volume, a topology, and a set of components. All four change, so the numbers expire. Re-measure and revise after: - A Harper version upgrade - A topology change, including adding or removing nodes - Significant data growth - Any incident that produced a surprise -Date every number. An undated measurement in a reliability plan is worse than a -blank field, because a blank field prompts someone to go and measure. +Date every number. An undated measurement in a reliability plan is worse than a blank field, because a blank field prompts someone to go and measure. ## Additional Resources -- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the - gate that populates this plan -- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how - to derive sections 3 and 11 +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the gate that populates this plan +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how to derive sections 3 and 11 - [Backup and Recovery](./backup-and-recovery.mdx) for section 10 -- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for - section 8 +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for section 8 - [Monitoring and Triage](./monitoring-and-triage.mdx) for section 9 diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index baf18bffc..9fdee4bb9 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -6,39 +6,24 @@ sidebar_position: 6 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an -immutable artifact onto your cluster from a pipeline. This guide is about what -happens around that call: how to limit who sees a change, what evidence justifies -expanding it, and how to reverse it when the evidence says stop. - -Harper makes reversal unusually cheap, because your application is a component -deployed from an immutable reference, so rolling back is deploying the previous -reference. There is no image to rebuild and no instance to replace. That only -helps if the previous reference is still addressable and someone has done it -before under calm conditions, which is what makes rollback a designed capability -rather than a hope. +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an immutable artifact onto your cluster from a pipeline. This guide is about what happens around that call: how to limit who sees a change, what evidence justifies expanding it, and how to reverse it when the evidence says stop. + +Harper makes reversal unusually cheap, because your application is a component deployed from an immutable reference, so rolling back is deploying the previous reference. There is no image to rebuild and no instance to replace. That only helps if the previous reference is still addressable and someone has done it before under calm conditions, which is what makes rollback a designed capability rather than a hope. ## What You Will Learn -- How to separate build, deploy, activate, and expose into four decisions with - four different control points +- How to separate build, deploy, activate, and expose into four decisions with four different control points - The change loop, and what counts as evidence at each step - Five rollout patterns and the Harper operations behind each -- What `"restart": true` and `"restart": "rolling"` actually do, which is not what - most people assume -- How to classify a change so you reverse the right thing, and why reversing the - wrong thing can cause data loss +- What `"restart": true` and `"restart": "rolling"` actually do, which is not what most people assume +- How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss ## Prerequisites -- A cluster of at least three nodes, so you can hold one out of rotation and still - meet capacity -- A working pipeline that deploys from an immutable reference - ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), - since every pattern here depends on being able to move traffic -- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here - depends on being able to compare versions +- A cluster of at least three nodes, so you can hold one out of rotation and still meet capacity +- A working pipeline that deploys from an immutable reference ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), since every pattern here depends on being able to move traffic +- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here depends on being able to compare versions ## Separate four decisions @@ -51,63 +36,31 @@ Most bad deployments come from collapsing these into one action. | **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | | **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | -Once these are separate you get options that do not exist otherwise. You can -install an artifact on every node and activate it nowhere. You can activate a -route and expose it only to internal traffic. And when something is wrong you can -reverse exposure in seconds without touching what is installed, which is the -fastest containment action available to you. +Once these are separate you get options that do not exist otherwise. You can install an artifact on every node and activate it nowhere. You can activate a route and expose it only to internal traffic. And when something is wrong you can reverse exposure in seconds without touching what is installed, which is the fastest containment action available to you. :::danger -**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no -`replicated` field replicates to every peer. So does `add_component`, -`drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and -the destructive `drop_database` and `drop_table`. Only `"replicated": false` -changes anything; `"replicated": true` is the default spelled out. - -This is the opposite of what most operators assume, and it is why the single-node -validation pattern below has to say `false` explicitly. If you intend to touch one -node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) -is the deliberate exception: it replicates only when you ask.) +**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no `replicated` field replicates to every peer. So does `add_component`, `drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and the destructive `drop_database` and `drop_table`. Only `"replicated": false` changes anything; `"replicated": true` is the default spelled out. + +This is the opposite of what most operators assume, and it is why the single-node validation pattern below has to say `false` explicitly. If you intend to touch one node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) is the deliberate exception: it replicates only when you ask.) ::: -`urlPath` mounts a component at an HTTP path. -`host` (Added in: v5.2.0) serves it on a virtual hostname. Both are -persisted on the component's root config entry, so they are part of the deployed -state rather than a runtime toggle. See -[HTTP middleware routing](/reference/v5/http/overview#middleware-routing). +`urlPath` mounts a component at an HTTP path. `host` (Added in: v5.2.0) serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). ## The change loop -1. **Preflight.** Confirm the target version, the component inventory from - `get_components`, `cluster_status` convergence, peer capacity with one node - held out, backup posture if data is at risk, and that the previous known-good - artifact is still addressable. +1. **Preflight.** Confirm the target version, the component inventory from `get_components`, `cluster_status` convergence, peer capacity with one node held out, backup posture if data is at risk, and that the previous known-good artifact is still addressable. -2. **Limit.** Choose the smallest cohort that produces useful evidence. One - drained node, an internal cohort, a low-risk geography, a tenant set, or a - small weighted slice. Smaller is better right up until the sample is too small - to distinguish signal from noise. +2. **Limit.** Choose the smallest cohort that produces useful evidence. One drained node, an internal cohort, a low-risk geography, a tenant set, or a small weighted slice. Smaller is better right up until the sample is too small to distinguish signal from noise. -3. **Observe.** Compare the new and old versions on the same metrics: request - success, latency at your SLO percentile, worker saturation, logs and traces, - data correctness, replication behavior, and downstream errors. Comparison - against the other cohort is the point. Absolute numbers on the new version tell - you much less. +3. **Observe.** Compare the new and old versions on the same metrics: request success, latency at your SLO percentile, worker saturation, logs and traces, data correctness, replication behavior, and downstream errors. Comparison against the other cohort is the point. Absolute numbers on the new version tell you much less. -4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared - before you started, with a named owner. Missing telemetry is a failed gate, not - a pass. +4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared before you started, with a named owner. Missing telemetry is a failed gate, not a pass. -5. **Expand.** Increase exposure only after minimum sample and hold conditions - pass, and keep enough healthy capacity in the old version to reverse. +5. **Expand.** Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity in the old version to reverse. -6. **Close.** Verify uniform artifact and configuration across nodes, restore - intended traffic, record the actual outcome, and keep the evidence with the - change record. +6. **Close.** Verify uniform artifact and configuration across nodes, restore intended traffic, record the actual outcome, and keep the evidence with the change record. -Declaring thresholds in step 4 before step 2 is the part that gets skipped and the -part that matters. A threshold invented while looking at a live graph is not a -threshold, it is a negotiation, and it always resolves toward shipping. +Declaring thresholds in step 4 before step 2 is the part that gets skipped and the part that matters. A threshold invented while looking at a live graph is not a threshold, it is a negotiation, and it always resolves toward shipping. ## Five rollout patterns @@ -134,8 +87,7 @@ Isolated node validation, concretely: } ``` -Drain the node first, deploy, run your readiness route and journey synthetic -against it directly, then admit a bounded cohort. +Drain the node first, deploy, run your readiness route and journey synthetic against it directly, then admit a bounded cohort. @@ -150,103 +102,49 @@ against it directly, then admit a bounded cohort. } ``` -Same immutable reference, now cluster-wide. The deploy itself is finished when the -call returns, so the thing still outstanding is the restart: poll the returned -`restartJobId` with -[`get_job`](/reference/v5/operations-api/operations#get_job). Read -[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the -per-peer detail the response summarizes away, including which peers received the -artifact and which failed. +Same immutable reference, now cluster-wide. The deploy itself is finished when the call returns, so the thing still outstanding is the restart: poll the returned `restartJobId` with [`get_job`](/reference/v5/operations-api/operations#get_job). Read [`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the per-peer detail the response summarizes away, including which peers received the artifact and which failed. -Prefer a pinned version or an immutable tarball over a moving branch reference. -A branch moves, so you can neither audit what was running yesterday nor redeploy -it. +Prefer a pinned version or an immutable tarball over a moving branch reference. A branch moves, so you can neither audit what was running yesterday nor redeploy it. :::tip -A pinned registry version such as `@my-org/orders-api@2.4.1` is the -best-travelled form. If you pin a tarball URL instead, note that a bare `https://` -URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git -clone rather than a download, so a GitHub release asset URL will not behave the -way the example above does. Host release tarballs somewhere neutral, or use a -registry version. +A pinned registry version such as `@my-org/orders-api@2.4.1` is the best-travelled form. If you pin a tarball URL instead, note that a bare `https://` URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git clone rather than a download, so a GitHub release asset URL will not behave the way the example above does. Host release tarballs somewhere neutral, or use a registry version. ::: ## What restart actually does This is worth reading carefully, because the naming invites a wrong assumption. -**`"restart": true`** starts a restart of the HTTP worker threads on the node -handling the call and returns immediately, without waiting for it. A `200` means -the deploy succeeded and a restart has been requested. It does not mean the new -code is serving yet. Until a worker has been replaced it is still running the -previous code, and on platforms where replacements share a listening port it keeps -accepting connections during the changeover. +**`"restart": true`** starts a restart of the HTTP worker threads on the node handling the call and returns immediately, without waiting for it. A `200` means the deploy succeeded and a restart has been requested. It does not mean the new code is serving yet. Until a worker has been replaced it is still running the previous code, and on platforms where replacements share a listening port it keeps accepting connections during the changeover. -**`"restart": "rolling"`** does not restart inline either, but it is observable. -It starts a replicated `restart_service` job and returns a `restartJobId` you can -poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job -walks the cluster **one node at a time**, waiting for each node to come back -before starting the next, so the cluster keeps serving throughout. +**`"restart": "rolling"`** does not restart inline either, but it is observable. It starts a replicated `restart_service` job and returns a `restartJobId` you can poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job walks the cluster **one node at a time**, waiting for each node to come back before starting the next, so the cluster keeps serving throughout. :::warning -**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to -every peer, and peers are dispatched in parallel, so a replicated deploy with -`"restart": true` restarts every node in the cluster at roughly the same moment. -That is a cluster-wide availability event, not a node-local one. Reserve -`"restart": true` for a single-node or development instance, and use `"rolling"` -anywhere you care about staying up. See -[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). +**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to every peer, and peers are dispatched in parallel, so a replicated deploy with `"restart": true` restarts every node in the cluster at roughly the same moment. That is a cluster-wide availability event, not a node-local one. Reserve `"restart": true` for a single-node or development instance, and use `"rolling"` anywhere you care about staying up. See [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). ::: -Two more consequences for your pipeline. Neither restart mode reports completion -in the deploy response, so if you need to know the new code is live, poll the -rolling restart job or probe the nodes themselves. And a failed restart does not -fail the deploy: the component is installed and replicated either way, so your -pipeline needs to check both outcomes separately rather than assuming one implies -the other. +Two more consequences for your pipeline. Neither restart mode reports completion in the deploy response, so if you need to know the new code is live, poll the rolling restart job or probe the nodes themselves. And a failed restart does not fail the deploy: the component is installed and replicated either way, so your pipeline needs to check both outcomes separately rather than assuming one implies the other. ### Waiting for the restart -From v5.3.0, `"restart": true` waits for the worker restart to finish before -responding, rather than returning as soon as it has been requested. The wait -follows the restart's own progress rather than a fixed timeout, so the call can -take tens of seconds on a slow install with many worker threads, with a hard -ceiling of ten minutes. +From v5.3.0, `"restart": true` waits for the worker restart to finish before responding, rather than returning as soon as it has been requested. The wait follows the restart's own progress rather than a fixed timeout, so the call can take tens of seconds on a slow install with many worker threads, with a hard ceiling of ten minutes. -Two things do not change, and both matter more than the wait itself. A caller that -gives up early does not stop the restart, it only loses the result. And the -response still does not carry the restart's outcome: a restart that stalls, times -out, or leaves workers on the old code is reported in the node's log, not to you. -So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and -confirm the version actually serving through your readiness route or journey -synthetic rather than through the deploy response. +Two things do not change, and both matter more than the wait itself. A caller that gives up early does not stop the restart, it only loses the result. And the response still does not carry the restart's outcome: a restart that stalls, times out, or leaves workers on the old code is reported in the node's log, not to you. So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and confirm the version actually serving through your readiness route or journey synthetic rather than through the deploy response. Two parameters worth setting deliberately on replicated deploys: -- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits - for the replicated payload before failing, defaulting to 120000 ms. Raise it for - large components or slow links. -- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that - fails to receive the deploy as non-fatal. By default a failed peer makes the - whole operation return a non-2xx status, while the component is still deployed - on the origin node. Decide which behavior you want before you need it, because - the default leaves you in a mixed-version state with a failed response, and that - is a confusing thing to reason about mid-incident. +- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. +- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it, because the default leaves you in a mixed-version state with a failed response, and that is a confusing thing to reason about mid-incident. ## Configuration changes are deployments too -A configuration change carries the same risk as a code change and gets less -ceremony, which is backwards. +A configuration change carries the same risk as a code change and gets less ceremony, which is backwards. -`set_configuration` supports `"replicated": true` (Added in: v5.2.0) -to apply a change across the cluster in one call, with per-node outcomes in the -response. To finish the change cluster-wide, follow with `restart_service` using -`"replicated": true`, which restarts nodes one at a time. +`set_configuration` supports `"replicated": true` (Added in: v5.2.0) to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. ```json { @@ -257,25 +155,14 @@ response. To finish the change cluster-wide, follow with `restart_service` using ``` :::danger -Only replicate cluster-appropriate parameters. Node-local values such as ports, -`node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or -`routes` would overwrite every peer's own values. Replicating one of these is a -cluster-wide outage delivered in a single API call. +Only replicate cluster-appropriate parameters. Node-local values such as ports, `node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or `routes` would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call. ::: -Two more things to hold onto. A change takes effect only after a restart, so a -node that has been reconfigured and not restarted is running the old -configuration while reporting the new one. And `get_status` reports a -`restartRequired` flag, but it tracks component and code restarts rather than -configuration changes, so it will not tell you a configuration change is still -pending. Track pending configuration in your change record instead, and read back -`get_configuration` after the restart to confirm. +Two more things to hold onto. A change takes effect only after a restart, so a node that has been reconfigured and not restarted is running the old configuration while reporting the new one. And `get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes, so it will not tell you a configuration change is still pending. Track pending configuration in your change record instead, and read back `get_configuration` after the restart to confirm. ## Rollback is a designed capability -Classify the change before choosing a reversal path, because these have different -compatibility requirements, different authorities, and very different blast -radii: +Classify the change before choosing a reversal path, because these have different compatibility requirements, different authorities, and very different blast radii: | Change type | Reversal | Watch out for | | ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------- | @@ -288,97 +175,59 @@ radii: Practices that make each of these real: -- **Keep the previous package addressable and rehearse redeploying it** before - launch, not during an incident. -- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, - then remove the old shape in a separate change. Destructive schema behavior is - operation and version dependent, so each destructive change needs an exact - written procedure and verification on every node. -- **Use the same gates for rollback as for forward movement.** Availability, - journey synthetic, peer stability, data validation, traffic reconciliation. A - rollback is a deployment and can fail like one. +- **Keep the previous package addressable and rehearse redeploying it** before launch, not during an incident. +- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior is operation and version dependent, so each destructive change needs an exact written procedure and verification on every node. +- **Use the same gates for rollback as for forward movement.** Availability, journey synthetic, peer stability, data validation, traffic reconciliation. A rollback is a deployment and can fail like one. :::warning -If a release changed the meaning of persisted data, code rollback alone will not -fix it, and restoring a database to undo application code is usually the wrong -move. A restore rolls back every write in the window, including all the valid -ones, so it can violate your RPO in order to fix a code bug. Define forward repair -or replay for these cases instead. See -[Backup and Recovery](./backup-and-recovery.mdx). +If a release changed the meaning of persisted data, code rollback alone will not fix it, and restoring a database to undo application code is usually the wrong move. A restore rolls back every write in the window, including all the valid ones, so it can violate your RPO in order to fix a code bug. Define forward repair or replay for these cases instead. See [Backup and Recovery](./backup-and-recovery.mdx). ::: ### Prove it Rehearse a stopped rollout end to end on a non-production cluster: -1. Declare a threshold before you start, for example "stop if journey success on - the new cohort is more than 0.5 percent below the control cohort over five - minutes." +1. Declare a threshold before you start, for example "stop if journey success on the new cohort is more than 0.5 percent below the control cohort over five minutes." 2. Deploy a component that fails that threshold deliberately, to a bounded cohort. 3. Detect it through your dashboards rather than because you know what you did. -4. Reverse it, and time from decision to restored traffic. That number is your - release RTO, and it belongs in your reliability plan. -5. Verify uniform state afterwards: `get_components` on every node, and - `list_deployments` showing the reversal. +4. Reverse it, and time from decision to restored traffic. That number is your release RTO, and it belongs in your reliability plan. +5. Verify uniform state afterwards: `get_components` on every node, and `list_deployments` showing the reversal. -The step people fail is 5. A partially reversed cluster looks fine on a dashboard -because the healthy majority dominates the average. +The step people fail is 5. A partially reversed cluster looks fine on a dashboard because the healthy majority dominates the average. ## Operational notes -- **Version parity is an operating requirement.** Mixed Harper versions in a - cluster change replication and deployment behavior, so a rollout that stalls - halfway is a state you want to detect and exit, not sit in. -- **Keep deploy credentials in your delivery platform's secret store**, use TLS - and least privilege, and retain the operation result as change evidence. See - [secrets](/reference/v5/security/secrets). -- **Component deployment can replicate, so a deploy is a cluster event.** Keep - isolated single-node deployment available for validation, because if the only - deployment path you have is replicated then you have no way to test anything on - one node. -- **Flag cleanup is part of the release.** A feature flag with no owner and no - removal date becomes permanent configuration that nobody understands, and it - will eventually be the thing nobody can explain during an incident. -- **Record who decided, not only what happened.** Named decision authority is what - makes a stop gate function under pressure, and it costs nothing to write down in - advance. +- **Version parity is an operating requirement.** Mixed Harper versions in a cluster change replication and deployment behavior, so a rollout that stalls halfway is a state you want to detect and exit, not sit in. +- **Keep deploy credentials in your delivery platform's secret store**, use TLS and least privilege, and retain the operation result as change evidence. See [secrets](/reference/v5/security/secrets). +- **Component deployment can replicate, so a deploy is a cluster event.** Keep isolated single-node deployment available for validation, because if the only deployment path you have is replicated then you have no way to test anything on one node. +- **Flag cleanup is part of the release.** A feature flag with no owner and no removal date becomes permanent configuration that nobody understands, and it will eventually be the thing nobody can explain during an incident. +- **Record who decided, not only what happened.** Named decision authority is what makes a stop gate function under pressure, and it costs nothing to write down in advance. ## Readiness checklist - [ ] Artifacts are immutable and referenced by pinned version, never by branch - [ ] Deploy and expose are separate actions in your pipeline -- [ ] The previous known-good artifact is addressable, and redeploying it has been - rehearsed +- [ ] The previous known-good artifact is addressable, and redeploying it has been rehearsed - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before the rollout starts - [ ] A named decision owner for advance, hold, stop, and roll back -- [ ] Pipeline polls the restart job rather than treating the deploy response as - proof the new code is serving -- [ ] `"replicated": false` used deliberately wherever a change is meant to reach - one node only, since replication is the default +- [ ] Pipeline polls the restart job rather than treating the deploy response as proof the new code is serving +- [ ] `"replicated": false` used deliberately wherever a change is meant to reach one node only, since replication is the default - [ ] Cluster deploys use `"restart": "rolling"`, not `"restart": true` - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change -- [ ] Schema changes follow expand-then-contract, with destructive operations - documented per node +- [ ] Schema changes follow expand-then-contract, with destructive operations documented per node - [ ] Forward repair defined for changes that alter persisted meaning - [ ] Measured release RTO recorded from a rehearsed reversal ## Additional Resources -- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the - pipeline mechanics and deploy credentials -- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) - for every parameter including `urlPath`, `host`, and replication controls -- [Applications reference](/reference/v5/components/applications) for component - structure and the full component operation set -- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) - for `list_deployments` and `get_deployment` -- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for - `urlPath` and `host` activation -- [Configuration operations](/reference/v5/configuration/operations) for - `set_configuration` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the pipeline mechanics and deploy credentials +- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) for every parameter including `urlPath`, `host`, and replication controls +- [Applications reference](/reference/v5/components/applications) for component structure and the full component operation set +- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) for `list_deployments` and `get_deployment` +- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for `urlPath` and `host` activation +- [Configuration operations](/reference/v5/configuration/operations) for `set_configuration` and restart requirements - [Database schema](/reference/v5/database/schema) for schema evolution -- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) - for running more than one component side by side +- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) for running more than one component side by side diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx index 9a226224f..4e695fdb7 100644 --- a/learn/administration/sizing-a-harper-cluster.mdx +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -6,33 +6,22 @@ sidebar_position: 3 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Most clusters get sized by dividing expected peak throughput by measured per-node -throughput and rounding up. That arithmetic produces a cluster that is exactly -large enough to handle a good day, which means the first node failure during peak -traffic becomes a user-visible outage. +Most clusters get sized by dividing expected peak throughput by measured per-node throughput and rounding up. That arithmetic produces a cluster that is exactly large enough to handle a good day, which means the first node failure during peak traffic becomes a user-visible outage. -Sizing Harper starts from a different question. Not "how much traffic do we -have," but "how many nodes are we willing to lose at once, and can the survivors -carry the load." This guide gives you the capacity rule that follows from that, -what to include when you measure a node, and how to turn an availability target -into an operating budget you can actually spend. +Sizing Harper starts from a different question. Not "how much traffic do we have," but "how many nodes are we willing to lose at once, and can the survivors carry the load." This guide gives you the capacity rule that follows from that, what to include when you measure a node, and how to turn an availability target into an operating budget you can actually spend. ## What You Will Learn - The capacity invariant that sizes for the failure state, with a worked example -- What to include in a per-node throughput measurement, and the three things - people leave out +- What to include in a per-node throughput measurement, and the three things people leave out - Why maintenance is a planned failure, and what that means for change windows - How to choose a topology from your objectives rather than from a diagram -- How to convert an availability SLO into minutes per month, and what each tier - demands of your automation +- How to convert an availability SLO into minutes per month, and what each tier demands of your automation ## Prerequisites -- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your - service boundary inventory -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), - because capacity headroom is meaningless if traffic cannot be moved off a node +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your service boundary inventory +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), because capacity headroom is meaningless if traffic cannot be moved off a node - A load generation tool and a non-production cluster you can push to saturation - A stated peak throughput requirement for your critical journeys @@ -44,15 +33,9 @@ The rule: (N - F) x tested per-node throughput x target utilization >= required peak throughput ``` -Where `N` is your node count, `F` is the largest simultaneous node loss the design -tolerates, and target utilization is the fraction of measured capacity you are -willing to run at, which should leave room for latency to stay acceptable rather -than merely for requests to complete. +Where `N` is your node count, `F` is the largest simultaneous node loss the design tolerates, and target utilization is the fraction of measured capacity you are willing to run at, which should leave room for latency to stay acceptable rather than merely for requests to complete. -Worked example. Suppose your critical journey needs 12,000 requests per second at -peak, a node sustains 5,000 requests per second in your own test with acceptable -latency, you will run at 70 percent utilization, and you want to survive losing -one node: +Worked example. Suppose your critical journey needs 12,000 requests per second at peak, a node sustains 5,000 requests per second in your own test with acceptable latency, you will run at 70 percent utilization, and you want to survive losing one node: ```text Required surviving capacity = 12,000 / 0.7 = 17,143 req/s @@ -60,36 +43,20 @@ Surviving nodes needed = 17,143 / 5,000 = 3.43 -> 4 N = 4 + F (1) = 5 nodes ``` -Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded -up to 3, and that cluster degrades the moment anything goes wrong. +Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded up to 3, and that cluster degrades the moment anything goes wrong. -Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not -double the cluster, so tolerating a second simultaneous failure is often cheaper -than people expect at this size. And the utilization factor is doing as much work -as `F` is: sizing to 100 percent of measured capacity means your "surviving" -nodes are at saturation, where latency degrades long before throughput does. +Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not double the cluster, so tolerating a second simultaneous failure is often cheaper than people expect at this size. And the utilization factor is doing as much work as `F` is: sizing to 100 percent of measured capacity means your "surviving" nodes are at saturation, where latency degrades long before throughput does. ## Measure a node honestly -The per-node number in that formula is the one most likely to be wrong, because -benchmark conditions are kinder than production. Include all of this: - -- **Representative application code, data shape, and query mix.** A Harper node - runs your component logic in the same process as the data access, so your code - is part of the capacity measurement in a way it would not be for a standalone - database. A synthetic key-value benchmark tells you very little about the node's - capacity to serve your journey. -- **Realistic downstream latency.** If your resource calls an upstream pricing - service, its latency occupies worker capacity on the node. -- **Replication catch-up load.** A node that is feeding a recovering peer is doing - work that does not appear in its own request metrics. Measure with a peer - synchronizing, because that is precisely the state you will be in when you are - already down a node. -- **Traffic imbalance.** Configured weights express intent. Actual request - distribution is what consumes capacity, and it is rarely even. - -Measure each node separately rather than dividing a cluster total by node count. -An average conceals the one node that is about to tip. +The per-node number in that formula is the one most likely to be wrong, because benchmark conditions are kinder than production. Include all of this: + +- **Representative application code, data shape, and query mix.** A Harper node runs your component logic in the same process as the data access, so your code is part of the capacity measurement in a way it would not be for a standalone database. A synthetic key-value benchmark tells you very little about the node's capacity to serve your journey. +- **Realistic downstream latency.** If your resource calls an upstream pricing service, its latency occupies worker capacity on the node. +- **Replication catch-up load.** A node that is feeding a recovering peer is doing work that does not appear in its own request metrics. Measure with a peer synchronizing, because that is precisely the state you will be in when you are already down a node. +- **Traffic imbalance.** Configured weights express intent. Actual request distribution is what consumes capacity, and it is rarely even. + +Measure each node separately rather than dividing a cluster total by node count. An average conceals the one node that is about to tip. @@ -121,39 +88,23 @@ await fetch('https://my-node.example.com:9925/', { -Watch CPU, memory, event loop and worker pressure from the `threads` attribute, -disk, and network, and pair them with request rate and latency from -[analytics](/reference/v5/analytics/overview). The number you want is not the -point where requests start failing. It is the point where your journey's latency -leaves your SLO, which arrives earlier. +Watch CPU, memory, event loop and worker pressure from the `threads` attribute, disk, and network, and pair them with request rate and latency from [analytics](/reference/v5/analytics/overview). The number you want is not the point where requests start failing. It is the point where your journey's latency leaves your SLO, which arrives earlier. :::tip -Record the conditions alongside the number: Harper version, node size, data -volume, query mix, and whether a peer was synchronizing. A per-node capacity -figure without its conditions is not reusable, and six months later nobody will -remember whether the test included replication load. +Record the conditions alongside the number: Harper version, node size, data volume, query mix, and whether a peer was synchronizing. A per-node capacity figure without its conditions is not reusable, and six months later nobody will remember whether the test included replication load. ::: ## Maintenance is a planned failure -Draining a node for a deploy, an upgrade, or an investigation consumes exactly the -same headroom that a node failure does. So a cluster sized for `F = 1` is running -at `F = 0` for the duration of every maintenance window, with no tolerance left. +Draining a node for a deploy, an upgrade, or an investigation consumes exactly the same headroom that a node failure does. So a cluster sized for `F = 1` is running at `F = 0` for the duration of every maintenance window, with no tolerance left. -The operating rule that follows: if draining one node means the service can no -longer tolerate its declared `F`, pause the change. Either wait for lower traffic, -or add capacity for the window, or accept and document the reduced tolerance for a -bounded period. What you should not do is treat the maintenance window as free -because nothing is technically broken. +The operating rule that follows: if draining one node means the service can no longer tolerate its declared `F`, pause the change. Either wait for lower traffic, or add capacity for the window, or accept and document the reduced tolerance for a bounded period. What you should not do is treat the maintenance window as free because nothing is technically broken. -This is also the argument for sizing `F` at 2 in a cluster that deploys -frequently. It is not paranoia about correlated hardware failure. It is that you -want to be able to deploy during business hours and still survive an incident. +This is also the argument for sizing `F` at 2 in a cluster that deploys frequently. It is not paranoia about correlated hardware failure. It is that you want to be able to deploy during business hours and still survive an incident. ## Choose a topology from objectives -Pick the pattern that matches the failures you actually need to survive, and know -what you have to prove for it to count. +Pick the pattern that matches the failures you actually need to survive, and know what you have to prove for it to count. | Pattern | When it fits | What you must prove | | ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | @@ -162,22 +113,13 @@ what you have to prove for it to count. | Multiple regions | Regional impairment or user locality is a real requirement | Data authority per region, route propagation time, convergence, degraded-mode behavior | | Edge or on-premises peers | Data locality or residency constrains where data can live | Replication scope per location, identity provisioning, backup path from each site | -The right-hand column is the useful one. A topology diagram is a claim, and the -claim is only true once you have exercised it. "Multiple failure domains" means -nothing if both domains draw from the same power feed, and you will not discover -that from your provider's documentation. +The right-hand column is the useful one. A topology diagram is a claim, and the claim is only true once you have exercised it. "Multiple failure domains" means nothing if both domains draw from the same power feed, and you will not discover that from your provider's documentation. -Note that adding a node is a data movement event as well as a capacity event, -since a Harper node carries data along with request handling. A node whose -databases have never synchronized downloads them in full before it is useful, so -scaling out is not instantaneous and cannot be your response to an unexpected -traffic spike. Size ahead of demand. +Note that adding a node is a data movement event as well as a capacity event, since a Harper node carries data along with request handling. A node whose databases have never synchronized downloads them in full before it is useful, so scaling out is not instantaneous and cannot be your response to an unexpected traffic spike. Size ahead of demand. ## Turn the SLO into a budget -An availability target is a quantity of unavailability you are permitted to spend -per month. Written that way it becomes an operating constraint rather than an -aspiration. +An availability target is a quantity of unavailability you are permitted to spend per month. Written that way it becomes an operating constraint rather than an aspiration. | Monthly SLO | Approximate maximum unavailability | What it demands | | ----------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -185,58 +127,34 @@ aspiration. | 99.95% | 21 min 55 sec | Automated node removal and a rehearsed release reversal become necessary | | 99.99% | 4 min 23 sec | Node failure must be close to transparent. Detection, traffic removal, and validation have to be automated, because no human response fits in the budget | -Figures assume a 30.44 day month. Your contractual definition, exclusions, and -measurement boundary may differ, and the boundary matters more than the number: -availability measured at your CDN edge and availability measured at the Harper -node are different quantities. +Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ, and the boundary matters more than the number: availability measured at your CDN edge and availability measured at the Harper node are different quantities. -The practical consequence is that the SLO tier determines how much automation you -need, not how many nodes. At three nines a person can be paged, look at a -dashboard, and drain a node. At four nines that same sequence has already spent -the entire month's budget. +The practical consequence is that the SLO tier determines how much automation you need, not how many nodes. At three nines a person can be paged, look at a dashboard, and drain a node. At four nines that same sequence has already spent the entire month's budget. ### Prove it -Run the failure state rather than calculating it. On a cluster carrying -representative load at your expected peak: +Run the failure state rather than calculating it. On a cluster carrying representative load at your expected peak: 1. Record per-node request rate, latency, and saturation as a baseline. -2. Drain one node using the sequence from - [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). -3. Measure what the survivors do: request rate, latency at your SLO percentile, - CPU and worker pressure, and whether any user-visible errors appeared during - the transition. -4. While still down a node, bring the drained node back and let it synchronize, so - the survivors are carrying peak traffic and feeding a recovering peer at the - same time. This is the real worst case and it is the one nobody tests. -5. Compare measured surviving capacity against your invariant. If step 4 pushed - latency out of SLO, your target utilization is too high or your `N` is too low. +2. Drain one node using the sequence from [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +3. Measure what the survivors do: request rate, latency at your SLO percentile, CPU and worker pressure, and whether any user-visible errors appeared during the transition. +4. While still down a node, bring the drained node back and let it synchronize, so the survivors are carrying peak traffic and feeding a recovering peer at the same time. This is the real worst case and it is the one nobody tests. +5. Compare measured surviving capacity against your invariant. If step 4 pushed latency out of SLO, your target utilization is too high or your `N` is too low. ## Operational notes -- **Utilization targets belong per journey, not per cluster.** A write-heavy - journey and a cached read journey consume very different resources on the same - node, so a single cluster-wide utilization figure will be wrong for both. -- **Re-measure after a version upgrade.** Per-node capacity is a property of a - Harper version, your component code, and your data volume. All three change. -- **Watch measured distribution, not configured weights.** Per-node request counts - from analytics are the ground truth. A misconfigured weight, a sticky session - policy, or DNS caching can leave one node doing far more work than the topology - claims. -- **Storage growth is a capacity dimension too.** Disk headroom, compaction - behavior, and backup space all scale with data volume rather than with request - rate, so they need their own thresholds. See - [compaction](/reference/v5/database/compaction). -- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are - still choosing `N` and living with `F`. +- **Utilization targets belong per journey, not per cluster.** A write-heavy journey and a cached read journey consume very different resources on the same node, so a single cluster-wide utilization figure will be wrong for both. +- **Re-measure after a version upgrade.** Per-node capacity is a property of a Harper version, your component code, and your data volume. All three change. +- **Watch measured distribution, not configured weights.** Per-node request counts from analytics are the ground truth. A misconfigured weight, a sticky session policy, or DNS caching can leave one node doing far more work than the topology claims. +- **Storage growth is a capacity dimension too.** Disk headroom, compaction behavior, and backup space all scale with data volume rather than with request rate, so they need their own thresholds. See [compaction](/reference/v5/database/compaction). +- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are still choosing `N` and living with `F`. ## Readiness checklist - [ ] `F` declared explicitly, and written down where change operators will see it - [ ] Per-node throughput measured with your own application code and data shape - [ ] Measurement taken with a peer synchronizing, not on an idle cluster -- [ ] Target utilization set from latency at your SLO percentile, not from request - success +- [ ] Target utilization set from latency at your SLO percentile, not from request success - [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers - [ ] Measurement conditions recorded alongside the capacity figure - [ ] Maintenance policy states what happens when a drain would breach `F` @@ -246,16 +164,10 @@ representative load at your expected peak: ## Additional Resources -- [Analytics overview](/reference/v5/analytics/overview) and - [analytics operations](/reference/v5/analytics/operations) for per-node request - and latency metrics +- [Analytics overview](/reference/v5/analytics/overview) and [analytics operations](/reference/v5/analytics/operations) for per-node request and latency metrics - [Grafana integration](/fabric/grafana-integration) for dashboards on Fabric -- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) - for self-managed metrics scraping -- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and - throughput trade-offs available per database +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) for self-managed metrics scraping +- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and throughput trade-offs available per database - [Compaction](/reference/v5/database/compaction) for storage growth behavior -- [Replication overview](/reference/v5/replication/overview) for peer - synchronization behavior when adding or returning a node -- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for - scenario-level targets built on this capacity model +- [Replication overview](/reference/v5/replication/overview) for peer synchronization behavior when adding or returning a node +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for scenario-level targets built on this capacity model