# Same outcome, two circuits

> Most operational problems in a software company have two or three well-worn answers. They aim at the same outcome and trade speed, cost and control differently. In Endoskeletal each answer is a circuit you can read, check and simulate against the other before you choose. Here are three common pairs.

Canonical: https://endoskeletal.com/two-ways/
Last updated: 2026-09-28

## Shipping a change to production

Both circuits end in the same external act, `execute Deploy`. They differ in who authorizes it, when, and what stops it.

### A. Continuous deployment

**Diagram: Shipping a change with continuous deployment.**

A merged pull request creates the change `c`. CI produces the claim `ci.passed(c)`. Under the **AutoShip** envelope, held by the Deployer program, a low-risk change with CI at T is deployed as a canary from 5 % to 100 %. The monitor's `error-rate(c)` claim is watched; above 1 % the **Rollback duty** fires with a 5-minute deadline. A high-risk change makes the envelope's condition false and routes to the **Review route**: an engineering manager decides within 4 hours, and that approval authorizes the deploy.

*Low-risk changes ship with no human in the loop; high-risk ones route to an engineering manager. Rollback is an obligation, not a hope.*

*continuous deployment*

```esk
use rz.RealizationEnvelope(
  holder      = Deployer,
  requirement = Api,
  operation   = eng.Deploy,
  admissible  = c.risk = "low" and holds ci.passed(c)
) as AutoShip

use org.Approval(
  approver   = EM,
  subject    = eng.Change,
  threshold  = c.risk = "high",
  deadline   = 4 h,
  escalateTo = VPEng
) as Review

norm Rollback {
  obligation of Deployer to persona
  when occurred execute(eng.Deploy, c) and holds ops.error-rate(c) > 1 %
  aim within 5 min: occurred execute(eng.Rollback, c)
  not tradeable
}
```

### B. Release train with a change advisory board

**Diagram: Shipping a change with a release train and change advisory board.**

Merged changes collect on a release branch. At the Thursday tick the train is cut and the **change advisory board** must record at least 3 of 5 vote-approve decisions within one business day; that aggregate is what gives the ReleaseManager the power to deploy. A **freeze window** prohibition blocks deploys Friday to Sunday. Severity-1 fixes take the **hotfix route**: a VP decides, and because the hotfix norm is more specific it overrides the freeze.

*Changes batch into a weekly train; a board votes; a freeze window blocks deploys; a narrow hotfix route overrides it.*

*release train*

```esk
norm Cut {
  obligation of ReleaseManager to persona
  when occurred tick(t) and t.boundary = "week" and t.weekday = "Thu"
  aim within 2 h: occurred derive(eng.ReleaseCandidate(t.period)) by ReleaseManager
}

norm CABVote {
  power of CAB to decide vote-approve | vote-reject about eng.ReleaseCandidate
}

norm ShipTrain {
  power of ReleaseManager to authorize execute about eng.Deploy
  when r is eng.ReleaseCandidate
       and count decide(vote-approve, r) by CAB over 1 business-days >= 3
}

norm Freeze {
  prohibition on any Party
  aim occurred execute(eng.Deploy, _)
      and holds cal.weekday in { "Fri", "Sat", "Sun" }
  priority specialis
}

norm Hotfix {
  power of VPEng to authorize execute about eng.Deploy
  when c is eng.Change and c.severity = 1
  overrides Freeze
}
```

**Lead time, same change merged Monday 10:00.** Continuous deployment: 18 min; Release train: 3 days 20 h.

|  | Continuous deployment | Release train + CAB |
| --- | --- | --- |
| Lead time | Minutes | Up to a week |
| Human touches per change | 0 for low risk, 1 for high risk | 3 votes per train, plus the release manager |
| What stops a bad change | An obligation to roll back within 5 minutes of error rate > 1 % | A board vote before release, and a freeze on weekends |
| Blast radius | One change at a time, canaried | A week of changes at once |
| Fits when | Strong tests and observability, reversible deploys | Regulated customers, contractual change windows, on-prem installs |

Many companies run both: the train for customers whose contracts require change windows, continuous deployment for everyone else. That is one more condition on each power, and the checker proves the two never authorize the same deploy.

## Incident response

### A. One rotation with an escalation ladder

**Diagram: Incident response with a single on-call rotation and an escalation ladder.**

An SLO burn-rate claim above 10× pages the Primary on-call. The Primary must acknowledge within 5 minutes; the or-else obliges the Secondary, whose or-else obliges the engineering manager. Whoever acknowledges appoints an **incident commander**. When the incident resolves, a **postmortem** obligation with a 5-business-day deadline detaches.

*One rotation, three rungs. Each rung is an obligation whose or-else is the next rung.*

*single rotation*

```esk
norm PrimaryAck {
  obligation of OnCall.Primary to persona
  when holds ops.burn-rate(ApiUp) > 10
  aim within 5 min: occurred assert(ops.Ack(_)) by OnCall.Primary
  or-else SecondaryAck
}

norm SecondaryAck {
  obligation of OnCall.Secondary to persona
  when violated commitment of PrimaryAck for _
  aim within 5 min: occurred assert(ops.Ack(_)) by OnCall.Secondary
  or-else ManagerAck
}

norm ManagerAck {
  obligation of EM to persona
  when violated commitment of SecondaryAck for _
  aim within 10 min: occurred assert(ops.Ack(_)) by EM
}

norm AppointIC {
  power of any org.Member to appoint _ into IncidentCommander
  when occurred assert(ops.Ack(_)) by self
}

norm Postmortem {
  obligation of IncidentCommander to persona
  when occurred assert(ops.Resolved(i))
  aim within 5 business-days: occurred derive(ops.Postmortem(i)) by IncidentCommander
}
```

### B. Follow the sun

**Diagram: Incident response with follow-the-sun regional on-call.**

The alert goes to a routing power that selects the on-call seat whose business hours contain the current time (Americas, EMEA or APAC). That seat's occupant acts as or appoints the **incident commander**. If the incident is still open at shift end, a **handoff duty** requires the next region to acknowledge within 15 minutes.

*Three regional seats, a routing power keyed to the calendar, and a handoff obligation at every shift boundary.*

*follow the sun*

```esk
pattern RegionalOnCall(region : Text) {
  position OnCall : org.Member {
    cardinality 1..
    eligible when holds hr.region(self) = region
  }

  norm Ack {
    obligation of OnCall to persona
    when occurred request(ops.Page(x)) and holds cal.business-hours(region)
    aim within 5 min: occurred assert(ops.Ack(x)) by OnCall
  }
}

use RegionalOnCall(region = "AMER") as Americas
use RegionalOnCall(region = "EMEA") as EMEA
use RegionalOnCall(region = "APAC") as APAC

norm Route {
  obligation of persona to persona
  when holds ops.burn-rate(ApiUp) > 10
  aim within 1 min: occurred request(ops.Page(_)) by persona
}

norm Handoff {
  obligation of IncidentCommander to persona
  when holds ops.open(i) and occurred tick(t) and t.boundary = "shift"
  aim within 15 min: occurred appoint(_, IncidentCommander)
      and occurred assert(ops.Ack(i)) by _
}
```

|  | Single rotation | Follow the sun |
| --- | --- | --- |
| Night pages | Frequent | None by design |
| Headcount needed | One team | Three regions with on-call depth |
| Risk | Fatigue, slow night acks | Context lost at handoff; the handoff duty exists to make it explicit |

## Access to production data

### A. Standing role-based access

**Diagram: Production data access through a standing role.**

The DBA position holds a standing **DataAccess** power, so a query by the DBA is valid and runs against the production database. Every query is recorded. A **quarterly access review** obliges Security to re-certify who occupies the position.

*Simple and fast: the power sits on a position, and a periodic review keeps the occupants honest.*

*standing access*

```esk
norm DataAccess {
  power of DBA to authorize execute about data.Query
  level operational
}

norm AccessReview {
  obligation of Security.Lead to persona
  when occurred tick(t) and t.boundary = "quarter"
  aim within 10 business-days: occurred derive(sec.AccessRecertified(DBA, t.period)) by Security.Lead
}
```

### B. Just-in-time break-glass

**Diagram: Production data access through just-in-time break-glass delegation.**

An engineer requests access with a reason. A peer must approve within 15 minutes; approval is a `delegate` of a power scoped to one database that expires after 60 minutes. Queries inside the window are valid and recorded, and a review obligation detaches for the next business day. After expiry the same query misfires: the delegation no longer covers it.

*Nobody holds standing access. A peer delegates a narrow power that expires on its own; anything after expiry misfires.*

*break-glass access*

```esk
norm Grant {
  power of Engineer to delegate execute about data.Query
  when r is sec.AccessRequest and r.requester != self
       and r.scope = "one-database"
  level operational
}

use org.Delegation(
  from    = Engineer,
  subject = data.Query,
  expires = 60 min
) as BreakGlass

norm Review {
  obligation of Security.Lead to persona
  when occurred delegate(_, r) under BreakGlass
  aim within 1 business-days: occurred derive(sec.Reviewed(r)) by Security.Lead
}
```

|  | Standing access | Break-glass |
| --- | --- | --- |
| Time to first query | Immediate | Minutes (a peer must approve) |
| Exposure if a credential leaks | Everything the role can reach, until the next review | One database, for at most an hour |
| Audit story | Quarterly re-certification | Every access has a reason, an approver and a review |

## Choosing between them

Because both versions are specifications, you don't have to argue about them. Simulate each against the same scenario, with the same seeds, and compare lead time, pages per engineer, human approvals, violated deadlines and cost. Differences in the results come from the circuit, not from luck. Then adopt one as an amendment, or keep both behind conditions.