SaaS organizations

Same outcome, two circuits

Most operational problems in a software company have two or three well-worn answers. They aim at the same outcome and trade speed, cost and control differently. In Endoskeletal each answer is a circuit you can read, check and simulate against the other before you choose. Here are three common pairs.

Shipping a change to production

Both circuits end in the same external act, execute Deploy. They differ in who authorizes it, when, and what stops it.

A. Continuous deployment

Low-risk changes ship with no human in the loop; high-risk ones route to an engineering manager. Rollback is an obligation, not a hope.

Text description of this diagram

A merged pull request creates the change c. CI produces the claim ci.passed(c). Under the AutoShip envelope, held by the Deployer program, a low-risk change with CI at T is deployed as a canary from 5 % to 100 %. The monitor's error-rate(c) claim is watched; above 1 % the Rollback duty fires with a 5-minute deadline. A high-risk change makes the envelope's condition false and routes to the Review route: an engineering manager decides within 4 hours, and that approval authorizes the deploy.

continuous deploymentesk

use rz.RealizationEnvelope(
  holder      = Deployer,
  requirement = Api,
  operation   = eng.Deploy,
  admissible  = c.risk = "low" and holds ci.passed(c)
) as AutoShip

use org.Approval(
  approver   = EM,
  subject    = eng.Change,
  threshold  = c.risk = "high",
  deadline   = 4 h,
  escalateTo = VPEng
) as Review

norm Rollback {
  obligation of Deployer to persona
  when occurred execute(eng.Deploy, c) and holds ops.error-rate(c) > 1 %
  aim within 5 min: occurred execute(eng.Rollback, c)
  not tradeable
}

B. Release train with a change advisory board

Changes batch into a weekly train; a board votes; a freeze window blocks deploys; a narrow hotfix route overrides it.

Text description of this diagram

Merged changes collect on a release branch. At the Thursday tick the train is cut and the change advisory board must record at least 3 of 5 vote-approve decisions within one business day; that aggregate is what gives the ReleaseManager the power to deploy. A freeze window prohibition blocks deploys Friday to Sunday. Severity-1 fixes take the hotfix route: a VP decides, and because the hotfix norm is more specific it overrides the freeze.

release trainesk

norm Cut {
  obligation of ReleaseManager to persona
  when occurred tick(t) and t.boundary = "week" and t.weekday = "Thu"
  aim within 2 h: occurred derive(eng.ReleaseCandidate(t.period)) by ReleaseManager
}

norm CABVote {
  power of CAB to decide vote-approve | vote-reject about eng.ReleaseCandidate
}

norm ShipTrain {
  power of ReleaseManager to authorize execute about eng.Deploy
  when r is eng.ReleaseCandidate
       and count decide(vote-approve, r) by CAB over 1 business-days >= 3
}

norm Freeze {
  prohibition on any Party
  aim occurred execute(eng.Deploy, _)
      and holds cal.weekday in { "Fri", "Sat", "Sun" }
  priority specialis
}

norm Hotfix {
  power of VPEng to authorize execute about eng.Deploy
  when c is eng.Change and c.severity = 1
  overrides Freeze
}

Continuous deployment

18 min

Release train

3 days 20 h

A change merged Monday at 10:00. Same code, same tests, two circuits.

	Continuous deployment	Release train + CAB
Lead time	Minutes	Up to a week
Human touches per change	0 for low risk, 1 for high risk	3 votes per train, plus the release manager
What stops a bad change	An obligation to roll back within 5 minutes of error rate > 1 %	A board vote before release, and a freeze on weekends
Blast radius	One change at a time, canaried	A week of changes at once
Fits when	Strong tests and observability, reversible deploys	Regulated customers, contractual change windows, on-prem installs

Many companies run both: the train for customers whose contracts require change windows, continuous deployment for everyone else. That is one more condition on each power, and the checker proves the two never authorize the same deploy.

Incident response

A. One rotation with an escalation ladder

One rotation, three rungs. Each rung is an obligation whose or-else is the next rung.

Text description of this diagram

An SLO burn-rate claim above 10× pages the Primary on-call. The Primary must acknowledge within 5 minutes; the or-else obliges the Secondary, whose or-else obliges the engineering manager. Whoever acknowledges appoints an incident commander. When the incident resolves, a postmortem obligation with a 5-business-day deadline detaches.

single rotationesk

norm PrimaryAck {
  obligation of OnCall.Primary to persona
  when holds ops.burn-rate(ApiUp) > 10
  aim within 5 min: occurred assert(ops.Ack(_)) by OnCall.Primary
  or-else SecondaryAck
}

norm SecondaryAck {
  obligation of OnCall.Secondary to persona
  when violated commitment of PrimaryAck for _
  aim within 5 min: occurred assert(ops.Ack(_)) by OnCall.Secondary
  or-else ManagerAck
}

norm ManagerAck {
  obligation of EM to persona
  when violated commitment of SecondaryAck for _
  aim within 10 min: occurred assert(ops.Ack(_)) by EM
}

norm AppointIC {
  power of any org.Member to appoint _ into IncidentCommander
  when occurred assert(ops.Ack(_)) by self
}

norm Postmortem {
  obligation of IncidentCommander to persona
  when occurred assert(ops.Resolved(i))
  aim within 5 business-days: occurred derive(ops.Postmortem(i)) by IncidentCommander
}

B. Follow the sun

Three regional seats, a routing power keyed to the calendar, and a handoff obligation at every shift boundary.

Text description of this diagram

The alert goes to a routing power that selects the on-call seat whose business hours contain the current time (Americas, EMEA or APAC). That seat's occupant acts as or appoints the incident commander. If the incident is still open at shift end, a handoff duty requires the next region to acknowledge within 15 minutes.

follow the sunesk

pattern RegionalOnCall(region : Text) {
  position OnCall : org.Member {
    cardinality 1..
    eligible when holds hr.region(self) = region
  }

  norm Ack {
    obligation of OnCall to persona
    when occurred request(ops.Page(x)) and holds cal.business-hours(region)
    aim within 5 min: occurred assert(ops.Ack(x)) by OnCall
  }
}

use RegionalOnCall(region = "AMER") as Americas
use RegionalOnCall(region = "EMEA") as EMEA
use RegionalOnCall(region = "APAC") as APAC

norm Route {
  obligation of persona to persona
  when holds ops.burn-rate(ApiUp) > 10
  aim within 1 min: occurred request(ops.Page(_)) by persona
}

norm Handoff {
  obligation of IncidentCommander to persona
  when holds ops.open(i) and occurred tick(t) and t.boundary = "shift"
  aim within 15 min: occurred appoint(_, IncidentCommander)
      and occurred assert(ops.Ack(i)) by _
}

	Single rotation	Follow the sun
Night pages	Frequent	None by design
Headcount needed	One team	Three regions with on-call depth
Risk	Fatigue, slow night acks	Context lost at handoff; the handoff duty exists to make it explicit

Access to production data

A. Standing role-based access

Simple and fast: the power sits on a position, and a periodic review keeps the occupants honest.

Text description of this diagram

The DBA position holds a standing DataAccess power, so a query by the DBA is valid and runs against the production database. Every query is recorded. A quarterly access review obliges Security to re-certify who occupies the position.

standing accessesk

norm DataAccess {
  power of DBA to authorize execute about data.Query
  level operational
}

norm AccessReview {
  obligation of Security.Lead to persona
  when occurred tick(t) and t.boundary = "quarter"
  aim within 10 business-days: occurred derive(sec.AccessRecertified(DBA, t.period)) by Security.Lead
}

B. Just-in-time break-glass

Nobody holds standing access. A peer delegates a narrow power that expires on its own; anything after expiry misfires.

Text description of this diagram

An engineer requests access with a reason. A peer must approve within 15 minutes; approval is a delegate of a power scoped to one database that expires after 60 minutes. Queries inside the window are valid and recorded, and a review obligation detaches for the next business day. After expiry the same query misfires: the delegation no longer covers it.

break-glass accessesk

norm Grant {
  power of Engineer to delegate execute about data.Query
  when r is sec.AccessRequest and r.requester != self
       and r.scope = "one-database"
  level operational
}

use org.Delegation(
  from    = Engineer,
  subject = data.Query,
  expires = 60 min
) as BreakGlass

norm Review {
  obligation of Security.Lead to persona
  when occurred delegate(_, r) under BreakGlass
  aim within 1 business-days: occurred derive(sec.Reviewed(r)) by Security.Lead
}

	Standing access	Break-glass
Time to first query	Immediate	Minutes (a peer must approve)
Exposure if a credential leaks	Everything the role can reach, until the next review	One database, for at most an hour
Audit story	Quarterly re-certification	Every access has a reason, an approver and a review

Choosing between them

Because both versions are specifications, you don't have to argue about them. Simulate each against the same scenario, with the same seeds, and compare lead time, pages per engineer, human approvals, violated deadlines and cost. Differences in the results come from the circuit, not from luck. Then adopt one as an amendment, or keep both behind conditions.