Software Engineering at Google

Fergus Henderson · article · 2019-02-19 source ↗ Reader 🖥️

Hierarchy & Org Charts Psychological Safety Pull Requests & Async Code Review Sociotechnical Architecture Technical Debt Trunk-Based & Continuous Delivery

Google sustains software velocity and quality at extreme scale (2+ billion lines, one repository, thousands of engineers) not through any single tool or policy but through a mutually reinforcing system of technical infrastructure (a single trunk-based monorepo, hermetic reproducible builds, and mandatory same-day code review) and organizational design (separated engineer/manager career tracks, OKRs deliberately decoupled from pay, 20% time, and periodic full rewrites) that together manufacture trust, ownership, and continuous renewal.

practical · expository — A practitioner's catalog-and-recommend piece: Henderson describes Google's engineering and people-management practices as of ~2017-2019 so other organizations can 'compare and contrast... and consider whether to apply some of these practices themselves.' It argues implicitly for the practices by describing their rationale, without claiming controlled evidence of causation.

Structure

Part	Role
Introduction (sec. 1)	context
Software development practices: source repository, build system, code review, testing, bug tracking, languages, debugging tools, release engineering, launch approval, post-mortems, frequent rewrites (sec. 2)	evidence — the technical-infrastructure half of the system
Project management: 20% time, OKRs, project approval, reorganizations, hack weeks (sec. 3)	evidence — how Google allocates attention and discretion
People management: roles (EM/SWE/Research Scientist/SRE/PM/TPM), facilities, training, transfers, performance appraisal (sec. 4)	evidence — the organizational-design half of the system
Conclusions (sec. 5)	concession + synthesis — explicitly admits no quantitative/causal evidence connecting individual practices to outcomes, only that they 'have stood the test of time... subject to the collective subjective judgement of many thousands of excellent software engineers'

Key terms

Build cop
A rotating team-member role whose job is to keep the trunk ('head') green by working with change authors to fix or roll back whatever broke the build, within minutes of a failure being detected.
Readability (certification)
A company-wide training-and-review process, per programming language, where an experienced engineer mentors another through a sustained series of real code reviews until satisfied the trainee writes idiomatic code in that language; once certified, that engineer's approval becomes required for non-trivial new code in that language.
Hermetic build step
A build action that depends only on its explicitly declared inputs — nothing ambient from the machine or environment — which is what makes the distributed build system's caching, determinism, and cross-user cache-sharing safe.
Tech Lead (vs. Engineering Manager)
A role, distinct from and not necessarily held by the Engineering Manager, that has final say on technical decisions for a project; management and technical leadership are deliberately separated so that career progression to the highest levels does not require managing people.
OKR score of 65%
A deliberately-not-100% target average: teams are expected to set ambitious objectives such that scoring ~65% on average is healthy, and OKR scores are visible but explicitly NOT used as direct input to performance appraisal — the score is a social/communication signal, not a grading instrument.

Propositions

explicitHaving (almost) all of Google's code in one repository, visible to all engineers regardless of team, with culturally-encouraged cross-boundary fixes, produces higher-quality shared infrastructure than siloed per-team repositories would.conf: high
explicitDevelopment should happen almost entirely at the head/trunk of the repository rather than on long-lived branches, because this surfaces integration problems early and minimizes merge work.conf: high
explicitCode review is most effective when performed as the code is being developed, not after the fact — which is why the 'experimental' section where review isn't enforced is discouraged for anything beyond throwaway exploration.conf: high
supportsFrom Async Code Reviews to Co-Creation Patterns · p14 — Henderson's 2017 observation that review is most effective done as code is developed (not afterward) is the same claim later generalized and argued more rigorously as 'continuous review is strictly stronger than delayed batch review.'
explicitEvery change to the main repository must be reviewed by at least one other engineer, and the author (not a central authority) chooses the reviewer, which keeps review fast by letting authors route around slow or overly-possessive reviewers.conf: high
supportsFrom Async Code Reviews to Co-Creation Patterns · p15 — Henderson's author-chosen-reviewer, review-during-development model treats review as a quality/learning mechanism rather than a gate — consistent with the later proposition that review's primary purpose is building in quality and accelerating learning, not gating.
explicitKeeping individual changes small makes review faster and makes it easier for authors to accept reviewer-suggested changes, because very large changes become too rigid to rework.conf: high
explicitBuild determinism and hermeticity (steps depending only on declared inputs) are what make a distributed, shared, cross-user build cache both fast and trustworthy at Google's scale.conf: high
explicitReleasing frequently (weekly, fortnightly, or daily) increases both engineer motivation and overall velocity, by shortening the feedback loop between writing code and seeing its effect.conf: high
explicitStaged rollout (staging → canary → gradual production rollout, with shadow traffic to staging) is the mechanism that makes frequent releases safe rather than risky.conf: high
explicitPost-mortems for production incidents should focus on the problem and remediation, never on blaming the people involved, because blame-focused review suppresses the honest reporting that root-cause analysis depends on.conf: high
explicitMost software at Google is rewritten every few years, and despite the heavy resource cost, this is net beneficial: it strips accumulated complexity tied to stale requirements, renews engineers' sense of ownership, and spreads engineer mobility and fresh technology across the company.conf: high
implicitA sense of personal ownership over code is not just a morale nicety but a direct productivity input — engineers who feel code is 'theirs' put more effort into it than into code they merely maintain.conf: medium
explicitSeparating the engineering career ladder from the management career ladder — so that reaching the highest technical levels never requires managing people — prevents a common failure mode where people become managers for status or pay rather than aptitude, at the cost of their teams' management quality.conf: high
explicitDeliberately decoupling OKR scores from performance appraisal — while still making them visible company-wide — lets the social pressure to 'score well' drive ambitious goal-setting without incentivizing people to set safe, easily-achieved goals.conf: high
explicitGiving engineers unsupervised discretionary time (20% time) is net-positive even though it has a real opportunity cost, because it surfaces ideas management wouldn't have approved, keeps motivation and retention up, and — being official rather than secret — gives management visibility into what would otherwise be hidden 'skunkwork.'conf: high
explicitSomewhat frequent corporate reorganization is a necessary cost of a large technology-driven organization, used (among other things) to counteract the 'shipping the org chart' failure mode where software architecture passively mirrors reporting structure rather than being designed.conf: high
supportsDesigning the AI-native engineering organization · p2 — Henderson's 'shipping the org chart' critique and the periodic-reorg remedy are an earlier instance of the same principle the later doc states directly: structural/org-chart change should track real need, not be reflexive.
implicitcontestedLow-stakes, frequent social recognition mechanisms (peer bonuses, kudos) that require no manager approval are a cheaper and faster way to reinforce desired behavior than relying solely on the formal, infrequent performance-appraisal cycle.conf: medium
contradictsThe engineering manager role is splitting in two · p6 — Henderson describes Google's engineer/manager split as a deliberate, stable, working policy; the later document describes the same kind of split elsewhere happening as an unplanned, cost-driven byproduct of layoffs rather than deliberate design — same structural feature, opposite account of why it exists and how durable it is.
explicitDespite having been at Google over ten years and become a manager, Henderson does not fully understand how project approval/cancellation decisions actually get made — the formal process is far less uniform and far more discretionary than the well-defined launch-approval process.conf: high

Arguments

Mandatory, author-chosen, same-day-ish code review at head is what lets thousands of engineers safely share one trunk without long-lived branches. strength: high

Explicit premises:

Every change requires at least one reviewer, and an owner if the author isn't one
Authors pick reviewers, which keeps turnaround fast
Changes are kept small to keep review tractable
Development in the unreviewed 'experimental' area is discouraged for anything beyond throwaway work

Implicit premises:

Review quality and speed degrade as the batch gets larger or more stale (further from authorship)
A trust-based, decentralized reviewer-selection process scales better than a centralized review-assignment bureaucracy would

Inferences:

As the volume and size of changes grow — e.g. with AI-generated code — the same logic implies review has to move even earlier/closer to authorship (continuous, in-the-loop review) rather than later, or the system breaks down; Henderson doesn't draw this inference because LLM-scale code generation didn't exist when he wrote this.

Separating the engineering and management career tracks, with technical leadership available without managing people, produces better outcomes than a single ladder that funnels senior ICs into management. strength: medium

Explicit premises:

Tech Lead is a distinct role from Engineering Manager and more often held by a Software Engineer
Promotion to the highest engineering levels requires 'leadership' but not people-management
People who lack the desire or skill to manage still have a full career path

Implicit premises:

Without this separation, some fraction of people become managers purely for status/compensation reasons, independent of managerial aptitude
Reluctant/ill-suited managers measurably harm their reports' experience and the org's output

Inferences:

Any organization that collapses the dual ladder (e.g. for cost reasons) should expect to reintroduce the 'accidental manager' failure mode Google's split was designed to avoid.

Treating OKR scores as visible-but-not-reward-linked is what allows Google to ask for genuinely ambitious (not sandbagged) goals.

Explicit premises:

OKRs and scores are made visible across the company
Scores are explicitly not used as direct input to performance appraisal
The target average score is ~65%, i.e. teams are asked to set ~50% more goals than they expect to hit

Implicit premises:

If scores did feed compensation, rational actors would set easier goals to protect their score
Social visibility alone (without a reward link) is sufficient motivational pressure to perform

Narratives

Teams install prominent displays or physical sculptures with color-coded lights (green/red/black) showing live build status, to keep the whole team's attention on keeping the build green.
→ Shows (rather than argues) that trunk-based development at scale depends on making build health a shared, ambient, almost physical social signal — not just a dashboard someone checks.
Code-review tooling labels large changes with increasingly disparaging size labels ("large", "freakin huge"), occasionally swapped for joke labels on days like talk-like-a-pirate day.
→ Implicitly supports the small-changes proposition by showing the social/cultural (not just procedural) pressure Google applies against large diffs — and that this pressure is kept light-hearted rather than punitive.

Questions addressed

✅ How does an organization sustain software quality and velocity when it has grown to billions of lines of code and thousands of engineers sharing a codebase?
Through a combination of shared, visible, culturally-reinforced infrastructure (monorepo, trunk, reviewed/tested/hermetic builds, staged releases) plus organizational choices (dual career ladders, decoupled OKRs, discretionary time, frequent rewrites) that sustain trust and ownership.
⭕ Do these practices generalize to organizations without Google's hiring bar, headcount, and financial resources?
⭕ What happens to the 'review at authorship time, keep changes small' model when the volume or size of authored code grows faster than human reviewer attention can track — i.e. a reviewer-capacity ceiling?

Gaps

incompleteThe paper's code-review model assumes human authorship at roughly human pace; it never considers what happens once a single engineer can direct an agent to produce far more reviewable code than they could write by hand, which breaks the 'small changes, author-chosen reviewer, review during authorship' model's load-bearing assumption that review capacity roughly tracks authorship capacity.
Read next: See extract-horthy-no-vibes-allowed-complex-codebases and extract-boonstra-spec-driven-production-grade-development for how review is being pushed up to the plan/verdict level specifically because code-diff-level review no longer scales against AI-generated volume.
incompleteHenderson presents the engineer/manager dual-ladder as simply working, with no acknowledgment that a later cost-driven flattening trend (fewer managers, broader spans, informal TLM blending) might erode exactly the separation his argument depends on.
Read next: See extract-engineering-manager-role-splitting-in-two for the 2020s-era pressure toward recombining tech-lead and manager duties, and whether Larson's 'TLM is unstable' critique applies once team sizes shrink.
uninformedHenderson himself flags that he doesn't understand how project approval/cancellation decisions get made despite a decade at the company and now being a manager — an explicit admission that the formal account of 'how work gets decided' (which he otherwise writes confidently about) has a real gap where politics/discretion actually governs.
Read next: Connects to the broader corpus theme of formal org charts vs. how work actually flows through informal, political channels.
incompleteThe case for 'frequent full rewrites' is made at the whole-system level, without considering whether incremental/boundary-preserving rewrites (extracting a module, a service, or a function at a time) could capture the same ownership-renewal and complexity-shedding benefits at lower risk — a distinction the corpus's later agentic-rewrite literature draws explicitly.
Read next: See extract-incremental-rewrite-granularity-adjacent material on how rewrite risk scales with the unit of change.

Software Engineering at Google

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Related documents