Designing Data-Intensive Applications: The Cloud & Doing the Right Thing

The Pragmatic Engineer · email · 2026-05-05 Reader 🖥️

Cloud computing changes the technical and economic architecture of data systems in ways that demand new operational thinking, while the growing power of data-intensive applications creates ethical obligations for engineers to consider the societal consequences of what they build.

practical · mixed — Combines expository technical analysis (cloud vs self-hosting tradeoffs) with argumentative ethical reasoning (engineers' responsibility). The excerpts are from a technical book but framed by a newsletter author who contextualizes and recommends action.

Structure

Part	Role
Introduction and framing by Gergely Orosz	Context — establishes the credibility of the source (Kleppmann/Riccomini, DDIA 2nd edition) and sets up the two excerpts
Cloud versus self-hosting tradeoffs (Chapter 1 excerpt)	Core technical argument — presents the build-vs-buy spectrum, the pros and cons of cloud services, and the architectural consequences (cloud-native, separation of storage and compute, changed operations role)
Pros & Cons of Cloud Services	Evidence — systematic enumeration of when cloud is cheaper/easier vs when self-hosting wins, plus the six major downsides of cloud dependency
Cloud-Native System Architecture	Technical deepening — explains how cloud-native design (layered services, disaggregated storage/compute, multitenancy) differs fundamentally from self-hosted architecture
Operations in the Cloud Era	Practical consequence — shows how cloud shifts operations from machine-level administration to service integration and financial planning
Doing the right thing as a software engineer (Chapter 14 excerpt)	Core ethical argument — asserts engineers' responsibility for consequences of data systems, covering predictive analytics bias, accountability, feedback loops, and surveillance
Predictive Analytics (bias, discrimination, accountability, feedback loops)	Evidence and argument — demonstrates how data-driven decisions can harm people through bias amplification, lack of accountability, and self-reinforcing feedback loops
Surveillance	Provocation — reframes 'data-intensive' as 'surveillance-intensive' to force recognition of what data collection actually means
Takeaways by Gergely Orosz	Synthesis — summarizes what changed between editions (cloud, AI, local-first, regulation) and endorses the book's ethical closing

Key terms

cloud native
Architecture designed from the ground up to build upon lower-level cloud services (object storage, etc.) rather than merely running traditional software on cloud VMs; characterized by separation of storage and compute, multitenancy, and layered service composition
separation of storage and compute
Cloud-native design pattern where data storage (e.g., S3) and computation (CPU/RAM) are disaggregated onto separate infrastructure, connected by network, rather than co-located on the same machine
algorithmic prison
The condition of being systematically excluded from jobs, travel, insurance, financial services, and other societal participation by automated risk-scoring systems, without proof of guilt or meaningful appeal
vendor lock-in
The situation where lack of standard APIs and migration paths between cloud services makes switching prohibitively expensive, giving the provider disproportionate power over the customer
capacity planning becomes financial planning
In cloud environments, the operational challenge shifts from provisioning physical resources to managing metered billing — understanding spend, avoiding waste, and planning for resource quotas
systems thinking Mental Models & Understanding
Analyzing an entire system including human interactions and feedback loops, not just the computerized parts, to predict unintended consequences of data-driven decisions

Propositions

explicitWhether cloud services are actually cheaper than self-hosting depends on your existing operational expertise and the predictability of your workload, not on cloud marketing claimsconf: high
explicitdevelopedThe biggest downside of cloud services is loss of control — over features, uptime, debugging, pricing, and even geopolitical accessconf: high
extends ←Podcast: Increasing Users' Data Agency: From BlueSky's AT Protocol to the Local-First Software Movement · p9 — Local-first — the primary copy of data on the user's own device — is Kleppmann's architectural remedy to the 'loss of control' that DDIA names as the biggest downside of cloud services: provider shutdown or account lockout can no longer take your data away.
explicitcorroboratedCloud-native architecture (designed for cloud from the ground up) delivers materially better performance, failure recovery, and scalability than self-hosted software merely deployed on cloud VMsconf: high
extends ←Presentation: Parting the Clouds: The Rise of Disaggregated Systems · p1 — ddia-cloud asserts generally that cloud-native architecture outperforms lift-and-shift deployment; Demirbas gives the specific architectural mechanism (compute/storage disaggregation) that produces that performance gap.
supports ←Podcast: Increasing Users' Data Agency: From BlueSky's AT Protocol to the Local-First Software Movement · p1 — Kleppmann here restates, conversationally, the cloud-native shift that DDIA credits with better performance and failure recovery: storage decoupled from compute atop inherently replicated object stores rather than database-managed local-disk replication.
explicitcorroboratedSoftware engineers bear personal ethical responsibility for the consequences of the systems they build — delegating to 'the algorithm' does not discharge that responsibilityconf: high
supports ←How Do You Know That? · p10 — Both insist that delegating a decision to software or 'the algorithm' does not discharge the human responsibility behind it.
explicitMachine learning systems trained on historically biased data will learn and amplify that bias, not correct for itconf: high
explicitPredictive analytics systems create self-reinforcing feedback loops that can trap individuals in downward spirals (e.g., bad credit score leading to joblessness leading to worse credit score)conf: high
explicitcorroboratedThe data economy has created the greatest mass surveillance infrastructure in human history, exceeding what totalitarian regimes could have achievedconf: high
supports ←AI Surveillance and Social Progress · p10 — Schneier's 'no historical precursor — it dwarfs the FBI and East Germany's Stasi, and even Orwell's telescreen' directly supports the DDIA claim that the modern data/surveillance economy has built the greatest mass-surveillance infrastructure in history, exceeding what totalitarian regimes could achieve.
implicitcorroboratedThe distinction between corporate data collection for services and government surveillance for control is less meaningful than it appears, because both can be used for predictive decision-making that constrains people's livesconf: high
supports ←AI Surveillance and Social Progress · p3 — Schneier treats public and private surveillance as one fused capability and cites a corporate CEO (Oracle's Ellison) enabling state-style social control — supporting DDIA's claim that the corporate-vs-government surveillance distinction is less meaningful than it appears.
explicitMoral imagination — the ability to envision a future different from the past — is something only humans can provide, and it is required to prevent predictive systems from merely codifying historical discriminationconf: high
explicitThe cloud does not eliminate the need for operations — it transforms operations from machine-level administration into service integration, financial planning, and security managementconf: high
implicitThe voluntary nature of corporate surveillance is partly explained by the privilege of those who accept it — people with nothing to hide are those aligned with existing power structuresconf: high
explicitAlgorithmic pricing can reduce market competition by enabling tacit collusion between competitorsconf: high

Arguments

Cloud vs self-hosting is a nuanced tradeoff, not a one-way migration strength: high

Explicit premises:

Cloud is cheaper for variable/spiky workloads
Self-hosting is often cheaper for predictable workloads with existing expertise
Cloud eliminates machine-level operations but introduces new operational challenges (integration, cost management, quotas)
Cloud creates vendor lock-in risk, especially without standard APIs

Implicit premises:

Organizations have stable enough requirements to make meaningful long-term infrastructure commitments
The cost comparison is primarily economic — political, sovereignty, and security dimensions are secondary

Inferences:

The optimal strategy for most organizations is a hybrid approach, not all-cloud or all-self-hosted
As cloud APIs remain non-standard, the switching cost problem will worsen over time as more services are adopted

Engineers have a personal ethical responsibility for the data systems they build strength: high

Explicit premises:

Software systems have far-reaching societal consequences, both intended and unintended
Data is often about people — their behavior, identities, interests
Professional ethics codes exist (ACM) but are rarely enforced or discussed
Technology is not inherently good or bad; it depends on how it's used

Implicit premises:

Engineers have meaningful agency in what gets built — they are not merely executing orders
The market and employers will not spontaneously ensure ethical outcomes without individual engineer resistance
Ethics is not reducible to legal compliance

Inferences:

Engineers who focus only on technical excellence while ignoring consequences are being professionally negligent
The industry needs institutional mechanisms (not just individual conscience) to enforce ethical standards

Algorithmic decision-making can systematically harm individuals through bias, lack of accountability, and feedback loops strength: high

Explicit premises:

ML systems amplify bias present in training data
Protected traits correlate with non-protected features (e.g., postal code correlates with race)
Algorithmic decisions lack accountability — no clear party is responsible for errors
Feedback loops can trap individuals in downward spirals

Implicit premises:

Statistical accuracy at the population level does not justify harm to wrongly-classified individuals
The burden of proof should rest on the system deployer, not the individual harmed
Market incentives favor false negatives (rejecting good applicants) over false positives (accepting bad ones), systematically biasing toward exclusion

Inferences:

The combination of bias amplification and feedback loops means algorithmic systems can create new forms of structural inequality that are harder to detect and challenge than explicit discrimination
Individual recourse mechanisms are insufficient — systemic regulation is needed

We have voluntarily built a mass surveillance infrastructure that rivals or exceeds totalitarian systems strength: low

Explicit premises:

Internet-connected microphones exist in nearly every inhabited space
Location, social, financial, and health data collection is nearly unavoidable
People accept corporate surveillance because the benefits of digital technology are great
Sensor accuracy and analysis algorithms will only improve

Implicit premises:

Corporate data collection, once established, can be co-opted by state actors through legal or extralegal means
The 'voluntary' nature of consent is illusory when opting out means foregoing essential services
The current benign use of surveillance data does not constrain future use

Inferences:

The line between 'data-driven services' and 'surveillance' is one of intent, not of capability — and intent can change

Narratives

The thought experiment of replacing 'data' with 'surveillance' in common business phrases ('data-driven organization' becomes 'surveillance-driven organization')
→ Supports the proposition that corporate data collection is functionally equivalent to surveillance; forces the reader to see familiar practices as sinister
The credit score feedback loop: a good worker hit by misfortune gets a bad credit score, which prevents employment, which worsens the credit score further
→ Demonstrates that algorithmic feedback loops can turn temporary misfortune into permanent exclusion — supporting the argument that predictive systems create new forms of structural inequality
German gas stations introducing algorithmic pricing, which led to tacit collusion and higher consumer prices
→ Shows that algorithmic feedback loops can harm at a market level, not just individual level — algorithms can learn anti-competitive behavior without explicit programming
Martin Kleppmann's journey from DDIA 1st edition (2016) to 2nd edition (2026) — startup founder, LinkedIn, years of research, now with co-author Chris Riccomini
→ Establishes the authority and rigor of the source; implies that the ethical chapter is not an afterthought but the culmination of a decade of thinking about data systems
Totalitarian regimes dreaming of putting a microphone in every room — contrasted with our voluntary adoption of smartphones, smart TVs, and voice assistants
→ Supports the surveillance argument by making the historical comparison visceral — what was once the aspiration of dictators is now our daily reality, accepted willingly

Questions addressed

✅ When should an organization use cloud services versus self-hosting?
It depends on existing expertise, workload predictability, and control requirements. Cloud suits variable workloads and teams without deep ops experience; self-hosting suits predictable workloads with existing expertise. Hybrid approaches are common.
✅ How does cloud computing change system architecture at a technical level?
Cloud-native architecture separates storage from compute, builds higher-level services on lower-level ones (layering), and uses multitenancy. This differs fundamentally from merely running traditional software on cloud VMs.
✅ How does the cloud change the role of operations?
Operations shifts from machine-level administration to service integration, financial planning (metered billing replaces capacity planning), security management, and monitoring cross-service interactions.
⭕ What ethical responsibilities do software engineers have regarding data systems?
Engineers bear personal responsibility for consequences and must consider bias, accountability, feedback loops, and surveillance. But the text acknowledges ethics is 'not a checklist' — it's an ongoing process of reflection and dialogue.
⭕ How can we prevent algorithmic bias and feedback loops from causing harm?
The text identifies the problems clearly but offers only general directions: systems thinking, moral imagination, accountability mechanisms. Specific institutional or technical solutions are not provided.
⭕ How should engineers concretely exercise ethical judgment when employed by organizations with misaligned incentives?

Gaps

incompleteThe ethical argument establishes that engineers bear responsibility but offers no concrete mechanisms for exercising it. What should an engineer actually do when their employer deploys a biased algorithm? The text says ethics is not a checklist, but provides no alternative framework for action — leaving a gap between 'you should care' and 'here is how to act.'
Read next: Ruha Benjamin, 'Race After Technology'; Meredith Broussard, 'Artificial Unintelligence'; or practical frameworks like the Markkula Center for Applied Ethics' technology ethics toolkit
incompleteThe cloud tradeoffs section is entirely from the customer's perspective. The societal implications of cloud concentration — a small number of providers controlling critical infrastructure for most of the economy — are not examined. This is surprising given the ethical emphasis of Chapter 14.
Read next: Cory Doctorow's work on enshittification; EU Digital Markets Act analysis; Corey Quinn's 'Last Week in AWS' for cloud market dynamics
closed byPodcast: Increasing Users' Data Agency: From BlueSky's AT Protocol to the Local-First Software Movement — DDIA-cloud examines cloud concentration only from the customer's cost side and names no remedy for the resulting control/lock-in problem. This conversation supplies the structural response: open standard protocols (AT Protocol's guaranteed provider-switching; local-first's device-held primary copy synced over an open protocol) that make providers substitutable and reduce dependence on any single controller of critical infrastructure — the missing 'what to do about it' next to DDIA's diagnosis.
incompleteThe surveillance section does not engage with the argument that surveillance enables genuine safety benefits (e.g., fraud prevention, child safety, counter-terrorism). By not steelmanning the opposing position, the argument is weaker than it needs to be.
Read next: Bruce Schneier, 'Data and Goliath' — which addresses both sides more comprehensively
incompleteThe connection between Part 1 (cloud architecture) and Part 2 (ethics) is left implicit. Cloud concentration amplifies every ethical concern raised — bias at scale, surveillance infrastructure, accountability gaps — but this connection is never drawn. The two excerpts sit side by side without synthesis.
Read next: This may be addressed in the full book; the gap may be an artifact of excerpting
incompleteThe text mentions AI-generated code and formal methods as new topics in the 2nd edition but does not explore how AI code generation itself raises the ethical questions discussed in Chapter 14 — e.g., who is responsible when AI-generated code produces biased outcomes?
Read next: The full DDIA 2nd edition likely covers this; also see emerging literature on AI coding liability