Thriving with complexity using open sociotechnical systems design

Trond Hjorteland · article · 2022-01-05 source ↗ Reader 🖥️

Ashby's Law (Requisite Variety)Coordination Cost Hierarchy & Org Charts Sociotechnical Architecture Team Autonomy Team Topologies

Software organizations should undergo a radical, participatively-designed structural transition from DP1 (bureaucratic, redundancy-of-parts, externally-coordinated) to DP2 (self-managing, redundancy-of-functions, internally-coordinated) sociotechnical design — the empirically-grounded Open Systems Theory tradition that agile, product, and 'team-first' movements have been independently and incompletely rediscovering — because only DP2 generates the variety needed to thrive in today's turbulent technical and social environment, and because that change must be structural and self-designed, not a culture or values campaign layered on top of unchanged structure.

practical · mixed — Grounds a practical prescription (ICT organizations should undergo a radical structural transition) in a theoretical tradition (Tavistock Institute / Open Systems Theory) the author treats as settled scientific research rather than opinion; expository history leading to an argumentative conclusion. Originally published at InfoQ, cross-posted to LinkedIn.

Structure

Part	Role
Framing: the ICT industry has mastered the technical side of sociotechnical systems but neglects the social side, despite claiming people/talent as its core asset	context
History: the Tavistock Institute coal-mine studies that discovered semi-autonomous work groups, and the parallel to agile/XP's rebellion against waterfall	evidence
Theoretical foundation: Emery & Trist's Open Systems Theory, the four causal textures of organizational environments (placid random/clustered, distributed reactive, turbulent field), and today's ICT environment as Type IV turbulent	evidence/context
The two genotypical design principles, DP1 (bureaucratic) and DP2 (self-managing), and the argument that only DP2 increases variety to match a turbulent environment	core argument
Diagnosis of failed or partial transitions: culture-first change doesn't work; mixing DP1 and DP2 ('mixed mode') is a maladaptation to maladaptation; industrialized/scaled Agile exemplifies this	evidence/counter-case
Prescription: complete, participatively-designed structural change to a non-dominant hierarchy of functions, business-capability-owning teams, and the six psychological criteria for productive work (now codified in Norwegian law)	conclusion
Summary and acknowledgment of the Tavistock/OST researchers	conclusion

Key terms

DP1 / DP2 (genotypical design principles)
Emery's two fundamental organizational design principles, distinguished by how redundancy is achieved: DP1 gets redundancy from interchangeable, simple parts coordinated by a level above the work (classic bureaucracy, maximum division of labor); DP2 gets redundancy from multi-skilled people who can cover for each other, with coordination and control held by the group itself (semi-autonomous, self-managing teams).
causal texture / Type I-IV environments
Emery & Trist's four ideal-typical organizational environments of increasing complexity — placid random, placid clustered, distributed reactive, and turbulent field — used as an evolutionary staging of how turbulent today's ICT environment has become.
non-dominant hierarchy of functions
The organizational structure the author argues DP2 produces: hierarchy persists (functions still need relating to each other across levels) but without personal dominance — management functions are distributed through self-managing groups rather than concentrated in managers who control subordinates.
the six criteria (Emery & Thorsrud)
The psychological requirements of productive, satisfying work validated during the Norwegian Industrial Democracy Program: reasonably demanding work, opportunity to learn, an area of decision-making, social support, relating work to social life, and a job leading to a desirable future — now measured in Participative Design Workshops and codified in Norwegian law.

Propositions

explicitSoftware development is a fundamentally sociotechnical enterprise in which the social and technical systems are not separable, so they must be jointly optimized rather than optimized independently.conf: high
explicitOptimizing the technical side of a sociotechnical system in isolation produces a dehumanizing work environment, while optimizing the social side in isolation prevents the technology from reaching its potential.conf: high
explicitToday's ICT industry operates in a 'turbulent field' (Type IV) environment — the most complex of Emery & Trist's four causal textures — driven by internet scale, machine-to-machine systems, and AI.conf: medium
explicitTwo genotypical organizational design principles exist — DP1 (redundancy of parts, coordination located above the work) and DP2 (redundancy of functions, coordination held by the group) — and most real organizations are neither purely one nor the other.conf: high
explicitDP2 structures increase organizational variety (adaptive capacity) while DP1 structures decrease it, so matching a turbulent (Type IV) environment requires a DP2 structure.conf: medium
explicitAttempting to change an organization's culture or values first, independent of structural change, has repeatedly failed to produce real change; behavior and structural change must come first, with values and attitudes following.conf: high
explicitA genuine DP1-to-DP2 transition must be a complete, organization-wide structural change, not one confined to the 'shop floor' level, because partial change leaves an unstable, maladaptive mix of DP1 and DP2 that is worse than either pure mode.conf: high
extendsTeam Topologies: Organizing Business and Technology Teams for Fast Flow · p15 — Hjorteland's DP1/DP2 mixed-mode analysis supplies the deeper theoretical mechanism behind Team Topologies' more pragmatic claim that org-chart/matrix-based structuring is counterproductive — org-chart reliance is a textbook DP1 remnant that destabilizes an attempted DP2-style (team-first) transition.
explicitIndustrialized, scaled Agile frameworks, applied to accommodate an existing bureaucracy, exemplify the DP1/DP2 mixed-mode failure, making things worse for both the bureaucratic and the agile-team sides.conf: medium
explicitAgile, product-organization, Spotify-model, Holacracy, Dialogic OD, and 'teal organization' movements are all independently seeking a DP2-like structure without anchoring their approach in Open Systems Theory.conf: medium
explicitcontestedA genuinely transformed team should own a complete business capability end-to-end, with full authority such that no one outside the team has a say in how the work is done, because everyone needed is already a team member.conf: high
supportsTeam Topologies: Organizing Business and Technology Teams for Fast Flow · p23 — Hjorteland's fully-authoritative, business-capability-owning team (nobody outside has a say) independently arrives, from the Open-Systems-Theory self-managing-group lineage, at treating the team as the atomic, self-sufficient unit of ownership — the same premise Team Topologies is flagged as implicitly assuming.
contradictsTeam Topologies: Organizing Business and Technology Teams for Fast Flow · p16 — Team Topologies argues architecture shouldn't be designed in isolation from teams, but still envisions topology as something leadership designs and evolves (reverse Conway maneuver, sensing and adapting); Hjorteland insists the design itself must be done participatively by the people who will implement it, treating any externally-designed restructuring — however team-aware — as reproducing a DP1 remnant of coordination located above the work.
explicitA well-resourced, cross-functional 'product team' with apparent authority is often, underneath, still a temporary structure embedded in a surrounding bureaucratic (DP1) organization — e.g. governed by PMOs or an ordering-supplier divide between IT and the business — as Spotify's own experience showed.conf: medium
extendsSociotechnical Architecture as Enabler of Product Thinking · p2 — da Silva's point that teams need the right scope/capacity (topology) to actually build the intended product is given a sharper, falsifiable theoretical edge by Hjorteland's Spotify example — apparent capacity/authority can mask an underlying DP1 bureaucratic structure that undermines it.
explicitProductive, satisfying work requires six psychological conditions — adequately demanding work, opportunity to learn, an area of decision-making, social support, relating work to social life, and a desirable future — now codified in Norwegian law (Work Environment Act §4.2).conf: high
explicitA structural change should be designed by the people who will have to implement it, not imposed on them from outside.conf: high
implicitThe DP1/DP2 theoretical apparatus, developed from mid-20th-century coal-mining and mechanical-processing industries, transfers cleanly to knowledge work in software organizations.conf: medium

Arguments

ICT organizations must undergo radical, complete DP1-to-DP2 structural change — not incremental culture change or partial local change — to thrive in a turbulent environment. strength: medium

Explicit premises:

Technical and social systems must be jointly optimized; neither works in isolation.
Culture/values-first change has repeatedly failed to produce real change.
DP2 increases organizational variety while DP1 decreases it.
Partial change produces an unstable DP1/DP2 mixed mode that is worse than either pure mode.

Implicit premises:

Matching environmental turbulence requires matching or exceeding it with internal organizational variety (an application of Ashby's law of requisite variety that the author invokes without naming it).
The only way to avoid the destabilizing mixed-mode trap is to change everything at once, rather than there being a viable staged or incremental path to full DP2.

Inferences:

If the mixed-mode trap is real, then well-intentioned but partial transformations (a single empowered 'product team' inside an unchanged bureaucracy) may be actively counterproductive rather than simply insufficient — a stronger and more falsifiable claim than it first appears.

Agile, product-org, and 'flat'/teal movements are reinventing DP2 incompletely because they lack Open Systems Theory's diagnostic and design apparatus. strength: medium