# The Internet Was Designed With a Narrow Waist

Source: https://www.oilshell.org/blog/2022/02/diagrams.html

## Summary

The article argues that the Internet's "narrow waist" concept, a shared abstraction that solves an interoperability problem, avoids O(M × N) code explosions in favor of O(M + N) code, and that software design should deliberately engineer such waists more often. It cites IP as the Internet's narrow waist, which allows applications on top to run over many physical transports below, and notes that web authors never had to change anything as the author moved across dial-up, Ethernet, DSL, wireless and cell networks. The author says most software code is glue and that byte streams and text are essential narrow waists, while the Unix and Web designs are equally inconvenient for every programmer. The post also traces the idea's history, mentioning a 1979 attribution to John Aschenbrenner, a 1994 National Research Council book, and Jon Postel's late-1970s hourglass diagram for IP.

## Article

| blog | oilshell.org

2022-02-26 (Last updated 2025-01-03)

In December, I reviewed eight 2021 posts on #software-architecture. This post goes deeper in that direction, showing diagrams of the narrow waist idea.

When you hear the term, you should picture an hourglass with M things on one side, N on the other, and an important concept in the middle. Recap:

A narrow waist is concept, interface, or protocol that solves an interoperability problem.

It avoids O(M × N) code explosions, letting us write O(M + N) amounts of code instead.

This is a big deal in practice! Most code is glue, but it doesn't have to be this way.

Byte streams and text are essential narrow waists in software, particularly in #distributed-systems.

IP: the Narrow Waist of Internet Architecture

Let's start with 2 diagrams I found about the architecture of the Internet itself.

The "IP" in TCP/IP stands for the Internet Protocol. It was developed by Vint Cerf and Bob Kahn, who described it in 1974. It allows interoperability between applications on the top and physical transports on the bottom.

Source: 2001 Presentation by Steve Deering (PDF)

For example, I started using the web in the 90's with a PC connected to a modem, connected to a phone line. (Guitar tablature was a type of text that traveled well over slow networks!)

In the next few decades, I used the Internet over Ethernet at college, over wires meant for cable television, over DSL from the phone company, with wireless routers, and with cell phones.

At no point did a web page author have to do anything to make this work! This is because web browsers and servers both use the Internet abstraction. (I'm mixing up some layers, but my ignorance is part of the point — it just worked.)

Source: 2019 Lecture Notes by John Kubiatowicz (PDF)

So in networking, the idea of a narrow waist is used and taught explicitly.

But we don't pay enough attention to this idea in software — particularly in the modern "cloud". In the next few posts, I'll review some Unix fallacies, i.e. local complaints which are ignorant of the large scale architecture.

Pre-Internet M × N Confusion

The first e-mail applications did not use the Internet; they were directly coupled to a local, physical network. You could send messages to your coworker, but not to your family across the ocean.

Moreover, you had to rewrite or port the app if you wanted to use different hardware, e.g. a wireless network instead of wired. You also had to rewrite or port your file transfer application.

So instead of just one e-mail application, you had N e-mail apps that worked on N networks. And this was true for each of M applications, giving O(M × N) amounts of code.

These days, each application uses the Internet, and each local network supports the Internet, which is an O(M + N) amount of code.

This is similar to rewriting assembly language programs when you got a new computer, which was the status quo before standard instruction set architectures were invented. It's hard to imagine now, but in the 60's, you would throw out your software after buying a new computer from the same vendor! (Back then, only large organizations owned computers.)

Fred Brooks' Turing Award cites seminal work in this area, e.g. IBM's System/360 line of mainframes.

2/28 Update: Reader CheCheDaWaff reminded me that The Dream Machine by Waldrop covers the development of the Internet in great detail. I read it in 2018 and mentioned it in this blog!

These Abstractions Can Be Designed

My point is that we take narrow waists for granted, like the air. But real people put a lot of effort into designing and inventing them!

I think we're still writing distributed systems the "wrong" way, with O(M × N) amounts of labor. This blog post from August goes deeper into that:

A major point is that crosscutting concerns like authentication, authorization, metrics, monitoring, alerting, and deployment are done for each service and for each programming language, leading to code explosions (and language wars). It cites the excellent Unix vs. Google video, which makes an analogy to Multics. (This discussion is similar but not identical to Kubernetes is Our Generation's Multics.)

With hindsight, this way of building systems now will seem as silly as throwing out your software when you buy a new computer. We should explicitly think about new narrow waists and fix old ones, like the shell.

Shell and Distributed Systems

I made two diagrams to show where this line of thought is going. Upcoming posts will substantiate and elaborate on them.

Bytes / Flat Files

This diagram shows that using the common abstraction of byte streams allows essential generic / polymorphic operations, avoiding O(M × N) amounts of code.

Slogans:

Most code is glue! Especially in distributed systems.

By glue, I mean copying and simple data transformations like parsing and serialization.

Shell is the language of byte streams and coarse-grained composition.

It's also the language of heterogeneity and diversity.

Text: Narrow Waist of Unix Architecture

An important special case of byte streams is human-readable text. When you represent data with plain text, you get useful operations and tools for free.

This isn't a theoretical issue; it has big consequences. This economy of engineering is why Unix and the Internet have been so successful.

It's also why programmers of all types lament the pervasiveness of text. I understand why: both #parsing and serialization are laborious, and have associated security problems. (String hygiene is still in the blog queue).

But I claim that programmers disregard system issues in favor of local convenience. And that we use poor abstractions provided by interested vendors. These abstractions don't interoperate, leading to bigger problems than parsing.

Slogan: Unix and the Web Are Equally Inconvenient for Every Programmer, And That's Good.

(Help wanted: I made these SVG diagrams with https://diagrams.net, and I want to illustrate more narrow waists. So it would be nice to generate them with a parameterized program. Can a reader get me started, e.g. with a pikchr or asymptote script?)

What's Next?

I started these drafts:

Characteristics of Narrow Waists. Let's define the idea more precisely and give examples. The most important point is that narrow waists are necessarily a compromise.

They make systems possible and feasible, not optimal. They can be good or bad.

Bytes And Text Are Essential Narrow Waists. Why are these two diagrams interesting and important? I give examples from real systems.

Slogans and Comments: Software Architecture. Architecture is abstract and difficult to talk about. One good way to explain the diagrams is with more slogans, and #comments from discussion threads.

While I write these articles, you may want to review Retrospective: Software Architecture, published December. It turned into a pretty good post with concrete claims and project updates!

Why Write About Narrow Waists?

My main goal is to "re-justify" the effort put into OSH, and the design of the new Oil language. This project has a difficult design and implementation because it sits at the center of many strong forces in architecture.

But I also think narrow waists are worth describing and thinking about independent of Oil. It's interesting and underexplored idea. More claims:

We should move toward explicit engineering of narrow waists, rather than letting them evolve haphazardly.

We should also be cognizant of "game theory" between billion dollar companies. This powerful force explains much of our shared computing infrastructure:

I quoted JavaScript: The First 20 Years (video, PDF) in the backlog section Don't Break X. It talks about "web browser game theory" as an overwhelming force in the development of the language.

It explains X windows, which I didn't understand until very recently.

Appendices

What's the History of This Idea?

I suppose there are two closely related but distinct claims here:

Narrow waists enable interoperability, e.g. in networking, but also in software.

Narrow waists avoid writing O(M × N) amounts of code. They make large systems feasible to implement.

I'd like to better understand the history of this idea. (You could even call it a meta-idea — an idea about ideas.) I asked for references in previous blog posts but I don't recall any noteworthy responses.

I think it's complicated by the varied terminology:

Narrow waist, Thin waist

Hourglass model

Distinguished layer, Spanning layer, Universal layer

The earliest writing I can find on it is a 1994 government publication, via this Berkeley article:

The Evolution of Internet Services (PDF) by Katz, Brewer, McCanne.

Was this ever published? It doesn't have a date or any citations in Google Scholar.

Figure 1: Kleinrock's Narrow Waist Model.

Citation 1: L. Kleinrock, et al., Realizing the Information Future: The Internet and Beyond, National Research Council, National Academy Press, Washington, DC, 1994.

Mirror of Large PDF (9 MB)

So I would like help finding references to narrow waists, interoperability, and O(M × N) code explosions! The idea is central in compilers (IRs) and operating systems as well, but I don't see a lot of explicit writing about it.

Here are some more links I dug up:

Narrow Waists: Taming Abstraction Explosion (2019). This is article covers a lot of the same ground! The idea is very general. But it's also lacking history, and it's closer to a set of notes.

https://a16z.com/2020/04/24/a16z-podcast-the-narrow-waist-of-the-internet/. Making an analogy from the Internet to a blockchain future.

(If anyone cares, my current view is that blockchains don't work very well, and the incentives have ruined some interesting projects. But I think they're possible to "steel man" and have non-zero value. I mostly care about software that works, like shell, Python, git, and BitTorrent.)

Addendum: February 27th

After publishing this article, I found an earlier source for the hourglass / narrow waist idea in the networking domain.

RFC 3234 - Middleboxes: Taxonomy and Issues (2002) references the 1994 National Resource Council book, but says that the "hourglass" metaphor was first used by John Aschenbrenner in 1979, with reference to the ISO Open Systems Interconnection model. (No citation.)

Protocol Layering and Internet Policy (2013, PDF)

Note 215: The hourglass model is usually attributed to John Aschenbrenner in connection with the OSI layered stack. See Jean Bartik, OSI: From Model to Prototype as Commerce Tries to Keep Pace, DATA COMM., Mar. 1984, at 307, 314-15

Addendum: February 28th

Reader Caleb Malchik pointed out a more convincing origin story for the narrow waist idea — both the analogy and the technical work itself!

The first 10 minutes of this talk by Van Jacobson are amazing:

Summary:

In the late 1970's, Jon Postel argued for IP to be separated from TCP, and he drew the hourglass shape.

Internet designers and engineers including Postel and Jacobson spent 5 years taking features out of IP, and we've basically been using this minimal design since then.

Postel was vigilant about the simplicity of IP for his 20 years as RFC editor.

I wasn't aware of this, but there are big echoes of Ken Thompson's design philosophy, which I covered in Unix Shell: History and Trivia.

Jacobson also mentions the explicit desire for vendors to lock in their customers.

Aside: In 2006, I attended a talk by Van Jacobson on Named Data Networking at Google. It proposes a new and explicitly designed narrow waist for the Internet. This talk is probably a major reason this idea stuck with me for the last 14 years!

More on Networking

Here are more links that show that the narrow waist idea is explicit in networking.

HTTP: An Evolvable Narrow Waist for the Future Internet (2012)

This paper convinced me that the idea is important. It has engineering consequences and predictive power! The fact that the narrow waist moves and evolves is fascinating. (Counterpoint: Fuck It, Overload HTTP and comments. This article has some substantial misunderstandings that I could write about later. This is another argument about local convenience vs. system integrity.)

Another article about a moving waist:

Internet Draft: UDP and TCP as the New Waist of the Internet Hourglass (2008)

One of the fundamental design principles of the Internet is that IP [is the narrow waist]. The preponderence of NATs and firewalls in the Internet has changed this reality, such that UDP and TCP are now the waist of the hourglass.

My main goal isn't to explore or debate the architecture of the Internet, but to make an analogy, and to make claims about how we write software.

These articles make use of the model, although I see their main ideas as experimental:

On the Hourglass Model (CACM 2019). It introduces a theoretical model, with conjectures about the "deployability" of new network technology.

How the Internet architecture got its hourglass shape and what that means for the future (2011). A simulation concerning the adoption of new network technology.