Clay Shirky's Writings About the Internet
Economics and Culture, Media and Community, Open Source
PCs Are the Dark matter of the Internet

Premature definition is a danger for any movement. Once a definitive
label is applied to a new phenomenon, it invariably begins shaping --
and possibly distorting -- people's views. So it is with the current
revolution, where Napster, SETI@Home, and their cousins now seem to be
part of a larger and more coherent change in the nature of the
internet. There have been many attempts to describe this change in a
phrase -- decentralization, distributed computing -- but the label
that seems to have stuck is peer-to-peer. And now that peer-to-peer is
the name of the game, the rush is on to apply this definition both as
a litmus test and as a marketing tool.

This is leading to silliness of the predictable sort -- businesses
that have nothing in common with Napster, Gnutella, or Freeserve are
nevertheless re-inventing themselves as "peer-to-peer" companies,
applying the term like a fresh coat of paint over a tired business
model. Meanwhile, newly vigilant interpreters of the revolution are
now suggesting that Napster itself is not "truly peer-to-peer",
because it relies on a centralized server to host its song list.

It seems obvious, but bears repeating: definitions are only useful as
tools for sharpening one's perception of reality. If Napster isn't
peer-to-peer, then "peer-to-peer" is a bad description of what's
happening. Napster is the killer app for this revolution, and defining
it out of the club after the fact is like saying "Sure it might work
in practice, but it will never fly in theory."

No matter what you call it, what is happening is this: PCs, and in
particular their latent computing power, are for the first time being
integrated directly into the fabric of the internet.

PCs are the dark matter of the internet. Like the barely detectable
stuff that makes up most of the mass of the universe, PCs are
connected to the internet by the hundreds of millions but have very
little discernable effect on the whole, because they are largely
unused as anything other than dumb clients (and _expensive_ dumb
clients to boot.) From the point of view of most of the internet
industry, a PC is nothing more than a life-support system for a
browser and a place to store cookies.

PCs have been restricted to this expensive-but-dumb client mode for
many historical reasons -- slow CPUs, small disks, flakey OSs, slow
and intermittant connections, no permanent IP addresses -- but with
the steady growth in hardware quality, connectivity, and user base,
the PCs at the edges of the network now represent an astonishing and
untapped pool of computing power.

At a conservative estimate, the world's net-connected PCs host an
aggregate 10 billion Mhz of processing power and 10 thousand terabytes
of storage. And this calculation assumes 100 million PCs among the
net's 300 million users, with an average chip speed of 100 Mhz and an
average 100 Mb hard drive. And these numbers continue to climb --
today, sub-$2K PCs have an order of magnitude more processing power
and two orders of magnitude more storage than this assumed average.

This is the fuel powering the current revolution -- the latent
capabilities of PC hardware made newly accessible represent a huge,
untapped resource. No matter how it gets labelled (and peer-to-peer
seems likely to stick), the thing that software like the Gnutella file
sharing system and the Popular Power distributed computing network
have in common is an ability to harness this dark matter, the otherwise
underused hardware at the edges of the net.

Note though that this isn't just "Return of the PC", because in these
new models, PCs aren't just personal computers, they're promiscious
computers, hosting data the rest of the world has access to, a la
Napster, and sometimes even hosting calculations that are of no use to
the PC's owner at all, like Popular Powers influenza virus
simulations. Furthermore, the PCs themselves are being disaggregated
-- Popular Power will take as much CPU time as it can get but needs
practically no storage, while Gnutella needs vast amounts of disk
space but almost no CPU time. And neither kind of business
particularly needs the operating system -- since the important
connection is often with the network rather than the local user, Intel
and Seagate matter more to the peer-to-peer companies than do
Microsoft or Apple.

Its early days yet for this architectural shift, and the danger of the
peer-to-peer label is that it may actually obscure the real
engineering changes afoot. With improvements in hardware, connectivity
and sheer numbers still mounting rapidly, anyone who can figure how to
light up the internet's dark matter gains access to a large and
growing pool of computing resources, even if some of the functions are
centralized (again, like Napster or Popular Power.)

Its still too soon to see who the major players will be, but don't
place any bets on people or companies reflexively using the
peer-to-peer label. Bet instead on the people figuring out how to
leverage the underused PC hardware, because the actual engineering
challenges in taking advantage of the world's PCs matters more -- and
will create more value -- than merely taking on the theoretical
challenges of peer-to-peer architecture.

Write with questions or comments.

Mail a copy of this essay:

Enter the email address of the recipient. Multiple addresses should be separated by commas.

Add your own message(optional):

Your name:(optional)

Note: Your name, and your recipient's email address, will only be used to transfer this article, and will not be stored or used for any other purpose.

Send the article URL only
Send the article as HTML
Send the article as plain text Clay Shirky's Writings About the Internet
Economics and Culture, Media and Community, Open Source