As a VC at Icon Ventures and a twenty year veteran of productizing and marketing high tech for VMware, Netscape and others, I've always been fascinated by how new technologies emerge and come to market. One of the major artifacts that tries to capture the state of our market and industry each year is the annual Gartner Hype Cycle - which I always read with interest. Just last month, I had an interesting thought: "Has anyone gone back and done a retrospective of Gartner Hype Cycles - because I'd totally read that article". A quick Google search didn't turn up anything useful, so I decided I'd do the work and write it myself. This article is the result.

As most of you know, the Gartner Hype Cycle for Emerging Technologies is practically an institution in high tech. First published in 1995, the Hype Cycle proposed a standard adoption model for new technologies. In this model, technologies all go through a process of :

  1. Emergence: "The Technology Trigger"
  2. Excessive enthusiasm: "The Peak of Inflated Expectations"
  3. Excessive disappointment : "The Trough of Disillusionment"
  4. Gradual, practical adoption: "The Slope of Enlightenment" and "The Plateau of Productivity"

By way of illustration, below is the first Hype Cycle - from 1995. And it's truly a fascinating historical document. Some of its technologies that have become so ubiquitous, that they're now background noise (Object-Oriented Programming). Some technologies have simply disappeared from public consciousness (Emergent Computing). Still others are technologies that we thought were almost baked but actually took decades longer to reach full maturity (Speech Recognition).   

The most hyped technology in 1995 was Intelligent Agents. Two years later, Office 97 introduced Clippy, the enthusiastic, but incompetent assistant which was so poorly received that it effectively killed off the idea for a generation. Today, twenty years later, we’re once again trying to build intelligent assistants, although now we call them Chatbots, and the core tech - contextual reasoning in a broad domain - is still a hard problem.

I think of the Gartner Hype Cycle as a Hero's Journey for technologies. And just like the hero's journey, the Hype Cycle is a compelling narrative structure. When we consider many of the technologies in use today, we tend to recall that they were overhyped when they first arrived, but eventually found their way to mainstream usage. But ... is that really how technologies emerge and gain adoption? After analyzing every Gartner Hype Cycle for Emerging Technology from 2000 to 2016 - all seventeen years of the post dotcom era - I’ve come to believe that the median technology doesn’t obey the Hype Cycle. We only think it does because when we recollect how technologies emerge, we're subject to cognitive biases that distort our recollection of the past:  

For example, Emergent Computation - the earliest technology on the 1995 Hype Cycle up above - is a great illustration of survivor bias. (Emergent computing, by the way, is computing based on distributed evolutionary algorithms - a kind of cousin to neural network based machine learning. (I had to look it up)). Today if I asked 20 Silicon Valley technologists to name which technologies succeeded and failed since 1995, I think I can guarantee that no-one would name Emergent Computation. But yet, there it is in the 1995 Hype Cycle, important enough to merit one of just ten slots in that year's listing.

(And incidentally, my intention is not to call out Gartner’s accuracy as a firm specifically. With some notable exceptions, such as the technology terms that Gartner coins itself, I think of the Gartner Hype Cycle as mostly a reflection of industry consensus.) 

But our inability to remember the past in proper context is not the only lesson from taking a deep dive into Gartner's past Hype Cycles. After analyzing every year from 2000 on, I think I can say with confidence that we are simply not very good at predicting the future. I've learned that lesson and seven more from my deep dive into the data. Read on for the details:

Lesson 1. We're terrible at making predictions. Especially about the future.

No surprise to any experienced Silicon Valley hands. In general, we're bad at making predictions. Out of the more than 200 unique technologies that have ever appeared on a Gartner Hype Cycle for Emerging Technology, just a handful of technologies - Cloud Computing, 3D Printing, Natural Language Search, Electronic Ink - have been identified early and traveled even somewhat predictably through a Hype Cycle from start to finish. 

Lesson 2. An alarming number of technology trends are flashes in the pan.

High tech has a pronounced propensity for getting extremely excited about a technology for a very short period of time. Out of the more than 200 technologies ever listed, just over 50 individual technologies appear for just a single year on the Hype Cycle - never to reappear again. Yes, it’s true that many of the Hype Cycle’s one hit wonders survive today, enjoying minor success or mindshare: Crowdsourcing - 2013, HTML5 - 2012, BYOD - 2012, Podcasting -2005. But it’s equally true that past Hype Cycles contain a long list of technologies that seem as poorly considered as parachute pants or perms. Just some of the one-hit wonders: Social TV (2011), Truth Verification (2004), Folksonomies (2006) and Expertise Location (2007). 

Lesson 3. Lots of technologies just die. Period.

Closely related to the last lesson, the last two decades are a graveyard of technologies that died permanent and premature deaths. By my rough count, an additional 20% of all technologies that were tracked for multiple years on the Hype Cycle became obsolete before reaching any kind of mainstream success. Some of the most notable technologies that appeared on multiple hype cycles but ultimately died include:

And those are just five of the many technologies that went to the graveyard without passing Go or collecting $200. There are more (Broadband Over Power Lines!). See the Appendix below for all the technologies that didn’t make it. Sic transit gloria mundi.

Lesson 4: The technical insight is often correct, but the implementation isn't there

I was often struck by how many times the Hype Cycle had an insight that was essentially correct, but the technology or the market just wasn’t ready yet. Some of the best examples:

Lesson 5: We've been working on a few core technical problems for decades

There are a number of core technologies that appear again and again in different guises over the years in Hype Cycles, sometimes under multiple aliases. Each reincarnation makes progress and leaves lessons for its successors without really breaking through. These are the technical marathons of the Hype Cycle.

Lesson 6: Some technologies keep receding into the future

There are some notable technologies that recur on the Hype Cycle and every time they appear they seem equally scifi. Although at some point, I'm sure they will not. The most notable are:

Lesson 7: Lots of technologies make progress when no-one is looking

Look at enough Hype Cycles and you can see a pattern where many technologies make steady and sometimes breakthrough progress after they're considered played out. Like machine learning in the aughts, many technologies are doggedly moved forward by researchers, startups and large tech companies when their previous generation is widely seen as having failed. A couple of my favorite examples:

For technologists looking for out-of-favor technologies that may be biding their time before their next breakthrough, past Hype Cycles can be a fruitful source of ideas. Some of the most intriguing technologies that in my humble opinion, may be due for their second or third round of visibility:

Lesson 8: Many major technologies flew under the Hype Cycle radar

One final lesson. It's remarkable the number of major technologies from the last 20 years that were either identified late or simply never appeared on a Hype Cycle. In technology, so many things that look trivial or transitory turn out to be the foundation of the next generation of business and consumer platforms. A brief list of the technologies that should clearly have been listed as important emerging technologies since the dot com wave:

Those are just a few of the major technology trends that never surfaced as part of an Emerging Technology Hype Cycle - although many of them appear on the various functional, vertical and specialist Hype Cycles that Gartner has produced in ever increasing diversity over the last decade.

The Lessons About The Lessons

What I take away from my analysis of these Hype Cycles is not just how difficult it is to make predictions, and how much wasted effort goes into technologies that doesn't tend to work, but also how exciting and wondrous is the progress that we've made in technology. The labor of the last two decades has given rise to an Age of Wonders: of self-driving cars, computers that almost understand us, and unfathomable scales of data analysis. More than ever, I feel personally privileged to work among and invest in the teams creating our technological future.

Update: If you liked this post, you might also enjoy my deep dive into the Magic Quadrant - looking at 16 years of data for the Business Intelligence category.

---------------------------------------------------------------------------------------------------------

Appendix: The Raw Data Dump

For my analysis, I tracked down every Hype Cycle for Emerging Technologies published by Gartner between 1995 and 2016, and created a list of all individual entries by year. I decided to discard 1995 through 1999because the dotcom technology categories seemed mostly irrelevant to our current technology conversations.

Next, I merged multiple terms into a single category if I felt that the difference between the two terms was minor. For example, I decided that "Personal Fuel Cells" and "Micro Fuel Cells" really referred to the same thing. As another example, I decided that Context Delivery Architectures, Context Brokering and Context Services all centered around the idea of Context abstraction, so I treated that as a single term. I'm very aware that I may have merged over-aggressively, but some of the more obscure and older terms are poorly documented on the web, so I'd be happy to receive feedback on how I could improve here.

Next, in order to track movement of each technology over multiple Hype Cycles, I coded each entry by where the term appeared. To code position, I came up with an eight point scale for Hype Cycle position, from stage 1 (earliest) to stage 8 (mature). My general categorization of the stages can be seen in the figure below.

Lastly, I color coded each entry from light red (stage 1), to deep red (stage 3) all the way to green (stage 8). And using Excel's value based color coding, I interpolated those colors across the range. What this gives is what I think as a nicely efficient 10,000 foot view of Hype Cycles from 2000 to 2016.

Below is the resulting database of Hype Cycle entries sorted by year of first appearance. For example, the first entry - Synthetic Characters (analagous to our current chatbots) - first appeared at Stage 2 in the 2000 Hype Cycle, appeared again in the 2001 Hype Cycle at the same position, and then never appeared thereafter.

Google Spreadsheet Link: https://docs.google.com/spreadsheets/d/1NkC0g60q-6w72nksayvdfzCT5oOmBy97XBCGw-tW1p8/edit?usp=sharing

'If you made it all the way down here, thanks for reading - I hope you find the retrospective useful. Comments welcome!

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.