0:00
49:40

How to Get Ideas and How to Measure - Stewart Butterfield & Adam D'Angelo

  • Sam Altman
  • Stewart B.
  • Speaker 3
  • Adam D.
  • Speaker 5
  • Speaker 6
  • Speaker 7
  • Speaker 8
  • Speaker 9
  • Speaker 10
  • Speaker 11
Sam Altman:
All right. Good afternoon. Today we have two different speakers. We have Stewart, the founder  and CEO of Slack and then we have Adam, the founder and CEO of Quora.  I'm going to interview Stewart. We're gonna talk about ideas and how to find good  ones and then Adam is going to talk about metrics, why they're important and what  to measure. Stewart, thank you for coming and spending some time with us. Stewart is  the found of Flickr and Slack, and what I didn't know until very recently is  that he tried to be the founder twice of Massively Multiplayer Online Games and they  turned into these two iconic companies. So I wanted to talk about just the process  of coming up with the idea for the game and then the ideas for Flickr  and Slack. 
Stewart B.:
All right. So, yes, twice we set out trying to build a web-based massively multiplayer  game and the origin of that was all the way back in 1992, I arrived  at college and I got my account on the school's Unix machine and was exposed  to the internet for the first time and it was kind of mind blowing experience.  This is about a year before the web became popular, so it was IRC, NewsNet,  Talk, email and I was really fascinated with the idea that people could find their  community, irrespective of what that community was. If you were a breast cancer survivor or  model train enthusiast or whatever, if you're interested in some, as I was, obscure 19th  century biologists, then you can find all that stuff. And the idea of a game  that lived online like that and had the game-like aspects or play as a pretext  for social interaction seemed really interesting. I'm not going to go into too long of  an explanation, because it turns out it's actually like a terrible, terrible idea and ...  At least as a commercial business. It might have been fun as a hobby. I  think it could have been the kind of business that sustained the number of employees  required to operate it, like a restaurant or something like that, but, when in 2009  we started a company that ended up making Slack, we raised $17.5 million in venture  capital and it was never going to become the kind of business that would justify  that investment. The first time was a little bit different. 2002 was after the dotcom  crash, it was after Worldcom and Enron accounting scandals, it was after 9/11 and the  NASDAQ was down 80%, S&P500 was down 65% from its peak. No one wanted to  invest in internet facing stuff, so it was very hard to tell whether this was  a good idea or not, because there's no signal from the market that said, "This  is an investible idea," because nothing was investible at that time. We raised a little  bit of friends and family money, we spent about a year or so building a  prototype. The prototype was well received, but it was going to take us another year  and a half to finish it and we literally couldn't raise any money at all.  We got to the point where the one person on the team who had kids  was the only person who was getting paid and then we decided to try to  find something else we could do that would make use of the technology that we'd  already developed, but that would be quicker to get to market. And that thing was  Flickr, but it was also a bad idea at first. Because we had developed this  game client and we had a messaging server on the backend, we just repurposed the  game client as a photo sharing client. In the game, you had a bunch of  slots along the bottom of the screen for inventory. Those became slots into which you  could fit photos. And because the game was all realtime interaction, the photo sharing was  all realtime interaction. Which meant that you had to be online at the same time  as someone you cared about in order to use it and while it demoed really  well, it was amazing technology especially for at the time we started on that, maybe  late 2003, it was a terrible product because you didn't care about any of the  people who were actually online. That morphed really quickly and the experience making Slack was  a little bit different. We had all of the resources we could ever ask for.  We had more than enough money because we had money left over. We decided to  abandon the idea and we had AWS, which we didn't have before, hardware was effectively  free, internet connections were much better, there were many more people online, hardware was better.  None of those external conditions was the reason this wasn't as Massively Multiplayer Game on  the web wasn't a good idea. It was just not a good idea because there  wasn't a big enough market for it. 
Sam Altman:
And why was Slack a good idea? How did you decide that Slack was looking  to build? 
Stewart B.:
Well, we started developing the system for internal communication that was based on IRC. And  those of you who had used IRC and also use Slack can recognize that Slack  is IRC with all of the things that were missing when it was originally designed  in 1989. I think, this is post facto analysis so it's hard to know if  this really true or not, but I think that the reason that Slack has been  so successful is because we had a three and a half year long design process  where we weren't conscious that we were designing it. So, we started using IRC. IRC  has many deficiencies, among them you can't store and forward messages. So if I wanted  to send you a message and you're not connected to the server at the moment  I want to send it, I just can't send you a message. So the first  thing we built was a bot to log all the messages and put them in  an archive. And once we had that, we had messages in our database and we  thought search would be useful so we built search for it. And this was 2009,  there were no good IRC clients for iPhones and so we built an HTML5 front-end  for the archives in Safari and once we had that, we wanted to be able  to post. And just like on and on and on. But, in the normal process  of software development, there's a huge amount of ego and speculation. So, when you're arguing  with someone whether this feature is a good idea or not, if you can get  to the point of enlightenment and a Buddha sense, then maybe you can do that  without any ego, but the practical reality is people are very ego-latent in those discussions  and there's a lot of speculation. Like, "I think this could be really valuable for  someone," whereas when we were building a system that was kind of the proto-Slack that  was the inspiration for Slack, it was whatever problem is the most irritating, like something  that we just absolutely can't stand, we would address that with a minimum amount of  effort and then go back to what we're supposed to be doing and that kind  of let it cook for several months and we had 45 people working on it,  working on the game, so we had a lot of real world feedback. Or when  there was a problem or an opportunity that was so obvious that we couldn't help  but take advantage of it, we would spend a minimum number of minutes on that  and go back to it. So at the end of that process we had this  fully designed product that we were using that was terrible implementation, not what you would  do if you started from scratch, but it was obvious that this was something that  had enormous value. Everyone who worked at the company agreed they would never work without  something like this again and so just on that basis, it seemed like it was  a great idea. We had years of evidence. 
Sam Altman:
Were there other things you consider pivoting to or was it just like, "There's the  stand. We're going to do it. It's working."? 
Stewart B.:
I wish that I had better access to the proposals that I had personally had  written, because I had a whole bunch of fucking terrible, terrible ideas and this was  the one that had the broadest consensus among the team as being something that would  be valuable. What's interesting at the time, though, so we had $5 million left in  the bank. None of our investors wanted their money back. They wanted to just go  try something else. Jason Horowitz is one of our investors so we went down and  did the partner meeting and kind of presented what our big idea was. And at  that time, this is before we actually started development of Slack, but when we had  this proto version, we said that one day this could be like a hundred million  dollar a year revenue business, so it can be valued at billon dollars on that  basis, and we though that that was a decade out and that would be the  culmination, like that would be, "We have a 100% of the ." We went past  that in year two, kind of blew past it. We'll do $200 million in gap  revenue this year and still be growing at a fast rate. Now we think that  number is more like $10 billion in revenue, like a $100 billion company. We didn't  realize how good of an idea it was, but it's also ... There's an interesting  story about deciding to shut it down. Because we still had money, because we had  a relatively big user base who were very engaged but we had a really a  leaky bucket, it was very difficult to add new users to the game we were  working on, which was called the Glitch, I felt very stuck in this position because  on the one hand you have a narrative of, "Good entrepreneurs are resilient," "When everyone  else thinks it's a bad idea, it's probably a really good idea," "You have to  dig in, prove people wrong," "Keep going even when times are dark." On the other  hand, there was a morning when I woke up and I was just like, "I  can't do this anymore. I'm the CEO, I'm the chairman of the board. I'm sure  that this is not going to work." And it's actually hard to convince everyone else. 
Sam Altman:
I think that's one of the hardest things about evaluating ideas, is saying, "Well, this  one is not going to work." So, do you remember what it was about that  morning when you woke up and said, "You know what? It's time to move on."? 
Stewart B.:
I think people have very little introspective access to their own cognitive processes to begin  with, so anything they say would be kind of making it up, but I think  it was, I had exhausted all of the ideas we had to fix the problems,  so we focused on new user experience and we really wanted to teach people what  this was and what the possibilities were and get them excited about it. Because we  were okay at attracting people onto the top of the funnel, but had a hard  time retaining them. And- 
Sam Altman:
That's always when I tell people to give up an idea. When you're out of  ideas about how to make this work. 
Stewart B.:
Yeah, exactly. So we would have an idea, we try it, didn't work. Have another  idea and then like ... At first, you can fill up posted notes, those giant  boards, you can fill up whiteboards with ideas and you argue about which ones are  the highest value and then one by one you kind of eliminate them. Months go  by, a year goes by, another six months goes by and it's like, "Shit. These  were all of the ideas that we thought were the worst ones to begin with  and we only have three left. So this is probably not going to work." 
Sam Altman:
You mentioned at one point you had a bunch of proposals. How do you come  up with ideas, or how did you even think that Slack is maybe this thing  is really good idea? How do you come up with new ideas and how do  you evaluate them? 
Stewart B.:
Coming up with new ideas is easy. I feel like it's hard almost to not  come up with ideas. They're not necessarily good, I guess, so the recognition of which  ones are good is actually much harder. I recently finally got around to reading Thinking,  Fast and Slow and there's this story there of a psychologist named Herbert Simon who  studied in Chess Grandmasters. Because Chess Grandmasters are one of the category of people, like  really experienced doctors, who people regard as being kind of miraculous in their ability to  intuit a situation, like glance at a board and say, "White checkmate in three moves."  And people thought that this was some super well developed intuition. And professor Simon argues  that it's just a recognition in the same way that a 18 month year old  can look at a dog and say, "Doggie." Or within the first 500 milliseconds of  someone speaking, you can tell if they're angry at you. You just develop the ability  to recognize situations and cues which are hard to disambiguate, that are hard to articulate  and enumerate in your mind and say exactly why you think this way, but over  time you get better and better at doing it. I've been making software professionally for  about 20 years, so the specific micro ideas or sub ideas about feature implementation or  the order of actions and the user experience, or whether a certain feature is more  important than another feature, those ones I feel like I have an intuitive sense, which  is actually just the ability to recognize over time. It's really a process of filtering.  You just have constant ideas, everyone has ideas. We at Slack have developed a lot  of organizational design and practices to try to absorb as many as we can. We  get about somewhere between 10,000 and 20,000 tweets at us a month and all of  those are responded to by an actual human being. We get a similar number of  customer support tickets and we have processes for trying to filter up and abstract all  of the feedback that we're getting. But even down to the level of all of  the tweets at Slack the company are pumped into our Slack instance and people check  them and they can add an emoji reaction and that filters into another channel for  the ones that seem above some threshold of, "This seems like a good idea and  it also seems practical. I could imagine that we could do it." And then there's  some debate at that level, but every ... Because almost every person I know in  the technology industry also uses Slack, any time I talk to any of them they  have ideas for me. I mean, they have complaints or proposals. 
Sam Altman:
The little ideas I get the intuition on. What about the big idea? Did you  spend a lot of time thinking before you decided to go ahead with Slack? Did  you spend a lot of time thinking about, "This would be really strategically valuable and  there'd be a deep network effect."? 
Stewart B.:
No, not at all. We thought that this would be valuable for individual teams and  we could see that value demonstrated in our own experience. We definitely didn't see how  applicable it would be, like how many, because by now we have 38,000 paid customers  around the world. That's a lot. We have a sales team, but we've never talked  to 90% plus of our customers because we just couldn't talk to that many people.  Again, this is already further than I thought we would get at the very end.  So we under-recognized it. Those big idea are, I don't know. We talked briefly about  this topic before and one of my favorite lines is, "I have a great idea  for a novel. All I need is for someone to write it for me." Which  is obviously preposterous, right? No one would really think, I mean only a very deluded  person would think that that's true, but people definitely have ideas for apps all the  time and those can be harder to assess and it's so rare that I hear  one where someone proposes an idea where it just, "Prima facie, that is a good  idea," as opposed to, "I don't know. It really depends on the execution." Actually, just  last night I was watching something on YouTube and the ad that came up before  was for an app called MileIQ and it's ... You turn it on, asks for  location permissions and then every time you go on a car ride, it just looks  to see the speed that you're traveling and then at the point you stop, it  stops it. Kind of like an automatic Strava, it just records all your trips and  then you can swipe right to say that, "This is a trip that my company  is supposed to reimburse me for, so I've left ... " Anyway, it's a very  simple idea. It's actually a really great idea because people have trouble recording those, they  have trouble in audits with the IRS to prove that they used the car for  this purpose, and stuff like that. And that's one that actually seems like, "Hmm." The  application doesn't seem very difficult. I think that would be a good idea. If you  could market it affectively, it'll probably be a successful business." But 95% of the ideas  I hear, I have no idea whether it'd be successful. 
Sam Altman:
Where are you generally on the spectrum of, "Ideas matter and it's worth the effort  to think of a good one." And, "It's just all about execution and ideas don't  matter at all. You just have to keep trying things until someday one of your  internal tools turns into Slack."? 
Stewart B.:
There's a lot of people in this class and then there's a lot of people  at Stanford and there's a lot of people in South Bay, there's a lot of  people in the Bay area and many, many people want to start tech companies and  all hope to be successful. To a certain extent, there's a 100 million monkeys banging  on a typewriter and someone's going to start a Shakespeare sonnet. There is a little  bit of that, like a little bit of bias and other cognitive biases that come  into play. On the other hand, I don't think either end of those spectrum is  right. I would definitely lean more towards execution. There's a good Steve Jobs elaboration of,  John Sculley got hired and started hiring all these people at Apple who thought that  only the ideas matter. The idea absolutely matters, but the idea when you start hating  execution has to accommodate so many things. In Steve Jobs' case it's about what it's  possible to get glass to do and how small you can make batteries and what  kind of tencel shrink the plastics have and stuff like that, because your idea has  to stand up to all these things and it'll transmogrify and alter and may even  be unrecognizable in the end, but if you're really good at execution then the core  of the idea can make it through all those steps. I absolutely favor execution, but  you can execute a lot on a terrible idea and then not get anywhere. So  you kind of need both. 
Sam Altman:
If you were starting a new company today, April 2017, where would you look for  ideas? What sorts of things would be thinking about? 
Stewart B.:
I don't know if this works for other people, but I would definitely look at  my own experiences as a consumer, generally, because it's really easy for me to see  things that are frustrating. I complain about stuff a lot, especially given how good my  life is. My life is really good and yet every time I have to fly  somewhere, I have at least 10 significant complaints and 50 minor complaints. And every one  of those is an opportunity. Every one of those things that don't go as well  as you would like are opportunities for improvement and some of those seem like someone  else should see them and it should be really obvious, but they're often not. There's  this story I tell in our internal onboarding process of me and our head of  project design going for a walk in Vancouver. And our office in Vancouver is in  neighborhood that's really narrow, so I'd walk, sidewalks have sandwich boards from vendors out so  it's very crowded, you kind of wind around, and so it starts raining while we're  on this walk and maybe two thirds of people had umbrellas with them and we  didn't and people are walking towards us on the sidewalk and almost no one would  move their umbrella out of the way so that the pocky things wouldn't get us  in the eye. Like I said, the sidewalks were very narrow so we had to  keep on ducking. I can come up with many explanations for why this would be,  but don't explain with malice that which can be explained by ignorance, so probably it  wasn't just that these people have few avenues for exercising power in their lives and  they were choosing but this moment to exercise some power, but then there's really only  two explanations. One is that they just walk through the world and they don't see  that we're going like this to get away from an umbrella, even though inevitably they  had that experience themselves, or they see that this is happening and they're like, "I  just can't think of anything I can do to ameliorate the situation." Despite the fact  that it's this, it is tiny, like a hundredth of a calorie worth of effort  and a tiny amount of consideration and the point of telling that story is that  that's the way, this is a sad way of looking at the world, but that's  the way that most people go through the world, they're oblivious to the problems that  other people have and if they notice the problems, they're unable to come up with  any kind of solution. Two thirds of the people just didn't tilt their umbrellas, which  means that if you're the kind of person who's willing to tilt your umbrella, there's  a whole world of opportunity out there. 
Sam Altman:
Do you have any tips on how to notice that you should tilt your umbrella? 
Stewart B.:
Pay attention. I mean, that is the biggest thing. But pay attention especially to other  people. So you can pay attention to your own signals and maybe if you get  practiced- 
Sam Altman:
This is actually worked in the Y Combinator history when people have needed to come  up with new idea on the fly. It doesn't usually work, but when they really  just commit to paying attention to other people and what seems broken that has worked.  And then, so if you do this while you come up with thousands of ideas,  anything you've noticed as a good way to filter those besides just building up the  experience of seeing a lot and the recognition? 
Stewart B.:
Yeah, there is the kind of dialectic between execution and idea, because there's definitely something  to be said when things are above some threshold of good to sticking with the  same idea. Not so much because that idea is the best one, but because the  wild and neurotic chasing of the next idea when you have something that has a  little bit of traction can be really dangerous. Slack is a good example of that.  We conceived up Slack as a tool for business. We developed it that way, we  priced it that way. It was successful, but there was also this huge amount of  social usage. Like now maybe a third of the Slack teams that get created are  for some kind of social usage and a lot of people had the idea that  we should pursue that, like we should make Slack for groups or even just pivot  to being a consumer company, kind of a private version of Tweeter or something like  that. We've been pretty steadfast in not doing that, because you can only do so  many things well and there's a lot of shiny objects like that that are very  distracting. 
Sam Altman:
Are you the one that says no to most of those new ideas or have  you had to build that into the company? When something is going as well as  Slack, there's so many new ideas. 
Stewart B.:
I'm chief-no-sayer, but there's definitely a whole bunch of people who are equally committed to  no. 
Sam Altman:
Great. We can probably take one question for Stewart, if someone has a really good  one, before it's time to go. 
Stewart B.:
Only a really good one. 
Sam Altman:
Maybe that was so much pressure . 
Speaker 3:
So you already had a successful company before Slack. I was wondering what's the differences  between as a first time founder and a second time founder. 
Sam Altman:
What are the differences as the second time founder? 
Stewart B.:
Every one of the external factors was much easier. So the first time, I mentioned,  that we ended up making Flickr because we couldn't raise any money and we were  desperate to try to finish something. Fast forward seven years and we could raise as  much money as we wanted, so that was easy. We didn't even make a .  We just said, "We would like some money," and then people gave us money. A  lot of those things were easier. So it was easier to get press, it was  easier to get attention, easier to recruit, easier to raise money. It's not any easier  to be successful certainly, because we started the company that made Slack in 2009 and  failed pretty hard for three and a half years, kind of like pushing into the  wind of something that wasn't ultimately going to work. And we could have just stopped  there, right? It was great for our investors that we didn't recap and we just  kept the same company and so that company has now been successful. We could have  also just walked away from the company and incorporated a new company and then it  would have been, "My second company was a failure," instead of, "My second company was  a great pivot." I'm not sure which things get harder. I'm not sure that any  do, other than the degree of self-criticism and awareness of your own limitations. 
Sam Altman:
Great. Thank you very much. 
Stewart B.:
No problem. 
Sam Altman:
I appreciate you coming in, and now I'll have Adam talk about metrics. 
Adam D.:
Great. Thanks for having me. I'm happy to be here. I'm going to talk about  metrics and measurement and this whole group of issues that come up as you're trying  to start a company. One of the things I think is really interesting with this  topic is that it's so much of what you have to do in a startup  is specific to the startup. You're going to have to get to be an expert  at some domain, you're going to have to get better than anyone else at that  domain and a lot of what it takes to do that is hard to teach  in a class, because it's just different for every startup. But metrics is this domain  that's kind of general enough to be useful to almost all startups. I'm going to  start off with this example. So, this is the first version of amazon.com from the  early '90s and I really like looking at these. You can look at the first  versions of lots of different products. It's a good reminder that it's much easier to  get these things started than you'd think. Even these companies that are massive today, if  you just go and look at how they started, this is a good way to  keep yourself in touch with the reality of how companies get started. Does anyone know  why Amazon decided to start with books? 
Speaker 5:
Choice. 
Adam D.:
Choice? Yeah, yeah. That's actually ... I think what you mean is that you have  a lot of ... You get to choose from a lot of different books? Yeah,  yeah. So it wasn't just that they chose books. It was that books give you  ... Jeff Bezos wanted to start a store and one of the most important things  when you're getting started is differentiation. So you don't have to make a product that's  going to appeal to everyone immediately, but you need to make a product that is  going to appeal to some people more than anything else. You want to find some  way to differentiate. And usually you want to do that by doing something that's going  to be easy for you, but that's really hard for all the existing companies. And  the internet enables this, in Amazon's case, because brick and mortar stores have limited space  to hold their inventory and so you realize that as on the internet you can  have big warehouse somewhere, you can have items shipped from all over and this is  something that enables you to offer much wider selection than a normal store. Then you  go and run through the thought experiment. You say, "Well, if what we can do  is provide better selection, then which category, what kind of store is going to benefit  the most from upgrade or selection?" And it turns out that books is a category  where you have the most ... There are more books than there are other products,  right? So, if you look at movies, it's very expensive to make a movie. It's  not an industry that would benefit immediately from the internet in a way that books  would. I think there's this general point here that measurement can turn a vague idea  into a good idea and if you do a lot of these kind of like  back of envelope calculations, it can really help sharpen your ideas and it can help  you iterate and move through the space of ideas a lot faster than if you  have to go and build a prototype. You can imagine what if Amazon started and  they started to focus on movies and after a few years they realized that it  wasn't working that well, then they're set back by a few years, now there's competition.  These kind of metrics on, even on the realm of ideas, can really help you  get started. And the one other thing that I would say about this is that  you ... I think using measurement here is very good for helping you figure out  what products are going to be possible, where you're going to be able to provide  a lot of value to a user. I actually don't think metrics work that well  for stuff like market size, so you just saw this example that Stewart, they thought  that market size for chat was a hundred million a year in revenue, it turns  out to be a hundred times bigger. Just very hard to figure those things out.  I wouldn't spend a lot of time on that. I think you want to know  that whatever you're doing, some day there's some path for it to eventually get really  big, but you want to really apply your metrics to how can you differentiate your  product and what's going to be possible that wasn't previously possible. Okay. So next to  that, now you're building your product and what should you measure. Just any guesses, if  you guys have products, what are you measuring, what would you recommend? 
Speaker 6:
Amount of users. 
Adam D.:
The amount of users. Yeah, yeah. So what exactly would you mean when you say  that? 
Speaker 6:
How many people use it total and how often people use it . 
Adam D.:
Yeah, yeah. So how many people use it total and how often they use it.  Go ahead. 
Speaker 7:
I think people share it . 
Adam D.:
Whether it's growing, whether they do something . Yeah, go ahead. 
Speaker 8:
Total addressable market, like when you're getting started. 
Adam D.:
Total addressable market, yeah. Yeah? 
Speaker 9:
User retention. 
Adam D.:
Yeah, user retention. So, these are all great ideas. There's lots of things that you  can measure. As a startup, you really need to focus, so the core concept I  would try to focus on is users that are getting value today. So, there's different  indicators of this. You might say you want to focus on active users, people are  going to be active if they're getting value, might want to focus on revenue, that's  a sign that people are getting value. In the case of a market place, where  you have things like buyers and sellers, that's usually a case where it makes sense  to measure transactions. Any time you have these different groups, if you say, "Okay, we're  going to measure buyers and then we're going to measure sellers," then there's actually a  lot of stuff you can do that will make things better for the buyers and  worse for the sellers and some of those things are good. Measuring transactions is a  way to unify them and align your work on things that are going to benefit  both sides. If you have a two-sided market or anything like that, then you want  to measure transactions or transaction values. Things like eBay, they'll measure total value of all  the goods that are sold on eBay. To start, this is kind of a quick  thing. And you go a long way just by measuring this. If you're not measuring  anything, you're susceptible to all these cognitive biases that Stewart talked about, all kinds of  problems, you're just not going to know what's going on. Next most important thing to  think about is retention. First, I want to make sure everyone understands what a cohort  is. A cohort is all the users who first use your product in a certain  time window. So for a startup, you might make the time window a week. So  you can say, "There's a cohort of all the users who first use your product  in the first week that it launched." And you can track those users over time.  There's a cohort of all the users that first used the product a week later  and you measure all these cohorts and you can track them. This tells you what's  happening to users over time. It's a really important way to balance you're just measuring  the total value that all the users are getting. This is a graph of total  of cohort usage for a relatively old internet product. The blue line is users who  joined this product in 2004 and so you can see ... The y-axis here is  what percent of them are still using it and x-axis is time. So over time,  if you just follow the blue line, you can see, basically, every month there's fewer  and fewer people that join the product in 2004 that are still using it. This  is basically the same story for all these different cohort lines. What do you think  is going to happen to this product in ten ... This is about six years,  the scale of this graph. What do think is going to happen in six years  later? Any guesses? Yeah? 
Speaker 10:
It's going to be opposite. 
Adam D.:
It's going to turn around? 
Speaker 10:
Yeah. 
Adam D.:
Yeah, good. I think something would need ... Something different would have to happen to  change that. 
Speaker 11:
No one wants to use it. 
Adam D.:
No one will be using it, yeah. It's hard to know what would happen. I  think probably everyone's going to stop using their product. Some companies will manage to turn  these around, but it's very hard to because you get fewer and fewer ... You  get less and less usage from your old users. It's also usually the old users  are what cause you to get new users. So if you have this problem, it's  pretty bad. This is a pretty scary place to be. You can have a ton  of users, you can have total usage growing, but if usage from every cohort is  declining, then you're going to run into big problems. There's this idea called the ring  of fire, which is a way to kind of visualize this. Unless there's some kind  of lower bound, you're going to burn out all your users. So here's the analogy.  Imagine there's a big field of, let's say, of dead grass and someone lights a  match in the middle. You're going to get a fire and the fire is going  to grow and grow, but then in the middle of the fire, you're going to  start to run out of fuel and it's going to burn out. So you get  this ring and the ring is going to get bigger and bigger and in the  center you're going to burn out more and more of the fuel. And so the  fire will get bigger and bigger, but eventually you're going to run out of the  whole field and the fire's going to stop. You don't want this to happen. Here's  an animation I made to try to illustrate this. Here's what you notice. It looks  great at first. There's more fire every frame than the previous frame, but in the  end, there's nothing. Right? You don't want this to happen. It's really important to internalize  this concept, because if you don't you can fool yourself and you can do a  lot of things that are going to cause you to get more and more usage  over time, but at the expense of the existing users. And if you don't do  this, eventually, your company is going to die. I think some recent examples are, I  think Groupon was a good example of this. They got huge, tons of usage. All  these merchants, but it wasn't a good experience for the merchants in the long term  so they all stopped using it. And then I think more recently, Pokemon Go is  another example, got really big, lots of users, but it just kind of faded away.  So, you don't want to build a fad. You want to build something that's going  to last, so you need to make sure that you're measuring the usage from the  existing users. It's not just that you're getting more users. You need to make sure  the existing users keep using the product. Actually, if this can go in reverse, you  are in an extremely powerful position. There's a small number of products where the cohort  usage actually will increase over time. One example is WhatsApp. I think most messaging apps,  as you get more of your friends on the app, there's more people to send  messages to and so they're able to get this increase in cohort usage over time.  Uber on the rider side, this is another good example, as people start to get  used to Uber, they use it more, as the pickup times come down, they use  it more, as prices come down, they use it more and so that's really ...  Uber is in a really strong position there. It's not true on the driver side.  Uber has problems on the driver side because they don't have this property there. And  then Facebook, you can see in the numbers they report publicly, they keep on ...  Their total usage is growing faster than their number of users. You know that they  must have this increasing usage per user over time. If you can get to this  ... This is a very hard to get to this, but if you can get  to this point then you're in a really strong position. One way to actually think  about an idea is whether ... As the product gets bigger, as you get more  users, as you develop the market, is it the kind of product where people would  use it more over time? I think that's another kind of lens you can apply  as you're considering ideas. Just one more data point. This venture capitalist named Tom Tunguz  did a study on what predicts valuations that startups raise money at and it's just  a simple corelation study and these are some different factors in how much they correlate  to the valuation that the startups can make at. First is growth of revenue and  that's 0.18 correlation. Second is just how much revenue they have in total. That actually  correlates a little bit more strongly than growth, but by far the strongest correlation is  account expansion, which means people spending more over time, which is same thing as retention  increasing over time. And this correlation is extremely strong. So for a complex number, like  what the valuation of a company is going to be, the fact that you can  explain 0.54 out of 1 of the variation in that number by the single variable  about whether the company is increasing revenue per user over time, that's a sign that  ... It's almost like there are just two classes of companies. One class where there's  increasing revenue over time and another class that gets totally different valuations where this is  not happening. The main thing I just want you to remember from today is you  should measure your retention. That's going to lead you to do a lot of things  that will help you in all kinds of ways. It's going to help you grow,  it's going to help you retain your users, it's going to help you build a  product that users really love. Next, I just want to talk about exponential growth. Retention  is the most important thing, but it's not sufficient. You need to have a product  that's going to retain your users, but that's not necessarily enough to make sure that  you're going to be able to build a big product and reach everyone. You also  need to make sure that you're going to grow. And by far the most powerful  way to grow is through exponential growth. This is just a few functions plotted. The  red line is linear growth and blue line is cubic growth and then the green  line is exponential growth. The linear growth path ... You can do this through things,  like continually doing PR you can get more and more users, but eventually ... There's  a limit to how big you're going to be able to get. It gets expensive  to run advertising, it gets expensive to do these kinds of things that attract a  fixed number of users, but if you can do things that cause you to get  users in proportion to the number of users you already have, then that's going to  get you to exponential growth. There's other places to learn about what to do to  grow faster, but as you're measuring your growth, you want to make sure that the  way you measure it is aligned with what you want. And if what you want  is exponential growth, then what you need to measure is the rate of growth. A  good way to measure this is just the percent increase per week in how much  usage you have or how much revenue you have. This is a tweet from Paul  that I especially agreed with. He said, "A suggestion for really tough founders is instead  of graphing your revenue, graph the percent growth per week." I actually don't know why  he said that this is for really tough founders. I actually think this is what's  going to make you really tough, if you measure this. I think can go the  other way. This is a very important thing to measure. The reason he says you  need to be tough is that it's going to be hard to keep up exponential  growth and it's going to be hard to ... It may not feel good to  just see this line that doesn't increase, but ... You can also measure your total  revenue. There's no problem with doing that, but just make sure that you're measuring the  percent growth per week. There's going to be things you can do that don't cause  exponential growth that would actually help you maybe more in the short term and you  want to focus on stuff that's going to cause the product to grow in the  short term and in the long term, which is basically things like viral growth, getting  users to invite more of their friends, getting word of mouth spreading, getting some kind  of dynamic where the bigger your product gets the easier it is to attract the  next user to it. Next I want to talk about iteration. One of the most  important advantages you have as a startup is you can iterate faster than a big  company. There's all kinds of things that cause it to be hard for companies to  change direction and you just don't have these forces when you're a small company. If  you can iterate much, much faster than other companies, then you can go through ideas  faster, you can go through changes to your product faster, you can go through changes  to improve retention faster, you can go through changes to increase growth faster. It's going  to add up to a really, really strong position. But at the same time, you  have to iterate in the right direction. So, one of the reasons metrics, even just  simple metrics are so important, is that they let you know whether you're going in  the right direction. They let you know whether what you're doing is working and they  let you ... Out of all these different ways you can go, they kind of  help you stay aligned. One other point I want to make, I think this isn't  mentioned too often, but you can measure how fast you're iterating and you can measure  all these components of iteration. Whether it's fully quantified or not, you should have in  your head "How long did this cycle take us and what can we do to  make the next cycle faster?" At Quora, one of the metrics we track is how  long does it take from an engineer hitting commit on their code to us being  able to run the unit test and then get that code out into production and  we try to keep that between 10 and 15 minutes from change in the code  base to it being totally out to all the users, because that allows people to  iterate just much, much faster. And if we didn't track that metric, there's just a  lot of forces that naturally would cause that to get slower. I think a lot  of companies end up ... They'll maybe start out like that, then they end up  releasing daily or they go to weekly or even monthly. I've heard of teams that  are on every three months they do a release. And you just imagine, how could  you possibly iterate if your releases are only going monthly or even weekly. Next, there's  a spectrum of how seriously to take metrics that I see a lot of companies  stumble on. On the one hand, there's not taking your metrics seriously enough and this  might mean things like, you don't have metrics, might be that you have metrics, they're  on some page but no one ever looks at that, it might mean that you  have metrics and one person in the company looks at it but no one else  looks at it. The issues that happen on this end of the spectrum are, you  kind of ... You can fool yourself, you go around in circles, you just lose  sense of what the right things to do are. But then there's another end of  the spectrum where you can actually take metrics too seriously. You got to remember that  metrics are an abstraction. What you actually care about underneath is, lets say, the value  to the users and maybe the long term value to the users. A good metric  will be strongly correlated with that, but it won't be perfect and so you don't  want to forget that there's a reality under the metric that's not exactly the metric  itself and if you focus too much on the metrics, then you start to get  into things like, "Oh, there's these ways to gain the metrics and we can make  the metrics go up but actually we're doing some things that are going to hurt  us long term and those things are going to outweigh the short term benefits." I  would really make sure to be careful about that. I'd say, personally, I think the  ideal is to be somewhere in the middle and I'd say you want to be  very close to taking the metrics extremely seriously. Not too seriously, but I think most  companies bias toward not being serious enough about their metrics. If you're in the middle,  in the right spot, then you're learning, you're making good decisions and you're informed, but  you're still aware that there is a product underneath and there is a strategy and  there's more to the world than just these metrics. I see a lot of the  times companies would oscillate between these. They'll really focus on metrics and that cause them  to do some things that they think are bad and they, "Well, these metrics are  terrible and we're going to throw away our metrics, not focus on them at all."  Then you get into this world where you don't actually know what you're doing and  there's no accountability and so oscillation is not very productive. You want to just make  sure you find the right point and try to stick there. Finally, I want to  just talk about some of these psychological aspects that come with metrics. I think one  of the big things that comes when you start measuring things is you just get  closer to being in touch with reality. That can be painful because sometimes the metrics  are going to be worse than you thought they were. Sometimes, actually, they can be  better than you thought and that's great and everyone feels great, but a lot of  the times if you weren't measuring something, chances are you weren't even aware of it,  it was hard for you to make it good and so it won't be good.  That means that as you set up metrics, there's going to be some pain and  that can cause problems for companies. I even think at the leadership level, this can  get even worse. I think the majority of leaders out there in the world rely  on some amount of delusion they're creating among their followers to get them excited, to  have them working on this project in the first place and metrics can get in  the way of that. People get into this ... They get into this question, "Okay,  well, should we ... " What should you do as a leader and this is  where I come out, after thinking about this for a long time, I think you  want to be learning from the past, so you want everyone fully aware of what  has happened, what has worked, what has not worked. You want to be totally in  touch with the reality of where you are today. So, if things are good, you  should know they're good. If things are not good, you should know they're not good,  you should know exactly where there are problems and where there are not problems. But  then, those things are not intention with being optimistic about the future. You should be  optimistic about the future because there's so much potential. The reason you're doing a startup  is because of this potential and if you don't feel confident that the path you're  on is able to get you to a place that you feel optimistic about, then  that's a good sign that you might want to change the path and you should  do something about that because that's totally within your control. There's not really a tension  between acknowledging that things are not in a great spot and having people not be  optimistic about the future. As long as you just make sure that ... The reality  is, and I don't think you should make the people optimistic about the future in  a way that you don't believe it. I think you should do the things you  need to do to get yourself to be confident and optimistic about the future and  then it's easy to get people to follow through that. That's all I have and  happy to take questions. 
Sam Altman:
He wrote the questions and answers out. He's go some questions . Thank you very  much, Adam. All right. See you all on Thursday.