All right. Good afternoon. Today we have two different speakers. We have Stewart, the founder
and CEO of Slack and then we have Adam, the founder and CEO of Quora.
I'm going to interview Stewart. We're gonna talk about ideas and how to find good
ones and then Adam is going to talk about metrics, why they're important and what
to measure. Stewart, thank you for coming and spending some time with us. Stewart is
the found of Flickr and Slack, and what I didn't know until very recently is
that he tried to be the founder twice of Massively Multiplayer Online Games and they
turned into these two iconic companies. So I wanted to talk about just the process
of coming up with the idea for the game and then the ideas for Flickr
and Slack.
All right. So, yes, twice we set out trying to build a web-based massively multiplayer
game and the origin of that was all the way back in 1992, I arrived
at college and I got my account on the school's Unix machine and was exposed
to the internet for the first time and it was kind of mind blowing experience.
This is about a year before the web became popular, so it was IRC, NewsNet,
Talk, email and I was really fascinated with the idea that people could find their
community, irrespective of what that community was. If you were a breast cancer survivor or
model train enthusiast or whatever, if you're interested in some, as I was, obscure 19th
century biologists, then you can find all that stuff. And the idea of a game
that lived online like that and had the game-like aspects or play as a pretext
for social interaction seemed really interesting. I'm not going to go into too long of
an explanation, because it turns out it's actually like a terrible, terrible idea and ...
At least as a commercial business. It might have been fun as a hobby. I
think it could have been the kind of business that sustained the number of employees
required to operate it, like a restaurant or something like that, but, when in 2009
we started a company that ended up making Slack, we raised $17.5 million in venture
capital and it was never going to become the kind of business that would justify
that investment. The first time was a little bit different. 2002 was after the dotcom
crash, it was after Worldcom and Enron accounting scandals, it was after 9/11 and the
NASDAQ was down 80%, S&P500 was down 65% from its peak. No one wanted to
invest in internet facing stuff, so it was very hard to tell whether this was
a good idea or not, because there's no signal from the market that said, "This
is an investible idea," because nothing was investible at that time. We raised a little
bit of friends and family money, we spent about a year or so building a
prototype. The prototype was well received, but it was going to take us another year
and a half to finish it and we literally couldn't raise any money at all.
We got to the point where the one person on the team who had kids
was the only person who was getting paid and then we decided to try to
find something else we could do that would make use of the technology that we'd
already developed, but that would be quicker to get to market. And that thing was
Flickr, but it was also a bad idea at first. Because we had developed this
game client and we had a messaging server on the backend, we just repurposed the
game client as a photo sharing client. In the game, you had a bunch of
slots along the bottom of the screen for inventory. Those became slots into which you
could fit photos. And because the game was all realtime interaction, the photo sharing was
all realtime interaction. Which meant that you had to be online at the same time
as someone you cared about in order to use it and while it demoed really
well, it was amazing technology especially for at the time we started on that, maybe
late 2003, it was a terrible product because you didn't care about any of the
people who were actually online. That morphed really quickly and the experience making Slack was
a little bit different. We had all of the resources we could ever ask for.
We had more than enough money because we had money left over. We decided to
abandon the idea and we had AWS, which we didn't have before, hardware was effectively
free, internet connections were much better, there were many more people online, hardware was better.
None of those external conditions was the reason this wasn't as Massively Multiplayer Game on
the web wasn't a good idea. It was just not a good idea because there
wasn't a big enough market for it.
And why was Slack a good idea? How did you decide that Slack was looking
to build?
Well, we started developing the system for internal communication that was based on IRC. And
those of you who had used IRC and also use Slack can recognize that Slack
is IRC with all of the things that were missing when it was originally designed
in 1989. I think, this is post facto analysis so it's hard to know if
this really true or not, but I think that the reason that Slack has been
so successful is because we had a three and a half year long design process
where we weren't conscious that we were designing it. So, we started using IRC. IRC
has many deficiencies, among them you can't store and forward messages. So if I wanted
to send you a message and you're not connected to the server at the moment
I want to send it, I just can't send you a message. So the first
thing we built was a bot to log all the messages and put them in
an archive. And once we had that, we had messages in our database and we
thought search would be useful so we built search for it. And this was 2009,
there were no good IRC clients for iPhones and so we built an HTML5 front-end
for the archives in Safari and once we had that, we wanted to be able
to post. And just like on and on and on. But, in the normal process
of software development, there's a huge amount of ego and speculation. So, when you're arguing
with someone whether this feature is a good idea or not, if you can get
to the point of enlightenment and a Buddha sense, then maybe you can do that
without any ego, but the practical reality is people are very ego-latent in those discussions
and there's a lot of speculation. Like, "I think this could be really valuable for
someone," whereas when we were building a system that was kind of the proto-Slack that
was the inspiration for Slack, it was whatever problem is the most irritating, like something
that we just absolutely can't stand, we would address that with a minimum amount of
effort and then go back to what we're supposed to be doing and that kind
of let it cook for several months and we had 45 people working on it,
working on the game, so we had a lot of real world feedback. Or when
there was a problem or an opportunity that was so obvious that we couldn't help
but take advantage of it, we would spend a minimum number of minutes on that
and go back to it. So at the end of that process we had this
fully designed product that we were using that was terrible implementation, not what you would
do if you started from scratch, but it was obvious that this was something that
had enormous value. Everyone who worked at the company agreed they would never work without
something like this again and so just on that basis, it seemed like it was
a great idea. We had years of evidence.
Were there other things you consider pivoting to or was it just like, "There's the
stand. We're going to do it. It's working."?
I wish that I had better access to the proposals that I had personally had
written, because I had a whole bunch of fucking terrible, terrible ideas and this was
the one that had the broadest consensus among the team as being something that would
be valuable. What's interesting at the time, though, so we had $5 million left in
the bank. None of our investors wanted their money back. They wanted to just go
try something else. Jason Horowitz is one of our investors so we went down and
did the partner meeting and kind of presented what our big idea was. And at
that time, this is before we actually started development of Slack, but when we had
this proto version, we said that one day this could be like a hundred million
dollar a year revenue business, so it can be valued at billon dollars on that
basis, and we though that that was a decade out and that would be the
culmination, like that would be, "We have a 100% of the ." We went past
that in year two, kind of blew past it. We'll do $200 million in gap
revenue this year and still be growing at a fast rate. Now we think that
number is more like $10 billion in revenue, like a $100 billion company. We didn't
realize how good of an idea it was, but it's also ... There's an interesting
story about deciding to shut it down. Because we still had money, because we had
a relatively big user base who were very engaged but we had a really a
leaky bucket, it was very difficult to add new users to the game we were
working on, which was called the Glitch, I felt very stuck in this position because
on the one hand you have a narrative of, "Good entrepreneurs are resilient," "When everyone
else thinks it's a bad idea, it's probably a really good idea," "You have to
dig in, prove people wrong," "Keep going even when times are dark." On the other
hand, there was a morning when I woke up and I was just like, "I
can't do this anymore. I'm the CEO, I'm the chairman of the board. I'm sure
that this is not going to work." And it's actually hard to convince everyone else.
I think that's one of the hardest things about evaluating ideas, is saying, "Well, this
one is not going to work." So, do you remember what it was about that
morning when you woke up and said, "You know what? It's time to move on."?
I think people have very little introspective access to their own cognitive processes to begin
with, so anything they say would be kind of making it up, but I think
it was, I had exhausted all of the ideas we had to fix the problems,
so we focused on new user experience and we really wanted to teach people what
this was and what the possibilities were and get them excited about it. Because we
were okay at attracting people onto the top of the funnel, but had a hard
time retaining them. And-
That's always when I tell people to give up an idea. When you're out of
ideas about how to make this work.
Yeah, exactly. So we would have an idea, we try it, didn't work. Have another
idea and then like ... At first, you can fill up posted notes, those giant
boards, you can fill up whiteboards with ideas and you argue about which ones are
the highest value and then one by one you kind of eliminate them. Months go
by, a year goes by, another six months goes by and it's like, "Shit. These
were all of the ideas that we thought were the worst ones to begin with
and we only have three left. So this is probably not going to work."
You mentioned at one point you had a bunch of proposals. How do you come
up with ideas, or how did you even think that Slack is maybe this thing
is really good idea? How do you come up with new ideas and how do
you evaluate them?
Coming up with new ideas is easy. I feel like it's hard almost to not
come up with ideas. They're not necessarily good, I guess, so the recognition of which
ones are good is actually much harder. I recently finally got around to reading Thinking,
Fast and Slow and there's this story there of a psychologist named Herbert Simon who
studied in Chess Grandmasters. Because Chess Grandmasters are one of the category of people, like
really experienced doctors, who people regard as being kind of miraculous in their ability to
intuit a situation, like glance at a board and say, "White checkmate in three moves."
And people thought that this was some super well developed intuition. And professor Simon argues
that it's just a recognition in the same way that a 18 month year old
can look at a dog and say, "Doggie." Or within the first 500 milliseconds of
someone speaking, you can tell if they're angry at you. You just develop the ability
to recognize situations and cues which are hard to disambiguate, that are hard to articulate
and enumerate in your mind and say exactly why you think this way, but over
time you get better and better at doing it. I've been making software professionally for
about 20 years, so the specific micro ideas or sub ideas about feature implementation or
the order of actions and the user experience, or whether a certain feature is more
important than another feature, those ones I feel like I have an intuitive sense, which
is actually just the ability to recognize over time. It's really a process of filtering.
You just have constant ideas, everyone has ideas. We at Slack have developed a lot
of organizational design and practices to try to absorb as many as we can. We
get about somewhere between 10,000 and 20,000 tweets at us a month and all of
those are responded to by an actual human being. We get a similar number of
customer support tickets and we have processes for trying to filter up and abstract all
of the feedback that we're getting. But even down to the level of all of
the tweets at Slack the company are pumped into our Slack instance and people check
them and they can add an emoji reaction and that filters into another channel for
the ones that seem above some threshold of, "This seems like a good idea and
it also seems practical. I could imagine that we could do it." And then there's
some debate at that level, but every ... Because almost every person I know in
the technology industry also uses Slack, any time I talk to any of them they
have ideas for me. I mean, they have complaints or proposals.
The little ideas I get the intuition on. What about the big idea? Did you
spend a lot of time thinking before you decided to go ahead with Slack? Did
you spend a lot of time thinking about, "This would be really strategically valuable and
there'd be a deep network effect."?
No, not at all. We thought that this would be valuable for individual teams and
we could see that value demonstrated in our own experience. We definitely didn't see how
applicable it would be, like how many, because by now we have 38,000 paid customers
around the world. That's a lot. We have a sales team, but we've never talked
to 90% plus of our customers because we just couldn't talk to that many people.
Again, this is already further than I thought we would get at the very end.
So we under-recognized it. Those big idea are, I don't know. We talked briefly about
this topic before and one of my favorite lines is, "I have a great idea
for a novel. All I need is for someone to write it for me." Which
is obviously preposterous, right? No one would really think, I mean only a very deluded
person would think that that's true, but people definitely have ideas for apps all the
time and those can be harder to assess and it's so rare that I hear
one where someone proposes an idea where it just, "Prima facie, that is a good
idea," as opposed to, "I don't know. It really depends on the execution." Actually, just
last night I was watching something on YouTube and the ad that came up before
was for an app called MileIQ and it's ... You turn it on, asks for
location permissions and then every time you go on a car ride, it just looks
to see the speed that you're traveling and then at the point you stop, it
stops it. Kind of like an automatic Strava, it just records all your trips and
then you can swipe right to say that, "This is a trip that my company
is supposed to reimburse me for, so I've left ... " Anyway, it's a very
simple idea. It's actually a really great idea because people have trouble recording those, they
have trouble in audits with the IRS to prove that they used the car for
this purpose, and stuff like that. And that's one that actually seems like, "Hmm." The
application doesn't seem very difficult. I think that would be a good idea. If you
could market it affectively, it'll probably be a successful business." But 95% of the ideas
I hear, I have no idea whether it'd be successful.
Where are you generally on the spectrum of, "Ideas matter and it's worth the effort
to think of a good one." And, "It's just all about execution and ideas don't
matter at all. You just have to keep trying things until someday one of your
internal tools turns into Slack."?
There's a lot of people in this class and then there's a lot of people
at Stanford and there's a lot of people in South Bay, there's a lot of
people in the Bay area and many, many people want to start tech companies and
all hope to be successful. To a certain extent, there's a 100 million monkeys banging
on a typewriter and someone's going to start a Shakespeare sonnet. There is a little
bit of that, like a little bit of bias and other cognitive biases that come
into play. On the other hand, I don't think either end of those spectrum is
right. I would definitely lean more towards execution. There's a good Steve Jobs elaboration of,
John Sculley got hired and started hiring all these people at Apple who thought that
only the ideas matter. The idea absolutely matters, but the idea when you start hating
execution has to accommodate so many things. In Steve Jobs' case it's about what it's
possible to get glass to do and how small you can make batteries and what
kind of tencel shrink the plastics have and stuff like that, because your idea has
to stand up to all these things and it'll transmogrify and alter and may even
be unrecognizable in the end, but if you're really good at execution then the core
of the idea can make it through all those steps. I absolutely favor execution, but
you can execute a lot on a terrible idea and then not get anywhere. So
you kind of need both.
If you were starting a new company today, April 2017, where would you look for
ideas? What sorts of things would be thinking about?
I don't know if this works for other people, but I would definitely look at
my own experiences as a consumer, generally, because it's really easy for me to see
things that are frustrating. I complain about stuff a lot, especially given how good my
life is. My life is really good and yet every time I have to fly
somewhere, I have at least 10 significant complaints and 50 minor complaints. And every one
of those is an opportunity. Every one of those things that don't go as well
as you would like are opportunities for improvement and some of those seem like someone
else should see them and it should be really obvious, but they're often not. There's
this story I tell in our internal onboarding process of me and our head of
project design going for a walk in Vancouver. And our office in Vancouver is in
neighborhood that's really narrow, so I'd walk, sidewalks have sandwich boards from vendors out so
it's very crowded, you kind of wind around, and so it starts raining while we're
on this walk and maybe two thirds of people had umbrellas with them and we
didn't and people are walking towards us on the sidewalk and almost no one would
move their umbrella out of the way so that the pocky things wouldn't get us
in the eye. Like I said, the sidewalks were very narrow so we had to
keep on ducking. I can come up with many explanations for why this would be,
but don't explain with malice that which can be explained by ignorance, so probably it
wasn't just that these people have few avenues for exercising power in their lives and
they were choosing but this moment to exercise some power, but then there's really only
two explanations. One is that they just walk through the world and they don't see
that we're going like this to get away from an umbrella, even though inevitably they
had that experience themselves, or they see that this is happening and they're like, "I
just can't think of anything I can do to ameliorate the situation." Despite the fact
that it's this, it is tiny, like a hundredth of a calorie worth of effort
and a tiny amount of consideration and the point of telling that story is that
that's the way, this is a sad way of looking at the world, but that's
the way that most people go through the world, they're oblivious to the problems that
other people have and if they notice the problems, they're unable to come up with
any kind of solution. Two thirds of the people just didn't tilt their umbrellas, which
means that if you're the kind of person who's willing to tilt your umbrella, there's
a whole world of opportunity out there.
Do you have any tips on how to notice that you should tilt your umbrella?
Pay attention. I mean, that is the biggest thing. But pay attention especially to other
people. So you can pay attention to your own signals and maybe if you get
practiced-
This is actually worked in the Y Combinator history when people have needed to come
up with new idea on the fly. It doesn't usually work, but when they really
just commit to paying attention to other people and what seems broken that has worked.
And then, so if you do this while you come up with thousands of ideas,
anything you've noticed as a good way to filter those besides just building up the
experience of seeing a lot and the recognition?
Yeah, there is the kind of dialectic between execution and idea, because there's definitely something
to be said when things are above some threshold of good to sticking with the
same idea. Not so much because that idea is the best one, but because the
wild and neurotic chasing of the next idea when you have something that has a
little bit of traction can be really dangerous. Slack is a good example of that.
We conceived up Slack as a tool for business. We developed it that way, we
priced it that way. It was successful, but there was also this huge amount of
social usage. Like now maybe a third of the Slack teams that get created are
for some kind of social usage and a lot of people had the idea that
we should pursue that, like we should make Slack for groups or even just pivot
to being a consumer company, kind of a private version of Tweeter or something like
that. We've been pretty steadfast in not doing that, because you can only do so
many things well and there's a lot of shiny objects like that that are very
distracting.
Are you the one that says no to most of those new ideas or have
you had to build that into the company? When something is going as well as
Slack, there's so many new ideas.
I'm chief-no-sayer, but there's definitely a whole bunch of people who are equally committed to
no.
Great. We can probably take one question for Stewart, if someone has a really good
one, before it's time to go.
Maybe that was so much pressure .
So you already had a successful company before Slack. I was wondering what's the differences
between as a first time founder and a second time founder.
What are the differences as the second time founder?
Every one of the external factors was much easier. So the first time, I mentioned,
that we ended up making Flickr because we couldn't raise any money and we were
desperate to try to finish something. Fast forward seven years and we could raise as
much money as we wanted, so that was easy. We didn't even make a .
We just said, "We would like some money," and then people gave us money. A
lot of those things were easier. So it was easier to get press, it was
easier to get attention, easier to recruit, easier to raise money. It's not any easier
to be successful certainly, because we started the company that made Slack in 2009 and
failed pretty hard for three and a half years, kind of like pushing into the
wind of something that wasn't ultimately going to work. And we could have just stopped
there, right? It was great for our investors that we didn't recap and we just
kept the same company and so that company has now been successful. We could have
also just walked away from the company and incorporated a new company and then it
would have been, "My second company was a failure," instead of, "My second company was
a great pivot." I'm not sure which things get harder. I'm not sure that any
do, other than the degree of self-criticism and awareness of your own limitations.
Great. Thank you very much.
I appreciate you coming in, and now I'll have Adam talk about metrics.
Great. Thanks for having me. I'm happy to be here. I'm going to talk about
metrics and measurement and this whole group of issues that come up as you're trying
to start a company. One of the things I think is really interesting with this
topic is that it's so much of what you have to do in a startup
is specific to the startup. You're going to have to get to be an expert
at some domain, you're going to have to get better than anyone else at that
domain and a lot of what it takes to do that is hard to teach
in a class, because it's just different for every startup. But metrics is this domain
that's kind of general enough to be useful to almost all startups. I'm going to
start off with this example. So, this is the first version of amazon.com from the
early '90s and I really like looking at these. You can look at the first
versions of lots of different products. It's a good reminder that it's much easier to
get these things started than you'd think. Even these companies that are massive today, if
you just go and look at how they started, this is a good way to
keep yourself in touch with the reality of how companies get started. Does anyone know
why Amazon decided to start with books?
Choice? Yeah, yeah. That's actually ... I think what you mean is that you have
a lot of ... You get to choose from a lot of different books? Yeah,
yeah. So it wasn't just that they chose books. It was that books give you
... Jeff Bezos wanted to start a store and one of the most important things
when you're getting started is differentiation. So you don't have to make a product that's
going to appeal to everyone immediately, but you need to make a product that is
going to appeal to some people more than anything else. You want to find some
way to differentiate. And usually you want to do that by doing something that's going
to be easy for you, but that's really hard for all the existing companies. And
the internet enables this, in Amazon's case, because brick and mortar stores have limited space
to hold their inventory and so you realize that as on the internet you can
have big warehouse somewhere, you can have items shipped from all over and this is
something that enables you to offer much wider selection than a normal store. Then you
go and run through the thought experiment. You say, "Well, if what we can do
is provide better selection, then which category, what kind of store is going to benefit
the most from upgrade or selection?" And it turns out that books is a category
where you have the most ... There are more books than there are other products,
right? So, if you look at movies, it's very expensive to make a movie. It's
not an industry that would benefit immediately from the internet in a way that books
would. I think there's this general point here that measurement can turn a vague idea
into a good idea and if you do a lot of these kind of like
back of envelope calculations, it can really help sharpen your ideas and it can help
you iterate and move through the space of ideas a lot faster than if you
have to go and build a prototype. You can imagine what if Amazon started and
they started to focus on movies and after a few years they realized that it
wasn't working that well, then they're set back by a few years, now there's competition.
These kind of metrics on, even on the realm of ideas, can really help you
get started. And the one other thing that I would say about this is that
you ... I think using measurement here is very good for helping you figure out
what products are going to be possible, where you're going to be able to provide
a lot of value to a user. I actually don't think metrics work that well
for stuff like market size, so you just saw this example that Stewart, they thought
that market size for chat was a hundred million a year in revenue, it turns
out to be a hundred times bigger. Just very hard to figure those things out.
I wouldn't spend a lot of time on that. I think you want to know
that whatever you're doing, some day there's some path for it to eventually get really
big, but you want to really apply your metrics to how can you differentiate your
product and what's going to be possible that wasn't previously possible. Okay. So next to
that, now you're building your product and what should you measure. Just any guesses, if
you guys have products, what are you measuring, what would you recommend?
The amount of users. Yeah, yeah. So what exactly would you mean when you say
that?
How many people use it total and how often people use it .
Yeah, yeah. So how many people use it total and how often they use it.
Go ahead.
I think people share it .
Whether it's growing, whether they do something . Yeah, go ahead.
Total addressable market, like when you're getting started.
Total addressable market, yeah. Yeah?
Yeah, user retention. So, these are all great ideas. There's lots of things that you
can measure. As a startup, you really need to focus, so the core concept I
would try to focus on is users that are getting value today. So, there's different
indicators of this. You might say you want to focus on active users, people are
going to be active if they're getting value, might want to focus on revenue, that's
a sign that people are getting value. In the case of a market place, where
you have things like buyers and sellers, that's usually a case where it makes sense
to measure transactions. Any time you have these different groups, if you say, "Okay, we're
going to measure buyers and then we're going to measure sellers," then there's actually a
lot of stuff you can do that will make things better for the buyers and
worse for the sellers and some of those things are good. Measuring transactions is a
way to unify them and align your work on things that are going to benefit
both sides. If you have a two-sided market or anything like that, then you want
to measure transactions or transaction values. Things like eBay, they'll measure total value of all
the goods that are sold on eBay. To start, this is kind of a quick
thing. And you go a long way just by measuring this. If you're not measuring
anything, you're susceptible to all these cognitive biases that Stewart talked about, all kinds of
problems, you're just not going to know what's going on. Next most important thing to
think about is retention. First, I want to make sure everyone understands what a cohort
is. A cohort is all the users who first use your product in a certain
time window. So for a startup, you might make the time window a week. So
you can say, "There's a cohort of all the users who first use your product
in the first week that it launched." And you can track those users over time.
There's a cohort of all the users that first used the product a week later
and you measure all these cohorts and you can track them. This tells you what's
happening to users over time. It's a really important way to balance you're just measuring
the total value that all the users are getting. This is a graph of total
of cohort usage for a relatively old internet product. The blue line is users who
joined this product in 2004 and so you can see ... The y-axis here is
what percent of them are still using it and x-axis is time. So over time,
if you just follow the blue line, you can see, basically, every month there's fewer
and fewer people that join the product in 2004 that are still using it. This
is basically the same story for all these different cohort lines. What do you think
is going to happen to this product in ten ... This is about six years,
the scale of this graph. What do think is going to happen in six years
later? Any guesses? Yeah?
It's going to be opposite.
It's going to turn around?
Yeah, good. I think something would need ... Something different would have to happen to
change that.
No one will be using it, yeah. It's hard to know what would happen. I
think probably everyone's going to stop using their product. Some companies will manage to turn
these around, but it's very hard to because you get fewer and fewer ... You
get less and less usage from your old users. It's also usually the old users
are what cause you to get new users. So if you have this problem, it's
pretty bad. This is a pretty scary place to be. You can have a ton
of users, you can have total usage growing, but if usage from every cohort is
declining, then you're going to run into big problems. There's this idea called the ring
of fire, which is a way to kind of visualize this. Unless there's some kind
of lower bound, you're going to burn out all your users. So here's the analogy.
Imagine there's a big field of, let's say, of dead grass and someone lights a
match in the middle. You're going to get a fire and the fire is going
to grow and grow, but then in the middle of the fire, you're going to
start to run out of fuel and it's going to burn out. So you get
this ring and the ring is going to get bigger and bigger and in the
center you're going to burn out more and more of the fuel. And so the
fire will get bigger and bigger, but eventually you're going to run out of the
whole field and the fire's going to stop. You don't want this to happen. Here's
an animation I made to try to illustrate this. Here's what you notice. It looks
great at first. There's more fire every frame than the previous frame, but in the
end, there's nothing. Right? You don't want this to happen. It's really important to internalize
this concept, because if you don't you can fool yourself and you can do a
lot of things that are going to cause you to get more and more usage
over time, but at the expense of the existing users. And if you don't do
this, eventually, your company is going to die. I think some recent examples are, I
think Groupon was a good example of this. They got huge, tons of usage. All
these merchants, but it wasn't a good experience for the merchants in the long term
so they all stopped using it. And then I think more recently, Pokemon Go is
another example, got really big, lots of users, but it just kind of faded away.
So, you don't want to build a fad. You want to build something that's going
to last, so you need to make sure that you're measuring the usage from the
existing users. It's not just that you're getting more users. You need to make sure
the existing users keep using the product. Actually, if this can go in reverse, you
are in an extremely powerful position. There's a small number of products where the cohort
usage actually will increase over time. One example is WhatsApp. I think most messaging apps,
as you get more of your friends on the app, there's more people to send
messages to and so they're able to get this increase in cohort usage over time.
Uber on the rider side, this is another good example, as people start to get
used to Uber, they use it more, as the pickup times come down, they use
it more, as prices come down, they use it more and so that's really ...
Uber is in a really strong position there. It's not true on the driver side.
Uber has problems on the driver side because they don't have this property there. And
then Facebook, you can see in the numbers they report publicly, they keep on ...
Their total usage is growing faster than their number of users. You know that they
must have this increasing usage per user over time. If you can get to this
... This is a very hard to get to this, but if you can get
to this point then you're in a really strong position. One way to actually think
about an idea is whether ... As the product gets bigger, as you get more
users, as you develop the market, is it the kind of product where people would
use it more over time? I think that's another kind of lens you can apply
as you're considering ideas. Just one more data point. This venture capitalist named Tom Tunguz
did a study on what predicts valuations that startups raise money at and it's just
a simple corelation study and these are some different factors in how much they correlate
to the valuation that the startups can make at. First is growth of revenue and
that's 0.18 correlation. Second is just how much revenue they have in total. That actually
correlates a little bit more strongly than growth, but by far the strongest correlation is
account expansion, which means people spending more over time, which is same thing as retention
increasing over time. And this correlation is extremely strong. So for a complex number, like
what the valuation of a company is going to be, the fact that you can
explain 0.54 out of 1 of the variation in that number by the single variable
about whether the company is increasing revenue per user over time, that's a sign that
... It's almost like there are just two classes of companies. One class where there's
increasing revenue over time and another class that gets totally different valuations where this is
not happening. The main thing I just want you to remember from today is you
should measure your retention. That's going to lead you to do a lot of things
that will help you in all kinds of ways. It's going to help you grow,
it's going to help you retain your users, it's going to help you build a
product that users really love. Next, I just want to talk about exponential growth. Retention
is the most important thing, but it's not sufficient. You need to have a product
that's going to retain your users, but that's not necessarily enough to make sure that
you're going to be able to build a big product and reach everyone. You also
need to make sure that you're going to grow. And by far the most powerful
way to grow is through exponential growth. This is just a few functions plotted. The
red line is linear growth and blue line is cubic growth and then the green
line is exponential growth. The linear growth path ... You can do this through things,
like continually doing PR you can get more and more users, but eventually ... There's
a limit to how big you're going to be able to get. It gets expensive
to run advertising, it gets expensive to do these kinds of things that attract a
fixed number of users, but if you can do things that cause you to get
users in proportion to the number of users you already have, then that's going to
get you to exponential growth. There's other places to learn about what to do to
grow faster, but as you're measuring your growth, you want to make sure that the
way you measure it is aligned with what you want. And if what you want
is exponential growth, then what you need to measure is the rate of growth. A
good way to measure this is just the percent increase per week in how much
usage you have or how much revenue you have. This is a tweet from Paul
that I especially agreed with. He said, "A suggestion for really tough founders is instead
of graphing your revenue, graph the percent growth per week." I actually don't know why
he said that this is for really tough founders. I actually think this is what's
going to make you really tough, if you measure this. I think can go the
other way. This is a very important thing to measure. The reason he says you
need to be tough is that it's going to be hard to keep up exponential
growth and it's going to be hard to ... It may not feel good to
just see this line that doesn't increase, but ... You can also measure your total
revenue. There's no problem with doing that, but just make sure that you're measuring the
percent growth per week. There's going to be things you can do that don't cause
exponential growth that would actually help you maybe more in the short term and you
want to focus on stuff that's going to cause the product to grow in the
short term and in the long term, which is basically things like viral growth, getting
users to invite more of their friends, getting word of mouth spreading, getting some kind
of dynamic where the bigger your product gets the easier it is to attract the
next user to it. Next I want to talk about iteration. One of the most
important advantages you have as a startup is you can iterate faster than a big
company. There's all kinds of things that cause it to be hard for companies to
change direction and you just don't have these forces when you're a small company. If
you can iterate much, much faster than other companies, then you can go through ideas
faster, you can go through changes to your product faster, you can go through changes
to improve retention faster, you can go through changes to increase growth faster. It's going
to add up to a really, really strong position. But at the same time, you
have to iterate in the right direction. So, one of the reasons metrics, even just
simple metrics are so important, is that they let you know whether you're going in
the right direction. They let you know whether what you're doing is working and they
let you ... Out of all these different ways you can go, they kind of
help you stay aligned. One other point I want to make, I think this isn't
mentioned too often, but you can measure how fast you're iterating and you can measure
all these components of iteration. Whether it's fully quantified or not, you should have in
your head "How long did this cycle take us and what can we do to
make the next cycle faster?" At Quora, one of the metrics we track is how
long does it take from an engineer hitting commit on their code to us being
able to run the unit test and then get that code out into production and
we try to keep that between 10 and 15 minutes from change in the code
base to it being totally out to all the users, because that allows people to
iterate just much, much faster. And if we didn't track that metric, there's just a
lot of forces that naturally would cause that to get slower. I think a lot
of companies end up ... They'll maybe start out like that, then they end up
releasing daily or they go to weekly or even monthly. I've heard of teams that
are on every three months they do a release. And you just imagine, how could
you possibly iterate if your releases are only going monthly or even weekly. Next, there's
a spectrum of how seriously to take metrics that I see a lot of companies
stumble on. On the one hand, there's not taking your metrics seriously enough and this
might mean things like, you don't have metrics, might be that you have metrics, they're
on some page but no one ever looks at that, it might mean that you
have metrics and one person in the company looks at it but no one else
looks at it. The issues that happen on this end of the spectrum are, you
kind of ... You can fool yourself, you go around in circles, you just lose
sense of what the right things to do are. But then there's another end of
the spectrum where you can actually take metrics too seriously. You got to remember that
metrics are an abstraction. What you actually care about underneath is, lets say, the value
to the users and maybe the long term value to the users. A good metric
will be strongly correlated with that, but it won't be perfect and so you don't
want to forget that there's a reality under the metric that's not exactly the metric
itself and if you focus too much on the metrics, then you start to get
into things like, "Oh, there's these ways to gain the metrics and we can make
the metrics go up but actually we're doing some things that are going to hurt
us long term and those things are going to outweigh the short term benefits." I
would really make sure to be careful about that. I'd say, personally, I think the
ideal is to be somewhere in the middle and I'd say you want to be
very close to taking the metrics extremely seriously. Not too seriously, but I think most
companies bias toward not being serious enough about their metrics. If you're in the middle,
in the right spot, then you're learning, you're making good decisions and you're informed, but
you're still aware that there is a product underneath and there is a strategy and
there's more to the world than just these metrics. I see a lot of the
times companies would oscillate between these. They'll really focus on metrics and that cause them
to do some things that they think are bad and they, "Well, these metrics are
terrible and we're going to throw away our metrics, not focus on them at all."
Then you get into this world where you don't actually know what you're doing and
there's no accountability and so oscillation is not very productive. You want to just make
sure you find the right point and try to stick there. Finally, I want to
just talk about some of these psychological aspects that come with metrics. I think one
of the big things that comes when you start measuring things is you just get
closer to being in touch with reality. That can be painful because sometimes the metrics
are going to be worse than you thought they were. Sometimes, actually, they can be
better than you thought and that's great and everyone feels great, but a lot of
the times if you weren't measuring something, chances are you weren't even aware of it,
it was hard for you to make it good and so it won't be good.
That means that as you set up metrics, there's going to be some pain and
that can cause problems for companies. I even think at the leadership level, this can
get even worse. I think the majority of leaders out there in the world rely
on some amount of delusion they're creating among their followers to get them excited, to
have them working on this project in the first place and metrics can get in
the way of that. People get into this ... They get into this question, "Okay,
well, should we ... " What should you do as a leader and this is
where I come out, after thinking about this for a long time, I think you
want to be learning from the past, so you want everyone fully aware of what
has happened, what has worked, what has not worked. You want to be totally in
touch with the reality of where you are today. So, if things are good, you
should know they're good. If things are not good, you should know they're not good,
you should know exactly where there are problems and where there are not problems. But
then, those things are not intention with being optimistic about the future. You should be
optimistic about the future because there's so much potential. The reason you're doing a startup
is because of this potential and if you don't feel confident that the path you're
on is able to get you to a place that you feel optimistic about, then
that's a good sign that you might want to change the path and you should
do something about that because that's totally within your control. There's not really a tension
between acknowledging that things are not in a great spot and having people not be
optimistic about the future. As long as you just make sure that ... The reality
is, and I don't think you should make the people optimistic about the future in
a way that you don't believe it. I think you should do the things you
need to do to get yourself to be confident and optimistic about the future and
then it's easy to get people to follow through that. That's all I have and
happy to take questions.
He wrote the questions and answers out. He's go some questions . Thank you very
much, Adam. All right. See you all on Thursday.