×

17
18

DL, MF, Multi, N, P[N] OpenAI Five DoTA2 Finals match livestream has begun: match against OG, plus additional OA announcement at end (twitch.tv)

submitted by gwern

all 26 comments
sorted by:
best

[–]gwern[S] 6 points7 points  (10 children)

Notes:

  • a big surprise is promised at the end: https://twitter.com/gdb/status/1117085741710860288 EDIT: coop matches, 2 humans+3 OA vs matches. So far they're pretty boring, like the AlphaGo-human colab matches were. The robots don't need to worry about being unemployed by 'centaurs' anytime soon, it seems... EDITEDIT: after 59 minutes of this, I've become increasingly embarrassed on humanity's behalf

  • Ilya estimated total training time now at 45,000 years of DoTA2 experience

    A few weeks wallclock on the latest DoTA2 patch, so it should be converged (unlike TI). Post summary:

    In total, the current version of OpenAI Five has consumed 800 petaflop/s-days and experienced about 45,000 years of Dota self-play over 10 realtime months (up from about 10,000 years over 1.5 realtime months as of The International), for an average of 250 years of simulated experience per day. The Finals version of OpenAI Five has a 99.9% winrate versus the TI version [2].

  • Match settings: current DoTA2 patch; still no summons/illusions; 17 heros for drafting (no mirror draft?); best of 3; 1 courier like TI (supposedly fully adapted down from 5 invulns now)

Matches: OA5 wins over OG, 2-0 (a flawless victory wiping away the shame of TI!); total: 8-0.

  1. victory for OA5; qualitatively, commentators are describing it as being similar to before (constant intense pushes, excellent exploitation of mistakes, paying to bring back heroes immediately to continue the fight), no major mistakes, OG fought hard but while it seemed even up to the middle, they started to crumble afterwards

    My impression is that OA5 is overall somewhat better, but it's hard to tell. We didn't see OA5 fall behind or any real exploitation of its apparent long-term strategy blindspots. OG might've been surprised by just how good OA5 really was and how fluidly & quickly it reacts, and wrongfooted by it, so we'll see if they can attempt to exploit OA5's weaknesses in game 2.

  2. victory for OA5: this game was a huge mess for OG. Despite going in with more of a draft advantage (OA5-estimated 60% vs 70%), OG just fell apart and lost towers within like 10 minutes. What a disappointment.

  3. OA also revealed other matches against pro teams:

  • 2-0
  • 2-0
  • 2-0

Links:

Commentary:

  • this was practically impressive, but it's also a little disappointing. Has OA5 repaired its long-term strategy understanding? Does it still fall apart when behind? Or did it simply improve its early game to the point where OG couldn't even try the TI stalling strategies & accumulation of long-term game? OG was unable to push OA hard enough to reveal anything interesting: we already knew it was eerily efficient at coordinating and timing and pouncing on mistakes, this 2-0 match (or the total 8-0) was mostly just more of the same.

    On the plus side, the Arena looks really nice. If the global DoTA2 community can't push it into the long game or otherwise cheese it while coordinating attempts, then we can safely conclude that OA5 really is damn good.

[–]sorrge 3 points4 points  (1 child)

Has OA5 repaired its long-term strategy understanding? Does it still fall apart when behind?

Does it really matter now? I doubt the "long-term strategy" is qualitatively more complex than the early game. To me there is little doubt that it can learn that as well as the early game, if trained specifically for that.

What's more interesting to me, is whether people can find flaws in it easily. With AlphaGo, it proved impossible: the learned strategy was not fragile at all, and seemingly no updates are necessary. That's similar to the chess situation, and IMHO is likely due to the tree search component. But with Dota, I'm not sure. It would be very interesting if they release it on the Internet to let people grind it for some time.

[–]gwern[S] 0 points1 point  (0 children)

if trained specifically for that.

That is why it matters.

[–]futureroboticist 1 point2 points  (2 children)

now is POMDP as complex as Dota solved?

[–][deleted] 1 point2 points  (0 children)

I think that’s a bit premature: this victory was only possible after spending an enormous amount of computation on a game with API access, after hugely restricting the complexity by limiting characters. Beyond that, Gwern’s questions above are highly pertinent as to how well OA5 has really mastered long term strategy.

I think we’ll see that problems at this general level of complexity continue to be a challenge for the foreseeable future. For instance, while I expect DeepMind to beat top players in SC2 within the next 1-2 years with all races and across a selection of maps, it seems likely that that will involve the use of multiple agents to represent different strategies, and that they will need different agents to play each race. That may not be an issue in the artificial world of video games, but it limits the applicability in the real world.

In other words, we’re seeing progress at limited aspects of games at the level of complexity of Dota 2 or SC2, but the resulting agents may not transfer easily to real world problems that you’d think are at roughly the same level of complexity, due to all the restrictions in place making this work.

[–]atlatic 0 points1 point  (0 children)

The biggest caveat is that it's only 17 heroes. Professional teams are used to playing the full game, and would need to practise to understand the 17-hero meta. But they are not likely to practise the 17-hero meta, since there's little incentive for them. Playing Dota 2 is hard work, and they have real tournaments which pay money to practise for. Unless OpenAI or someone else offers a big prize pool (say, $1M), professional teams are unlikely to analyze and/or practise the 17-hero meta. In the Finals event too, OG didn't seem like they are seriously playing the game. Kept all-chatting and they also played worse than how they were at the last major tournament.

So, in summary, full Dota 2 is nowhere near solved since it hasn't been attempted. We don't know whether 17-hero Dota 2 is solved, since professional teams don't seem to be taking it seriously, but it's probably not solved given how many clear mistakes Five keeps making.

[–]Taleuntum 0 points1 point  (2 children)

Is it confirmed that the surprise is the coop match(es)? If it is, I'm going to sleep, becuase it is pretty boring.

[–]gwern[S] 0 points1 point  (1 child)

Unless I completely misunderstood it, the coops are the surprise, yes. EDIT: lol whups, no, looks like the surprise is they are offering matches against OA5 online for a limited time: https://arena.openai.com/ !

[–][deleted] 0 points1 point  (0 children)

Last surprise is that they are rolling the system out for everyone to play against or play with at arena.openai.com for a limited time (for now).

It’s open from April 18th-April 21st.

[–]FatChocobo 0 points1 point  (1 child)

Were there any big changes this time? Or just retraining on the current patch?

Doesn't seem there are any blog posts giving details this time around.

[–][deleted] 2 points3 points  (9 children)

Looking forward to this! I expect we’ll see a similar limit on character choice to TI, but that they’ll do much better this time. I’d give OpenAI Five good odds of beating OG.

Still, even if they win the series, today’s outcome will only reinforce the need for transfer learning, as it sounds like training on the full set of characters is compute and time prohibitive due to lack of transfer preventing learning new characters the way a human can.

As an aside, I initially misheard GDB to be saying they had a StarCraft bot they would show later, and got really excited, since DeepMind seems to have found the naive approach (LSTM combined with self play) to get stuck in bad strategies. My guess for the surprise is some kind of human plus bot cooperative play.

Edit: restrictions announced: captain’s draft limited to 17 characters, no summon or illusions.

Reflections: very impressive showing from OA5, but the restrictions make this a somewhat hollow victory, and I don’t think this result tells us anything we didn’t already know. They still can’t play the full game, and it doesn’t look like current approaches will be able to any time soon. Current AI is still so brittle and inflexible compared to human cognition.

Part two: Cooperative gameplay confirmed!

[–]panties_in_my_ass 1 point2 points  (8 children)

but the restrictions make this a somewhat hollow victory

Mere months ago it was 1v1 in a single lane with one hero only. Here we are now with some minor item restrictions on a 5v5 with a 17 hero pool. The bot even played draft.

Why are people calling this match “so restricted”? Genuine question.

[–]aquamarlin391 2 points3 points  (1 child)

Because Dota actually has 117 heroes with various different strengths and niche. The current set of heroes limit viable meta strategy to one that the bots now excel at.

The typical counter to bots' early push (group up and brute force objectives) is split pushing (using highly mobile heroes to maneuver around the enemies and take undefended objectives). Heroes that enable split push are not available with the 17 pool.

I have no doubt that bots can learn more strategies with enough time and compute theoretically. However, given how much resources were needed for this mini-Dota, learning the full game may be impractical with the current method.

[–]panties_in_my_ass 0 points1 point  (0 children)

learning the full game may be impractical with the current method.

That's a fair hypothesis! Time will tell and I'm excited to watch.

[–][deleted] 1 point2 points  (4 children)

Not denying the achievement—as you point out, this is impressive progress, and not something that we knew was possible just a year or so ago. But I find the claims that this is “beating top pros at Dota” to be misleading, since it’s clear that this approach can’t handle the full game with all characters due to combinatorial explosion.

Basically they’ve achieved remarkable success at beating top pros at a simplified version of the full game that removes most of the strategic elements humans go through while playing, in terms of banning and drafting from the full character roster, and the strategies that entails. On top of that, there are lingering questions that Gwern raises above about how to interpret this victory in terms of strategy, given the AI’s mechanics and reaction time advantages.

TLDR: I’m impressed by the victory, but I think the restrictions they had to put in place to achieve it (which didn’t change from TI over half a year ago) means that current RL approaches cannot in fact handle the full game of Dota 2 or others of similar complexity.

[–]gwern[S] 3 points4 points  (3 children)

But I find the claims that this is “beating top pros at Dota” to be misleading, since it’s clear that this approach can’t handle the full game with all characters due to combinatorial explosion.

That's just as clear as it was clear from the 1x1 results that PPO would never scale to playing the full 5x5 game, much less more than 1 draft of heroes or normal couriers or...

[–][deleted] 0 points1 point  (2 children)

I would absolutely love to be wrong about this, but don’t you think if PPO could handle this, they would have done it? I guess I’m willing to cautiously believe that it would be possible with, say, two or more orders of magnitude more training, but surely it would be more useful to pursue more efficient approaches. FWIW my initial prediction based on the 1 v 1 results was that it would scale to more or less what we saw here, with a limited set of characters, so I don’t think I’m completely off the mark.

[–]gwern[S] 2 points3 points  (0 children)

They claim it does work but they didn't have time: https://openai.com/blog/how-to-train-your-openai-five/

We saw very little slowdown in training going from 5 to 18 heroes. We hypothesized the same would be true going to even more heroes, and after The International, we put a lot of effort into integrating new ones.

We spent several weeks training with hero pools up to 25 heroes, bringing those heroes to approximately 5k MMR (about 95th percentile of Dota players). Although they were still improving, they weren’t learning fast enough to reach pro level before Finals. We haven’t yet had time to investigate why, but our hypotheses range from insufficient model capacity to needing better matchmaking for the expanded hero pool to requiring more training time for new heroes to catch up to old heroes. Imagine how hard it is for a human to learn a new hero when everyone else has mastered theirs!

We believe these issues are fundamentally solvable, and solving them could be interesting in its own right. The Finals version plays with 17 heroes—we removed Lich because his abilities were changed significantly in Dota version 7.20.

[–]panties_in_my_ass 1 point2 points  (0 children)

don’t you think if PPO could handle this, they would have done it?

This is how I would state it: their current ability to wield PPO is not capable enough. It will take more experimentation and improvement to see where the limits are. They may or may not be sufficient for "full dota"

[–]FatChocobo 0 points1 point  (0 children)

As others have said, the achievement itself is great, but no doubt there'll be countless posts and news stories about how OpenAI have 'beat dota', which is really disingenuous.

The players playing in these games are used to having a much larger hero pool to select from, which has huge implications in terms of being able to counter certain strategies etc.

[–]Teradimich 0 points1 point  (2 children)

45,000 years of Dota 2 gameplay experience it is years totally or counting each hero separately? In the first case, it should be ~14 days of training, in the second — it is 250 days.

[–]gwern[S] 1 point2 points  (1 child)

It wouldn't make much sense to count each hero separately. It's not like they're training a separate OA5 for every possible hero. (That wouldn't work with their drafting method, for starters.) And where do you get 14 days? They've been training OA5 pretty much continuously, using transfer learning/NN surgery to expand the model every time they need to tweak the arch.

[–]Teradimich 1 point2 points  (0 children)

I tried to calculate how long the training took.

Here we can see how the OpenAI was considered earlier:

“~180 years per day (~900 years per day counting each hero separately)“ https://openai.com/blog/openai-five/

So, how did the OpenAI get the number 45k this time and what does this say about training time?

(45,000/180) / 18 ≈ 14 days of training. This is the case if 45k years is a generally accumulated experience for each of the heroes.

But if this is counting each hero separately ... 45,000/180 = 250 days of training.

Now I realized that there were 17 heroes, but this does not negate the huge difference in training time depending on how the OpenAI got the number 45k.

If I'm an idiot and did the calculations wrong, just tell me.

[–]MasterScrat 0 points1 point  (1 child)

/u/gwern maybe we can unsticky this now?

[–]gwern[S] 0 points1 point  (0 children)

Not like there's been any major RL news which ought to be stickied instead, but sure.