Hello everybody. I made this notebook. If you like this kind of thing, please subscribe to gwern's patreon. https://patreon.com/gwern
It's a GPT-2 1.5B model trained on the kingbase 2019 dataset. (>3M games of >2000 ELO rating.) It was trained for 400k steps with batch size 6 using 140 TPUs in 24h using a technique known as swarm training. Here's an incomplete whitepaper on swarm training: https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
Each line is of the form [Result "0-1"] [WhiteElo "2715"] [BlackElo "2793"] 1. e4 ...
Result 0-1 means black won; 1-0 means white won; 1/2-1/2 means a draw.
At runtime I prompt it with [Result "0-1"] and a high elo for white and black to make it more likely to generate higher level moves.
Our next project will be a GPT-2 IRC bot where you can talk with simulated people. We currently have one that wasn't trained for very long, yet the preliminary results are interesting enough to warrant a more serious time investment. https://twitter.com/theshawwn/status/1208667331230089216
Many people have asked for a thorough technical writeup which I hope to make available soon. In the meantime, you an read some of our GPT-2 1.5B adventures here: https://gwern.net/GPT-2#gpt-2-1.5b
Lastly, someone on /r/slatestarcodex apparently did this exact same thing a few months ago. They trained on algebraic notation instead of PGN format, which is basically x1y1x2y2 coordinate form with no mention of the type of piece. It was also trained on 1B moves. The engine is superior to ours and can apparently reach move 40 without blundering, according to the replay. https://www.reddit.com/r/slatestarcodex/comments/el87vo/a_ve...
I have also been porting the stylegan2 codebase to TPUs to facilitate swarm training. We hope to train on a very large dataset like the entirety of danbooru2018. No promises, but results are interesting so far. https://twitter.com/theshawwn/status/1214245145664802817
I hope you all found this enjoyable. The GCE bill is currently $50, which I'm keeping an eye on. (Go subscribe to gwern's patreon to see more projects like this!)
It's a GPT-2 1.5B model trained on the kingbase 2019 dataset. (>3M games of >2000 ELO rating.) It was trained for 400k steps with batch size 6 using 140 TPUs in 24h using a technique known as swarm training. Here's an incomplete whitepaper on swarm training: https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
The dataset is available here:
Each line is of the form [Result "0-1"] [WhiteElo "2715"] [BlackElo "2793"] 1. e4 ...Result 0-1 means black won; 1-0 means white won; 1/2-1/2 means a draw.
At runtime I prompt it with [Result "0-1"] and a high elo for white and black to make it more likely to generate higher level moves.
Our next project will be a GPT-2 IRC bot where you can talk with simulated people. We currently have one that wasn't trained for very long, yet the preliminary results are interesting enough to warrant a more serious time investment. https://twitter.com/theshawwn/status/1208667331230089216
Many people have asked for a thorough technical writeup which I hope to make available soon. In the meantime, you an read some of our GPT-2 1.5B adventures here: https://gwern.net/GPT-2#gpt-2-1.5b
Lastly, someone on /r/slatestarcodex apparently did this exact same thing a few months ago. They trained on algebraic notation instead of PGN format, which is basically x1y1x2y2 coordinate form with no mention of the type of piece. It was also trained on 1B moves. The engine is superior to ours and can apparently reach move 40 without blundering, according to the replay. https://www.reddit.com/r/slatestarcodex/comments/el87vo/a_ve...
I have also been porting the stylegan2 codebase to TPUs to facilitate swarm training. We hope to train on a very large dataset like the entirety of danbooru2018. No promises, but results are interesting so far. https://twitter.com/theshawwn/status/1214245145664802817
I hope you all found this enjoyable. The GCE bill is currently $50, which I'm keeping an eye on. (Go subscribe to gwern's patreon to see more projects like this!)