(Originally posted by Chessbase here.)
Before GM John Nunn published his first “Nunn Positions”, trying to find out a chess engine’s strength as opposed to another was an unsure task at best. Programs were usually pitted against each other in long matches, results were compiled, and their ratings calculated. A recurring issue was that new programs could often kill older ones straight from the opening thanks to more up-to-date opening books, sometimes even prepared specifically against them. This made it hard to answer the question of which was the best engine to analyze one’s games.
Nunn’s idea was quickly accepted as simple and brilliant, helping provide a controlled testing environment for engines. Instead of their opening books, engines would be tested against each other using a set of pre-selected opening positions, played as both white and then black, so that only the engine’s playing strength was tested, and not the quality of its openings book.
Over time, the concept has validated itself repeatedly, and the original suites have become a deserved standard, though a few shortcomings have also appeared: the chosen positions, while excellent, don’t actually cover the full gamut of openings or types of positions encountered in the opening phase, and thus the middlegame. Also, some positions have been known to initially play out almost always the same way in the first moves, as the top engines all agree on a specific continuation. The following openings suite was designed to address some of these issues, and provide a complementary testbed for engine testing.
This openings suite is comprised of 50 positions, leading to a round 100-game match, covering not only the major openings, but also standard pawn structures and opening systems. Therefore not only will you find an Alekhine, some French Defenses, and several flavors of the Sicilian, but there are also a Hedgehog, a Stonewall, a Colle, and a King’s Indian Attack for example.
To determine their frequency, the number of positions for each opening found in the suite is proportionate to the number of times played as per statistics culled from Mega Database 2010. In other words, the number of Sicilians found in the suite (9), is roughly proportionate to the number found in the 4.5 million game database. This isn’t true in an absolute sense, as minor concessions were made to favor variety as well.
Finally, to ensure diversity, the positions lead to junctures in theory, where each engine is free to choose the direction it prefers. Each and every position was tested with several top engines for a few minutes, and their choices were compared. If they all chose the same continuation, the position was rejected. This preliminary testing, followed by several test matches, overseeing the games, helped refine the positions chosen.
---
Mr. Silver talks more about his opening suite and how its goal is to provide engines with the greatest number of questions here.Here are the 50 positions I test every engine with as White and Black:
(Thank you, GM John Nunn and Albert Silver, for this fine opening suite.)
[Round "1-2"]
[Opening "Bird's Opening"]
[ECO "A03"]
[PlyCount "15"]
1. f4 d5 2. Nf3 Nf6 3. b3 g6 4. Bb2 Bg7 5. e3 O-O 6. Be2 c5 7. O-O Nc6 8. Ne5 *
[Round "3-4"]
[Opening "Reti Opening"]
[ECO "A05"]
[PlyCount "11"]
1. Nf3 Nf6 2. g3 g6 3. Bg2 Bg7 4. O-O O-O 5. d3 d6 6. e4 *
[Round "5-6"]
[Opening "English, Four Knights"]
[ECO "A29"]
[PlyCount "6"]
1. c4 e5 2. Nc3 Nf6 3. Nf3 Nc6 *
[Round "7-8"]
[Opening "Symmetrical English, Hedgehog"]
[ECO "A30"]
[PlyCount "18"]
1. c4 c5 2. Nf3 Nf6 3. g3 b6 4. Bg2 Bb7 5. Nc3 e6 6. O-O Be7 7. d4 cxd4 8. Qxd4
d6 9. Rd1 a6 *
[Round "9-10"]
[Opening "Benko Gambit"]
[ECO "A58"]
[PlyCount "12"]
1. d4 Nf6 2. c4 c5 3. d5 b5 4. cxb5 a6 5. bxa6 g6 6. Nc3 Bxa6 *
[Round "11-12"]
[Opening "Benoni, Modern"]
[ECO "A73"]
[PlyCount "12"]
1. d4 Nf6 2. c4 c5 3. d5 e6 4. Nc3 exd5 5. cxd5 d6 6. e4 g6 *
[Round "13-14"]
[Opening "Dutch, Leningrad"]
[ECO "A87"]
[PlyCount "15"]
1. d4 f5 2. g3 Nf6 3. Bg2 g6 4. Nf3 Bg7 5. O-O O-O 6. c4 d6 7. Nc3 Qe8 8. d5 *
[Round "15-16"]
[Opening "Dutch, Stonewall"]
[ECO "A90"]
[PlyCount "14"]
1. d4 f5 2. g3 Nf6 3. Bg2 e6 4. c4 d5 5. Nf3 c6 6. O-O Bd6 7. b3 Qe7 *
[Round "17-18"]
[Opening "Scandinavian"]
[ECO "B01"]
[PlyCount "11"]
1. e4 d5 2. exd5 Qxd5 3. Nc3 Qa5 4. d4 Nf6 5. Nf3 c6 6. Bc4 *
[Round "19-20"]
[Opening "Alekhine, 4. Nf3"]
[ECO "B05"]
[PlyCount "7"]
1. e4 Nf6 2. e5 Nd5 3. d4 d6 4. Nf3 *
[Round "21-22"]
[Opening "Pirc, Classical"]
[ECO "B08"]
[PlyCount "12"]
1. e4 d6 2. d4 Nf6 3. Nc3 g6 4. Nf3 Bg7 5. Be2 O-O 6. O-O c6 *
[Round "23-24"]
[Opening "Pirc, Austrian attack"]
[ECO "B09"]
[PlyCount "10"]
1. e4 d6 2. d4 Nf6 3. Nc3 g6 4. f4 Bg7 5. Nf3 O-O *
[Round "25-26"]
[Opening "Caro-Kann, Advance"]
[ECO "B12"]
[PlyCount "10"]
1. e4 c6 2. d4 d5 3. e5 Bf5 4. Nf3 e6 5. Be2 c5 *
[Round "27-28"]
[Opening "Caro-Kann, Panov"]
[ECO "B14"]
[PlyCount "11"]
1. e4 c6 2. d4 d5 3. exd5 cxd5 4. c4 Nf6 5. Nc3 e6 6. Nf3 *
[Round "29-30"]
[Opening "Caro-Kann, Classical"]
[ECO "B19"]
[PlyCount "19"]
1. e4 c6 2. d4 d5 3. Nc3 dxe4 4. Nxe4 Bf5 5. Ng3 Bg6 6. h4 h6 7. Nf3 Nd7 8. h5
Bh7 9. Bd3 Bxd3 10. Qxd3 *
[Round "31-32"]
[Opening "Sicilian, 2.c3"]
[ECO "B22"]
[PlyCount "10"]
1. e4 c5 2. c3 d5 3. exd5 Qxd5 4. d4 Nf6 5. Nf3 e6 *
[Round "33-34"]
[Opening "Sicilian, Closed"]
[ECO "B25"]
[PlyCount "16"]
1. e4 c5 2. Nc3 Nc6 3. g3 g6 4. Bg2 Bg7 5. d3 d6 6. f4 e6 7. Nf3 Nge7 8. O-O
O-O *
[Round "35-36"]
[Opening "Sicilian, Sveshnikov"]
[ECO "B33"]
[PlyCount "20"]
1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 e5 6. Ndb5 d6 7. Bg5 a6 8.
Na3 b5 9. Bxf6 gxf6 10. Nd5 f5 *
[Round "37-38"]
[Opening "Sicilian, Maroczy Bind"]
[ECO "B38"]
[PlyCount "13"]
1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 g6 5. c4 Nf6 6. Nc3 d6 7. Be2 *
[Round "39-40"]
[Opening "Sicilian, Paulsen"]
[ECO "B46"]
[PlyCount "10"]
1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nc6 5. Nc3 a6 *
[Round "41-42"]
[Opening "Sicilian, Richter-Rauzer"]
[ECO "B66"]
[PlyCount "16"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 Nc6 6. Bg5 e6 7. Qd2 a6 8.
O-O-O h6 *
[Round "43-44"]
[Opening "Sicilian, Dragon"]
[ECO "B79"]
[PlyCount "22"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 g6 6. Be3 Bg7 7. f3 O-O 8. Qd2
Nc6 9. Bc4 Bd7 10. O-O-O Ne5 11. Bb3 Rc8 *
[Round "45-46"]
[Opening "Sicilian, Najdorf"]
[ECO "B80"]
[PlyCount "10"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 *
[Round "47-48"]
[Opening "Sicilian, Scheveningen"]
[ECO "B84"]
[PlyCount "16"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. Be2 e6 7. O-O Be7 8. f4
O-O *
[Round "49-50"]
[Opening "French, Advance"]
[ECO "C02"]
[PlyCount "12"]
1. e4 e6 2. d4 d5 3. e5 c5 4. c3 Nc6 5. Nf3 Qb6 6. a3 c4 *
[Round "51-52"]
[Opening "French, Tarrasch"]
[ECO "C05"]
[PlyCount "12"]
1. e4 e6 2. d4 d5 3. Nd2 Nf6 4. e5 Nfd7 5. c3 c5 6. Bd3 Nc6 *
[Round "53-54"]
[Opening "French, Winawer"]
[ECO "C19"]
[PlyCount "12"]
1. e4 e6 2. d4 d5 3. Nc3 Bb4 4. e5 c5 5. a3 Bxc3+ 6. bxc3 Ne7 *
[Round "55-56"]
[Opening "Philidor's Defense"]
[ECO "C41"]
[PlyCount "10"]
1. e4 e5 2. Nf3 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Be7 *
[Round "57-58"]
[Opening "Petroff Defense"]
[ECO "C42"]
[PlyCount "11"]
1. e4 e5 2. Nf3 Nf6 3. Nxe5 d6 4. Nf3 Nxe4 5. d4 d5 6. Bd3 *
[Round "59-60"]
[Opening "Scotch"]
[ECO "C45"]
[PlyCount "7"]
1. e4 e5 2. Nf3 Nc6 3. d4 exd4 4. Nxd4 *
[Round "61-62"]
[Opening "Italian"]
[ECO "C54"]
[PlyCount "14"]
1. e4 e5 2. Nf3 Nc6 3. Bc4 Bc5 4. d3 Nf6 5. c3 d6 6. b4 Bb6 7. a4 a6 *
[Round "63-64"]
[Opening "Ruy Lopez, Exchange var"]
[ECO "C68"]
[PlyCount "13"]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Bxc6 dxc6 5. d4 exd4 6. Qxd4 Qxd4 7. Nxd4 *
[Round "65-66"]
[Opening "Spanish, Closed - Archangels"]
[ECO "C78"]
[PlyCount "12"]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O b5 6. Bb3 Bc5 *
[Round "67-68"]
[Opening "Spanish, Open"]
[ECO "C82"]
[PlyCount "16"]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O Nxe4 6. d4 b5 7. Bb3 d5 8. dxe5
Be6 *
[Round "69-70"]
[Opening "Spanish, Closed - Zaitsev"]
[ECO "C92"]
[PlyCount "21"]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O Be7 6. Re1 b5 7. Bb3 d6 8. c3
O-O 9. h3 Bb7 10. d4 Re8 11. Nbd2 *
[Round "71-72"]
[Opening "Colle system"]
[ECO "D05"]
[PlyCount "9"]
1. Nf3 Nf6 2. d4 d5 3. e3 e6 4. c3 c5 5. Bd3 *
[Round "73-74"]
[Opening "Queen's Gambit, Slav"]
[ECO "D18"]
[PlyCount "10"]
1. d4 d5 2. c4 c6 3. Nf3 Nf6 4. Nc3 dxc4 5. a4 Bf5 *
[Round "75-76"]
[Opening "Tarrasch Defense"]
[ECO "D34"]
[PlyCount "17"]
1. d4 d5 2. c4 e6 3. Nc3 c5 4. cxd5 exd5 5. Nf3 Nc6 6. g3 Nf6 7. Bg2 Be7 8. O-O
O-O 9. Bg5 *
[Round "77-78"]
[Opening "Queen's Gambit, Semi-Slav"]
[ECO "D46"]
[PlyCount "11"]
1. d4 d5 2. c4 c6 3. Nf3 Nf6 4. Nc3 e6 5. e3 Nbd7 6. Bd3 *
[Round "79-80"]
[Opening "Queen's Gambit declined, Tartakower"]
[ECO "D58"]
[PlyCount "14"]
1. d4 d5 2. c4 e6 3. Nc3 Nf6 4. Bg5 Be7 5. e3 O-O 6. Nf3 h6 7. Bh4 b6 *
[Round "81-82"]
[Opening "Gruenfeld, Fianchetto"]
[ECO "D75"]
[PlyCount "13"]
1. d4 Nf6 2. c4 g6 3. g3 Bg7 4. Bg2 O-O 5. Nf3 d5 6. cxd5 Nxd5 7. O-O *
[Round "83-84"]
[Opening "Gruenfeld, Classical"]
[ECO "D86"]
[PlyCount "15"]
1. d4 Nf6 2. c4 g6 3. Nc3 d5 4. cxd5 Nxd5 5. e4 Nxc3 6. bxc3 Bg7 7. Bc4 c5 8.
Ne2 *
[Round "85-86"]
[Opening "Catalan, Closed"]
[ECO "E06"]
[PlyCount "10"]
1. d4 Nf6 2. c4 e6 3. Nf3 d5 4. g3 Be7 5. Bg2 O-O *
[Round "87-88"]
[Opening "Bogo-Indian"]
[ECO "E11"]
[PlyCount "9"]
1. d4 Nf6 2. c4 e6 3. Nf3 Bb4+ 4. Bd2 Qe7 5. g3 *
[Round "89-90"]
[Opening "Queen's Indian"]
[ECO "E15"]
[PlyCount "7"]
1. d4 Nf6 2. c4 e6 3. Nf3 b6 4. g3 *
[Round "91-92"]
[Opening "Nimzo-Indian, 4.Qc2"]
[ECO "E32"]
[PlyCount "11"]
1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. Qc2 O-O 5. a3 Bxc3+ 6. Qxc3 *
[Round "93-94"]
[Opening "Nimzo-Indian, Rubinstein"]
[ECO "E53"]
[PlyCount "13"]
1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. e3 O-O 5. Bd3 d5 6. Nf3 c5 7. O-O *
[Round "95-96"]
[Opening "King's Indian, Fianchetto"]
[ECO "E66"]
[PlyCount "15"]
1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. g3 O-O 5. Bg2 d6 6. Nf3 Nbd7 7. O-O e5 8. e4 *
[Round "97-98"]
[Opening "King's Indian, Saemisch"]
[ECO "E86"]
[PlyCount "14"]
1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. e4 d6 5. f3 O-O 6. Be3 e5 7. Nge2 c6 *
[Round "99-100"]
[Opening "King's Indian, Classical"]
[ECO "E99"]
[PlyCount "24"]
1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. e4 d6 5. Nf3 O-O 6. Be2 e5 7. O-O Nc6 8. d5
Ne7 9. Ne1 Nd7 10. Be3 f5 11. f3 f4 12. Bf2 g5 *
Click here to download the 50 Silver Suite positions in PGN format.
