Statistical Deduction

Quantifying the Most Skilled Format of All Time

Players of the mid to late aughts typically contend that the formats they competed in were among the most skill based and enjoyable in the history of the Pokémon TCG, as evidenced by the intricate and diverse strategies of the time, made possible by the modalities of consistency at one’s disposal (namely Pidgeot RG, Holon Transceiver, Claydol GE, and Uxie LA).

Jak Stewart-Armstead argued in “Power to the People” that a lack of similar consistency in recent years has led to a more open game, where less versed players have a better chance of succeeding. Brit Pybas countered this assertion in his recent article, “A Deserving Draw,” by making the case that the consistency of today is comparable to that of the late 2000s because the same players continue to do well. How else could sustained success be reasonably explained?

This made me then wonder if the top players have indeed maintained excellence. Is there any way to verify Brit’s statement? It may seem impossible or at best subjective to prove such a postulate, but here’s my take on it:

If a format, probably better termed “competitive environment” for the purposes of this investigation, is truly skill based, then we should see a higher percentage repeat qualifiers at the World Championships for that season.

Once a player qualifies for Worlds at least once, I think that usually shows they are a player of above average skill. Skilled players don’t typically regress in adeptness unless they lose interest in the game. And a full season should be ample time for variance to play out and allow players to prove themselves as being world-class competitors that year.

By good fortune, there are actually incredibly thorough records of the participants in nearly every World Championship to date since Nintendo took over the game. With this data in hand, links to which are listed here, I began my search for the most skilled format.

The Premises

command center organization map computerpokemonscreenshots.tumblr.com

1. United States only.

I began recording participants from all countries, but quickly realized the data would not be very telling considering how few invites certain countries receive (meaning their sample size is probably too small to be statistically significant), plus certain countries have received notably more or less invites over the duration of the World Championships. Brazil, for example, had 13 attendees in 2013, but none in 2011. That’s a big jump! Also considering the differences in invite and tournament structures between nations, I think it makes sense to hone in on the country that has had the most continuity in organized play.

2. Masters only.

The Masters Division is the age group most likely to keep a somewhat consistent pool of players, as Masters have more purchasing power and a greater ability to travel to tournaments (plus there is no division above Masters to graduate into). Lower attrition is relevant.

3. Repeat attendees means repeat Masters attendees.

I don’t know how valid it is to compare Juniors and Seniors Worlds qualifications to those of Masters, so I prefer to keep them separate. I’m sure the accomplishments of select Seniors could place them at the level of Masters before aging up, but I’m not one to make judgement calls on the data.

4. The data from 2004 is incomplete.

I could only find a list of invitees (not attendees) along with the top 32 bracket from 2004. This means most last-chance qualifiers are excluded and the list of invitees may not have been thorough. However, being as this event was 10 years ago and 68 US players were still accounted for, the missing entrants shouldn’t have much pull.

5. The data from 2005 might be missing a few dropped players.

I have access to the final standings from this year, which do not include players who did not finish from the tournament. I was able to pick up a few missing names though, thanks to Ross Cawthon’s efforts 9 years ago. It’s unlikely there is more than a small handful of US players missing from the data.

6. The data from 2005-2013 accounts for all attendees, but not invitees.

This includes players who made it through the last-chance qualifier. Ideally I’d like to include invitees in the analysis, since they too performed at a high level throughout the season, but I think relatively few US invitees do not attend Worlds (as the event has always been held in North America, not too far from home). It might be possible to determine the non-attendee invitees by looking at the old rankings data, but I did not put forth the effort to do so. (I welcome any readers to take up this task.)

7. The data from 2014 accounts for invitees, but not attendees (yet).

Shortly after the player roster goes up on the 16th, I will revise the data and include last-chance qualifiers.

The Data

The complete spreadsheet can be found here. 1 stands for attended, 1 stands for invited (but not attended), and 0 stands for no attendance (or invitation known of).

Sorted by year, it’s difficult to glean anything meaningful from the data. But let’s try sorting by the percentage of 1+ repeat attendees (meaning they’ve qualified for Worlds at least once prior to that year):

Year Players 1+ Repeat 2+ Repeat
2007 17 82.35% 58.82%
2008 43 67.44% 39.53%
2009 46 58.70% 41.30%
2014 64 56.25% 31.25%
2010 41 56.10% 43.90%
2006 68 52.94% 16.18%
2013 61 50.82% 26.23%
2011 52 48.08% 25.00%
2012 42 47.62% 35.71%
2005 89 32.58% 0.00%
2004 68 0.00% 0.00%

I’ve included the number of players in the table because I think this stat is relevant for discussion. As less invites are available, we should expect to see a higher percentage of repeat competitors. The reason for this is that the most skilled players should, in theory, take the higher ranking spots. Less-seasoned players should fall outside the upper echelon and occupy more invites on the fringe of the cutoff. This is why we see such a staggering number of repeat competitors in 2007; only 17 invites were given to US players and a mere 3-of them were taken by first-time competitors.

This doesn’t necessarily mean that 2007 was the most skilled format of all time, as it’s unfair to compare it to years with triple the amount of invites, but it does show that the veterans distinctly rose to the top. It could also be telling us that few fresh faces attempted to qualify for Worlds this year. I was a good player back then and I remember being disheartened by the drop in invites, to the point where I questioned whether it was worth to effort to try and secure one. I’m sure I’m not the only person who felt this way. The task would be even more daunting to a player who had never qualified before.

Next, let’s talk about the early years: 2004, 2005, and 2006. Clearly, what we should expect to see over time is an increase in the percentage of repeat attendees, assuming players stick around and continue to compete at a high level each season. And that is exactly what happens here. With more first timers qualifying each year, the pool of past qualifiers increases, though the amount Masters Division players also increases (because of Seniors aging up and plain old newcomers to the game). These factors combat each other in a way I don’t totally understand.

I’m not sure at what point the percentage of repeat attendees should stabilize, but considering the anomaly of 2007, from then onward is probably where we should focus our attention. These first 3 years of Worlds serve as bedrock for this discussion and unfortunately cannot be scrutinized for skill (as I will further demonstrate in a few moments).

With that said, let’s check out 2008, 2009, and 2010. This is the time period which is most interesting to me. The number of players is pretty similar during this 3-year span (41, 46, and 43), so I think that lets us make sound comparisons. 2008 had the highest percentage of 1+ repeat attendees by a fair margin, despite being the earliest of the 3 years. This is telling to me because we would normally not expect this to be the case; I would think the percentage would continue to rise slightly or start to hover if the skillfulness remained static. The player base has grown at a fairly steady rate over the years, which contributes to my expectation. Also, 2007 added so few new invitees to the mix, which makes the numbers for 2008 even more striking. We instead see a drop in repeat competitors, though percentages for 2009 and 2010 are still relatively high.

Where we can make the most revealing comparisons, I believe, are among years 2008, 2010, and 2012:

  • 2008: 43 players, 67.44% 1+ repeat
  • 2010: 41 players, 56.10% 1+ repeat
  • 2012: 42 players: 47.62% 1+ repeat

It’s dubious to relate 2011, 2013, and 2014 to 2008, 2009, and 2010 because of the discrepancy in attendance numbers (as we talked before about the type of player who is likely to be earning the outlying invites), but 2012 had extremely similar attendance numbers to two of those years, so I think it’s reasonable to point out the drop.

Yes, the player base has grown over time (US Nationals attendance numbers are evidence of this), which should at least partially explain the drop. But we have the competing factor of more first-time qualifiers in the competitive midst, so it does seem suspect, at least to me, for the percentage to fall in such a manner. This isn’t to say first-time qualifiers aren’t skilled; they may very well have been the more driven and competent players at the time, and there are undoubtedly many other factors involved (like the proliferation of Pokémon TCG information available on the internet).

The 1+ repeat percentages from 2011 and 2013 are near that of 2012, which then lets us extrapolate and assume there might have been more variance those years as well. The percentage of 1+ repeats for this year, 2014, looks comparable those from 2008-2010, yet with players, but might this be the result of first-time qualifiers from the previous few years?

Let’s try sorting the data by the percentage of 2+ repeat attendees (meaning they’ve qualified at least two times prior, which hints they are even more likely to be highly-skilled players):

Year Players 1+ Repeat 2+ Repeat
2007 17 82.35% 58.82%
2010 41 56.10% 43.90%
2009 46 58.70% 41.30%
2008 43 67.44% 39.53%
2012 42 47.62% 35.71%
2014 64 56.25% 31.25%
2013 61 50.82% 26.23%
2011 52 48.08% 25.00%
2006 68 52.94% 16.18%
2005 89 32.58% 0.00%
2004 68 0.00% 0.00%

This is another case where I’d expect the percentage to rise, if the variance of the game remained about the same and the growth of the game was steady, since the pool of 2+ repeat players will increase each year. And that’s exactly what we see from 2008 to 2010. There is then a pretty significant drop-off of 2+ repeat player between 2010 and 2011: 43.90% to 25.00%. To me, that indicates something changed. It seems odd for such a drop when Worlds (and Nationals) attendance didn’t increase that much.

2012 then follows the theory that the less participants, the more that will be high-level players. 2013 sets a somewhat radically new benchmark for invites (compared to 2012), and 2014 builds on the number of 2+ repeat players.

Years 2004, 2005, and 2006 are moot for discussion when sorting in this fashion. They serve as years in which players begin to establish themselves.

Conclusion

Does this data mean anything? I’m not sure. I am impressed by 2008 though. The numbers don’t necessarily mean the format, defined by the cards and strategies available, was that skillful, but rather the competitive environment was prime for the pro players. The scene at the time really did allow good players to pull through.

For any readers who didn’t play back then, 2007-2008 was the season of . It emerged as the clear-cut BDIF during Cities and never relinquished its crown. Some players used Gardevoir the entire season, which, as one could image, allowed them to become very skilled with it. The mirror match was strategically nuanced and swayed by a few card choices here and there, most of which the average player was unknowing of as the deck evolved throughout the season. With enough practice, I believe Gardevoir was a favorite over almost every other deck. The culmination of Jason Klaczynski winning Worlds with Gardy that year seems fitting when discussing what I am coming to believe was the most skilled format of all time.

This doesn’t mean 2008 was the most enjoyable format, however, unless you find pleasure in playing mirror matches over and over. For anyone looking to build old decks, you’ll probably want to try 2007 or 2010. Jay Hornung is actually writing a comprehensive article on the 2010 format, the first part of which is located here, to help get you started.

Reader Interactions

Leave a Reply

You are logged out. Register. Log in. Legacy discussion: 10