Playtesting - an advantage or disadvantage for competitive play?

[Jack Armstrong]

So with some chat recently about playtester advantage I wanted to do a bit more analysis.  I personally think it's a disadvantage (take LRL, I playtested them a long time ago and not against anything in the current 'meta' and had to get the book back out last week to remember what any of the scrolls/combos are).  I also don’t play many practice games in the current ‘meta’ as lots of my theory and actual games are in a meta 6-12 months forward from now.  I appreciate that some people think that it’s an advantage though and without breaking NDA’s it’s hard to have a full and open discussion about it.

Instead I thought I would have a look at statistics.  Personally, the first book I tested was Blades of Khorne and pre that in AoS I had attended 18 events with an average placing of 2.77 (including Masters/6N which isn’t on Bad Dice).  Post Blades (March 2019) I have attended 7 events with an average placing of 3 – so technically I have got worse although I am not sure it would be viewed as statistically significant.  (I attended one team tournament in March 2019 with a placing of 53^rd which would have skewed either stat so hugely I just discounted it)

Hopefully from that data you can see that my own performance hasn’t statistically changed with Playtesting.

I’ve then pulled a list from Bad Dice Rankings of the last 49 tournaments.  Of them the following Playtesters won:

Ben Curry – DoK (March 2020) - released Feb 2018

Tom Mawdsley – Cities (Nov 2019) - released Oct 2019

James Tinsdale – Slannesh (Sept 2019) released April 2018

Tom Mawdsley – LoG (Aug 2019) released May 2019

James Tinsdale – Deepkin (Aug 2019) released April 2018

James Tinsdale – Deepkin (June 2019) released April 2018

Les Martin – Deepkin (May 2019) released April 2018

Jack Armstrong – DoK (May 2019)  released Feb 2018

Jack Armstrong – DoK (April 2019)  released Feb 2018

James Tinsdale – Nighthaunt (April 2019) released June 2018

James Tinsdale – Deepkin (March 2019) released April 2018

James Tinsdale – Deepkin (Feb 2019) released April 2018

If someone wants to go back further you’re more than welcome, I ran out of energy.  If we look at what people have won with they are all books that have been out for a long time before the event.  The only one that was relatively new was Tom with Cities (released 5^th October).  Tom was however very new to the playtesting team and hadn’t had anything to do with that book.  I'm not trying to discount a statistic that goes counter to my narrative, I'm just trying to add some context that people may not appreciate.

The other data we could run is looking at top 3 placings but anecdotally I can’t think of any events where someone who playtested a book has taken it in the first couple of months and rinsed the tournament scene.  If anyone else wants to I would be interested to see it.

Any reasoned debate / analysis appreciated!

Jack

[zilberfrid]

Even if playtesting gave some advantages (you're most likely to positively review things that are your playstyle (which may shape the final version), you have more experience than others with the new faction), I don't see this as an issue. In essence, you're doing work for every player out there, and it's not cheating in any way, shape or form, just knowledge.

Especially if you speak about possible counters when the faction is out. Not to mention the final shape of the army may be different from the intermediate one.

Now if you spot a major loophole during testing which greatly upsets balance and decide not to report it, that's a different matter. If you spot the loophole, report it and it doesn't get changed, you can use it in a tournament environment, of course.

Thanks for testing for us!

Edited June 23, 2020 by zilberfrid

[Tokyo Nift]

Nice post. I think there are a large amount of factors that run contrary to the popular opinion that playtesters are getting an unfair advantage.

For context, I am a reasonably competitive AoS player who has played against England multiple times at the 6N and thus have a vested interest in all of their players being permanently banned from everything. 

1) Wrt to singles events - playtesters are already picked because they've already proven themselves to be among the best players. If you're winning/nearly winning all your events - there isn't much room for an unfair advantage anyway.

2) As discussed, new books are not playtested against the current meta - see how OBR's standing changed after Tzeentch was released.

3) The top lists and strategies for new books are very often discovered by the community before the book is even released, I can't think of a single example of a playtester breaking out a new-book-list that I hadn't seen multiple times before in a WhatsApp group.

4) At the last 6N, Slaanesh had just received a new book shortly before list submission. As I'm sure people are aware that Slaanesh book was one of the most powerful books AoS has ever seen. Despite this, the England team did not have a Slaanesh list in their roster, suggesting that despite insider knowledge they had not had sufficient time to prepare a Slaanesh build for the current competitive team meta.

5) Even if we were claiming that a playtester advantage existed, we have to think of a reasonable response. It is good for the scene to have top competitive players playtesting new books, so we have to weigh up whether the advantage these players (who are already winning basically all their games anyway) would be large enough to merit a decrease in the quality of playtesting. An extreme example would be if we banned playtesters from events, we would most likely find that competitive players would give up playtesting - and playtesters would be players who didn't go to events.

6) Any advantage of playtesting a book decreases as time goes on, the longer the book is out the more it gets "solved" by the community. If we really are worried about playtester advantage, we can set up deadlines for list submission so that there aren't any last minute army releases to which only the playtesting team could react. 

Those are my main thoughts, I think overall the most compelling evidence is from the OP - there just isn't data to support the idea that playtesters are gaining any advantage at actual AoS events.

Much love,
Jp 

[Donal]

@Ben won an event? This feels like fake news

[MrCharisma]

Interesting topic raised @Jack Armstrong.

First, i’d love to define “advantage”. At its core, prior knowledge before the market is an advantage, no matter the tournament outcome. There could be an argument that you could have done worse without that knowledge.

I imagine your problem statement is more focused on if being a play tester improves your chances of winning a tournament because of prior experience with battletomes before public release. 

The challenge is that to be a play tester you must have proven yourself to be successful competitive prior to joining as a play tester. I’m not sure if there has been a play tester who joined the team without winning an event or consistent high performer to see such a spike in from play testers “advantage”.

I’d say more data is required, especially looking at performances at tournaments within the first few months of a play tested battle tome is released. Through the play tester experience was it quicker to find the optimum build, to gain the tactical knowledge to pilot, to have the models built and ready for an event after release.

BTW I actually don’t care. I thank you all for your hard work and dedication to putting out the best product with your limited influence. 

[Jack Armstrong]

Some interesting points Mr Charisma (sorry not sure of your name).

The point about prior knowledge is only an applicable advantage if you can act on it.  I think your point about having models ready is probably very valid as we could anticipate the strong build painted and ready for release.  Then it becomes more of an advantage based on how similar the 'new build' is to the old book one.  For example Tz new Changehost is very similar model set to previous book one, any new releases we only access at the same time.  Trying to think of a book where something new became good, maybe Cities is best example (I had 18 Demi's painted and ready!!).

I was using the example of winning a tournament as it was the easiest 'stat' I could get without any specific knowledge or tool (or time) to trawl through results.  The best data would be if we could find out average game performance variance over books with prior knowledge of with the variable of how long the books been out.  If anyone could run that data it would be great and I would be really interested to see.

[Donal]

5 minutes ago, Jack Armstrong said:

Some interesting points Mr Charisma (sorry not sure of your name).

I was using the example of winning a tournament as it was the easiest 'stat' I could get without any specific knowledge or tool (or time) to trawl through results.  The best data would be if we could find out average game performance variance over books with prior knowledge of with the variable of how long the books been out.  If anyone could run that data it would be great and I would be really interested to see.

i suspect @LLV has all the relevant data in the most digestible format to (most) easily run the analysis.

Ps @Jack Armstrong Mr Charisma is Mr AoS Coach / Anthony 

[MrCharisma]

It’s Anthony / AoS Coach 😁

I’m not busting your chops on the data mate. If somebody wants to make a claim that playtesters gain a significant advantage, I believe it’s up to them to prove it via data, not you lads defending yourselves with data.

I’m sure there are some examples where a unit gained a significant benefit that wasn’t already strong in the meta. Witch Aelves is the first that comes to mind. Did a play tester get the DoK jump by having 90-120 ready upon release? I doubt a Dark Elf player ever had that many. As an example.

The big consideration is if the play tester even got to use their “advantage”. You might have playtested Tzeentch but did that play tester play with Tzeentch or even face them at a tournament post-release?
 

Like I mentioned earlier, I’m not too bothered because playtesters don’t play at events where I live.  It doesn’t impact me but I can see how people locally might perceive advantage and be frustrated. How do you solve it, ban playtesters from events? That’s a poor outcome that I don’t think many want to see. Do event organisers enforce different rules for you? Again, I don’t think that’s fair either. 

[Jack Armstrong]

I should have guessed from the avatar.

 

Yeah no problem, I thought you had some good points.  Would definitely be keen to see more data on it JP/LLV!

 

[Fairbanks]

14 minutes ago, MrCharisma said:

I’m not busting your chops on the data mate. If somebody wants to make a claim that playtesters gain a significant advantage, I believe it’s up to them to prove it via data, not you lads defending yourselves with data. 

No disrespect to what you provide to the community, but it’s disingenuous to say “well go prove it then” when we all know how little data is easily available to look at. I don’t even know who all the playtesters are, much less which books they tested.

That said, considering that playtesters are really good players anyway, and, if the terminology carries over from betatesters, are looking for things that are game-breaking, instead of playing to learn the army better, then there probably won’t be a huge gap between them and the rest of the field.

[Dunmer_SC2]

First off I will say that competitively apt players are needed as playtesters to help sort out bad play experiences that can arise from abusive rules. As an aside I also think some casual players should be involved in playtesting, in fact I would say a whole spectrum of players should be testing to help balance the narrative and meta balance of any book but thats on the company GW itself to think about. So its not an argument for this post.

Concerning whether or not you get an advantage, I see above Mr Charisma talking about prior knowledge of how an army plays being an advantage. As you pointed out its only valid if you can act on it and regarding the recent podcast, the playtesters were talking about aetherquartz giving you re rolls when now we know its just a +1 modifier. I do not if it was a mistake by them but I am guessing that the playtesters version of the rules could be an evolving rule set as balance and rules interactions is sorted out. So I also agree that playing the army early can be a very confusing and challenging thing when then entering the "meta" that the rest off us have. So it is debateable if thats a real advantage.

As for the results, you shouldnt look at 1st places as a metric, especially in this enviroment. Top 8 scores should be considered and the lists used to get those scores and also the timing off those events. You might win an even with an older book, but lets say FEC was just released and you know how to beat it. So as I will explain below thats the real advantage(if any) here. Looking at Bad Dice rankings and from the playtesters we know and the lists used at events then you can get real empirical data to support or debunk this whole thing. I had a quick look and I dont know when books were released so I cant make any real judgement. Its also unfair because playtesters are good players so will probably do well anyway so really this whole thing is objective.

But lets roll back the clocks, we have had several tournaments where a player(doesnt even matter if it was a tester) bring a hard as nails list and smash everyone out of  the water. Happens all the time and nobody can deny that, now this is what I call a meta busting list and its real power comes from your opponent not knowing what the list can do/ not knowing how to deal with it/also lists arent equipped for it. Gavriel surcharge was a dominating list but when people figured it out and learned what it does and practiced against it and brought screens. That list fell back into mediocry.

So my point to this is the advantage of knowledge prior to everyone else. I do alright at tournaments, playing with the NI and Ireland 6 nations guys. I played the 2018 Facehammer Champion as a practice partner for tonnes of games prior to going. This was the 90 witch elves Hagg Nar list, first off its kind and still today is a pretty strong list. That army tabled plenty of people because like I said above, they did not know what the list did and how it functioned with buffs etc.  So in turn you do not know how to take it apart and dismantle it and most importantly peoples lists were not equipped to deal with it. I played a DoK list while at facehammer and I won confidently against another good DoK player, I won because I knew his army and what to do with it, adding to this I brought a list nobody had seen before and many people spend 10 mins at the start of the game listening to me explain the rules interactions. This is a personal experience but I am only using it to show an advantaged gained by bringing not only a new list but also the knowledge of knowing said army.

Of course we have all had similiar experiences where you dont know what something really does and make a mistake, or you win because of said mistake. It happens, but when you are playtesting an army. You know the top lists coming out off it and you know how to play, what they can achieve, their shortcomings, what you need in your list to deal with these.

And the most important part as you guys mentioned is just having the models painted and ready. Which is a situational advantage but majority of people dont have time to paint a new tech choice for their list in the 2 weeks after a book drops and lists are figured out.

Its a bit of rant and by no means a perfect argument but I feel the dust lies somewhere in the middle. Should playtesters be allowed to play in events? Of course.

Should everyone be allowed to play an army so close to a book dropping at an event? Well thats where we find this balance I think. Make it 2-3 weeks after a full army is release and then everyone has had fair game to design lists and counter play them. So I think its on the TO's to mitigate this whole thing, whether you get an advantage or not becomes irrelevant when we all have time to build our lists and get some practice.

As an aside I have played Team England twice at 6 nations and played plenty of the old and current English players at events. From the guys I have played and meeting some, you can tell they are good at the game and even looking at my batshit crazy lists they can piece together a plan in their heads so the top players do invalidate my arguments. Like I said though there is a bit of meeting in the middle here, most of the guys I met I got on pretty well with. Playtesters are good for the game and we can get rid of any advantage real or not by adding one or 2 weeks to the cut off points on new books or the fully released new armies.

[Sleboda]

Just for me, personally, when I used to be a playtester in the old Warhammer days, I found it to be a small disadvantage in two ways.

1. I ended up with six versions of rules in my head (ushabti with varying rules based on their heads in an example). It's bad enough keeping rules from different games and different editions straight. Add in rules that came and went half a dozen times for just one army and it gets crazy for me.

2. A strange variation on Cassandra Complex. Sometimes I would know with absolute certainty how a rule is played, but an opponent (or worse yet, judge) would want to tell me it is otherwise. It's really hard to not pull out the "look, trust me, I playtested that rule and talked to the author" card. 🤪

Edited June 23, 2020 by Sleboda
Typo

[Tokyo Nift]

38 minutes ago, Jack Armstrong said:

Would definitely be keen to see more data on it JP/LLV!

 

Honestly, the data is probably pretty straightforward but limited. We essentially can't measure the improvement of top players because they have nowhere to go.

Playtesters are all averaging above 4 wins per event both before and after playtesting. This means that they're essentially not dropping games at events regardless of the circumstances, so the questions:

1) Do playtesters perform better than other players with new factions compared to their previous results with other factions?

No. Non-playtesters hopping onto newly released factions are a place we can see players get a big jump in their winrate (if the new faction is very strong, e.g. Ironjawz players moving to Big Waaagh after Orruk Warclans came out). Playtesters, because there is essentially no measurable way for them to improve, stay the same (4.x win average).

2) Do playtesters perform better AGAINST new factions than other players?

No, again, because they can't show measurable improvement. The factions/players they lose to are rare and the games they lose will almost always be to other top players - and player skill is a much better winrate predictor than any faction information.


So really what we're asking is if there is a marginal gain to playtesters that is worth less than the 0.5 games they lose on average at an event. If there is, it would be smaller than the variance in game results due to battleplans, priority rolls, etc - so for me I would class that as statistically irrelevant, given that there is no useful way to apply that information.



 

[HollowHills]

2 hours ago, Jack Armstrong said:

So with some chat recently about playtester advantage I wanted to do a bit more analysis.  I personally think it's a disadvantage (take LRL, I playtested them a long time ago and not against anything in the current 'meta' and had to get the book back out last week to remember what any of the scrolls/combos are).  I also don’t play many practice games in the current ‘meta’ as lots of my theory and actual games are in a meta 6-12 months forward from now.  I appreciate that some people think that it’s an advantage though and without breaking NDA’s it’s hard to have a full and open discussion about it.

Instead I thought I would have a look at statistics.  Personally, the first book I tested was Blades of Khorne and pre that in AoS I had attended 18 events with an average placing of 2.77 (including Masters/6N which isn’t on Bad Dice).  Post Blades (March 2019) I have attended 7 events with an average placing of 3 – so technically I have got worse although I am not sure it would be viewed as statistically significant.  (I attended one team tournament in March 2019 with a placing of 53^rd which would have skewed either stat so hugely I just discounted it)

Hopefully from that data you can see that my own performance hasn’t statistically changed with Playtesting.

I’ve then pulled a list from Bad Dice Rankings of the last 49 tournaments.  Of them the following Playtesters won:

Ben Curry – DoK (March 2020) - released Feb 2018

Tom Mawdsley – Cities (Nov 2019) - released Oct 2019

James Tinsdale – Slannesh (Sept 2019) released April 2018

Tom Mawdsley – LoG (Aug 2019) released May 2019

James Tinsdale – Deepkin (Aug 2019) released April 2018

James Tinsdale – Deepkin (June 2019) released April 2018

Les Martin – Deepkin (May 2019) released April 2018

Jack Armstrong – DoK (May 2019)  released Feb 2018

Jack Armstrong – DoK (April 2019)  released Feb 2018

James Tinsdale – Nighthaunt (April 2019) released June 2018

James Tinsdale – Deepkin (March 2019) released April 2018

James Tinsdale – Deepkin (Feb 2019) released April 2018

If someone wants to go back further you’re more than welcome, I ran out of energy.  If we look at what people have won with they are all books that have been out for a long time before the event.  The only one that was relatively new was Tom with Cities (released 5^th October).  Tom was however very new to the playtesting team and hadn’t had anything to do with that book.  I'm not trying to discount a statistic that goes counter to my narrative, I'm just trying to add some context that people may not appreciate.

The other data we could run is looking at top 3 placings but anecdotally I can’t think of any events where someone who playtested a book has taken it in the first couple of months and rinsed the tournament scene.  If anyone else wants to I would be interested to see it.

Any reasoned debate / analysis appreciated!

Jack

I guess it depends.

The more you play a given army then the better you will be with them, that seems like a fair assumption. A playtester can have a large headstart in getting familiar with the way and army plays. Presumably they can also keep playing that army amongst other playtesters after official testing ends. (I imagine most of you keep a good supply of empty round bases).

On the other hand, you might have different rules in your head (as mentioned before). Plus we all know stuff slips through the net rules wise and that sometimes what seems strong at release turns out a bit differently down the line.

We also don't know the release order. An army like LRL might be tested a long way in advance and then again as release looms. A release that is a new version of an existing army might need much less testing. So LRL could have been first tested a year ago, but then Tzeentch tested only 6 months ago. Tzeentch then comes out first. 

By default your playtesters are going to be some of the most dedicated players and those who attend a lot of events. I dont see any of them hoarding special knowledge or secrets that others can't work out within a month or two of a tome being released.

New armies going into their first tournaments also have the element of surprise. I.e. James knew how Idoneth played as well as the current meta,  LoN / DoK. Whereas the DoK and LoN players would be less familiar with IDK.

Even armies which stay at the top often see a decrease in win ratio after the first few events.

[Dunmer_SC2]

10 minutes ago, HollowHills said:

I guess it depends.

The more you play a given army then the better you will be with them, that seems like a fair assumption. A playtester can have a large headstart in getting familiar with the way and army plays. Presumably they can also keep playing that army amongst other playtesters after official testing ends. (I imagine most of you keep a good supply of empty round bases).

On the other hand, you might have different rules in your head (as mentioned before). Plus we all know stuff slips through the net rules wise and that sometimes what seems strong at release turns out a bit differently down the line.

We also don't know the release order. An army like LRL might be tested a long way in advance and then again as release looms. A release that is a new version of an existing army might need much less testing. So LRL could have been first tested a year ago, but then Tzeentch tested only 6 months ago. Tzeentch then comes out first. 

By default your playtesters are going to be some of the most dedicated players and those who attend a lot of events. I dont see any of them hoarding special knowledge or secrets that others can't work out within a month or two of a tome being released.

New armies going into their first tournaments also have the element of surprise. I.e. James knew how Idoneth played as well as the current meta,  LoN / DoK. Whereas the DoK and LoN players would be less familiar with IDK.

Even armies which stay at the top often see a decrease in win ratio after the first few events.

As I suggested, just extend the cut off points for books and any advantage whether percieved or real will be negated.

Then we can all go back to rolling dice and wait for this whole thing to blow over.

[Landohammer]

Lots of good questions being asked here. I just wanted to throw in my two cents:

I don't have rule playtesters in my meta so its not entirely relevant to me, but if this is truly a problem, then I think there is a simple answer: 

Competitive tournaments should require armies to be painted and based. Or at least you shouldn't be eligible for prizes unless your army is fully painted.

Hear me out. I know this is controversial for many reasons.  But it serves as a pretty strong mitigating/delay factor for new armies crushing a meta. And a player with insider information can't paint a model he doesn't own yet. 

Also, while we have the attention of playtesters...

Can someone give me the forum names of the guys who playtested Slaanesh? I have some very mean emojis to send them. 😤😡🤬😠😫

 

 

[Jack Armstrong]

Really interesting points thanks JP.

 

[Sleboda]

18 minutes ago, Landohammer said:

Competitive tournaments should require armies to be painted and based. 

Remove "competitive" and add "by the player" to the end and I'm with you.

[Landohammer]

11 minutes ago, Sleboda said:

Remove "competitive" and add "by the player" to the end and I'm with you.

Haha that probably requires a thread of its own but I don't necessarily disagree.

Personally I prefer to paint my own stuff, but I take no issue with people hiring others to paint for them. The end result is a painted army on the other side of the table (and a paid artist)

And it still serves the same purpose to mitigate meta shock. Even a speed painter will take a while to assemble and paint a full army. 

Now whether those folks are eligible for painting awards is another story. But policing that is a challenge many TOs aren't interested in pursuing. 

[Jack Armstrong]

I might be missing some events but I think it's pretty standard in the UK now to have 100% painted and based armies at all events.  There might be the odd one dayer where people can bring unpainted armies but that's the exception.

 

Definitely something that's massively improved since old Warhammer days.

[Chikout]

I think the problem is the perception of advantage rather than the reality of it. It is probably down to the ndas you have to sign but there is a fair amount of mystery surrounding the playtesting process. How far in advance you playtest, how much you playtest, how much influence you have over the final product and whether you get early access to a finished book is all a bit of a mystery. 

It is my opinion that if you are paid by gw it probably reasonable that shouldn't get to use new books at a tournament. As volunteers it becomes a very different discussion. It is good for all of us if playtesters are good at the game. Putting restrictions on playtesters might stop top players from volunteering. 

People like Rob are very interested in preserving the integrity of tournaments as sporting events and there's certainly a debate to be had about whether that's the direction things should be headed but if you want to ensure a level competive playing field it is probably better to err on the side of caution. 

A flat ban on new books until FAQs are out might be advisable. 

[Dan_Elkington]

It's a basic question of fairness. Playtesters have an advantage as they know the rules beforehand. Whether they can capitalise on that advantage is an irrelevance.

It's interesting you display a load of wins and say that playtesters haven't got an advantage because they win with relatively old books, you're making the case that by not playing the new stuff, it won't affect their performance.
 

Therefore, playtesters not being able to enter a tournament with the new book doesn't cause them a detriment and eliminates the potential for an unfair advantage.

 

Put another way, if it's no Biggie to play with the new stuff, then why argue that you should be able to, when you could easily not, and protect your reputation.

 

[Wayne the game]

I would like to add that while playtesters are mostly top players some are not. This is to create balance as the 'fun players will always look at books differently. When the skaven book was being tested I played more games than anyone else on the team but was the lowest placed skaven player at the first tournament I attended. My point is that top players are just that and will continue to be so while they put the effort in that they do.

[LLV]

6 hours ago, Donal said:

i suspect @LLV has all the relevant data in the most digestible format to (most) easily run the analysis.

Ps @Jack Armstrong Mr Charisma is Mr AoS Coach / Anthony 

The topic is a bit of a minefield so I’ll stay clear of that part. 
 

ive got a bunch of untapped data in though that would likely be helpful in determining lots of things and playtesters are and always have been welcome to reach out and I’ll be happy to work with them for the betterment of the game in general. 

[HobbyKiller]

You say that we have an advantage because we know things before other people and that side of the equation is indeed true. However what you are maybe not aware of or do not fully appreciate the impact of is the disadvantages that we also incur, which I will list below.

1: Having to have in our heads multiple iterations of books with only a broad idea of release schedules, this means when we playtest we are a period of time in the future but also factoring in the current meta and books we have completed. But when we are playing normal games we have to adjust to a different environment.

2: Having to essentially abandon playing ‘practice’ games for tournaments as certainly for me the vast majority of my non tournament games are playtest ones

3: Not being able to share what we are painting / playing with / thinking about as people infer things from it.

Having said all of that we all love being playtesters but it is hugely impactful on our personal hobby and several people have stepped down / taken a break from it for that very reason.

GW are not going to reveal the process and nor should they - it is their intellectual property in exactly the same way that Pepsi don’t reveal how they develop and test new products, am not really sure why people think they should - the only thing I will say is that we don’t get paid to do it and do it for the good of the hobby and our love of it.