Playtesting - an advantage or disadvantage for competitive play?

[Matt Sandi]

Really interesting comments here. For me it boils down to the following:

I'm really glad we have playtesters- the hobby seems (for me at least) getting more enjoyable every year with some great rule sets and armies being released- something for everyone I'd say, whatever style or taste. A big part of that is the time and effort GW have put in as well as the play testers themselves not to mention the community at large.

All the playtesters mentioned are all excellent players who spent years playing the hobby- this is what makes them great play testers as well as competitive players. I'd expect all the above names mentioned above to finish in the top standings regardless of whether they've playtestested a new book or not. 

It's also playtesters hobby/passion and something they've invested a lot of (personal) time in despite working in other jobs/family commitments. I stress the time elements because this relates back to  one of the many reasons why they are successful players- they shouldn't be penalised for this in the tournament setting or ostracized for gaining an unfair advantage in the 'meta' - they've given time, thought and energy for the benefit of all.

It's an awsome dice game and you can be as experienced as you like, played all the armies under the sun new or old but if the dice fail you on the day then it doesnt matter playtesters or not. There should be some trust from the community that the people play testing are doing it for the benefit of all - again there should be no penalizing in  the tournament setting for this or ostrocising through social media outlets. Whilst I'm sure everyone wants to win ( I certainly do in the very few tournaments I enter!) the most important thing is that everyone has fun, meet some new people and have some awesome games with armies they love and not get too hung up on whether someone may/may not have a slight advantage because they've played a bit earlier with an army before its released. I stress that its about trust, trust that they're doing this for the community and we are all the better for it. If they play with an army I havent had chance to experience before  then that's a great challenge for any general as well as a great opportunity to pick the brains of an experienced player and hopefully you'll learn something about yourself and your army in the process.

 

 

[HollowHills]

2 hours ago, HobbyKiller said:

3: Not being able to share what we are painting / playing with / thinking about as people infer things from it.

That's sad, but not going to lie I would 100% infer stuff from it.

[Sleboda]

5 hours ago, Chikout said:

A flat ban on new books until FAQs are out might be advisable. 

I may have to think that through a but, dang that seems brilliant.  It gives folks time to learn the new army and gives GW time to fix things. Plus it helps painters.

 

I like it!

[Sleboda]

2 hours ago, HobbyKiller said:

2: Having to essentially abandon playing ‘practice’ games for tournaments as certainly for me the vast majority of my non tournament games are playtest ones

3: Not being able to share what we are painting / playing with / thinking about as people infer things from it.

I left out your #1 because I pretty much touched on it earlier, but I wanted to say a thing on each of these. 

 

First, the #3. Yep. A very good friend of mine is a current tester and wow does he ever do a great job of keeping things to himself. I don't really poke him for info since I recall what it was like when I was a tester, so I can sympathize with how he must feel sometimes. It really is a bummer.  I don't think this is a tournament disadvantage, but I guess I just wanted to take the opportunity to praise the integrity of the current crop of testers.

 

For #2 - yeah, it stinks. I know it's come up in other threads, but it's worth mentioning here. The US is big. We don't have 20 tournaments running each weekend, all 5 minutes from our houses (exaggerating, I know, but still), and the concept of a club or store league is mostly a fantasy. Also, our friend group is likely to have its members living an hour or more apart, and most of us don't have domestic partners who also play (I'm lucky as a nurgling in grandfather's garden to have @TwiceIfILikeIt as the love of my life and constant hobby partner/nemesis so I can game pretty much at will now, but that was not the case before).

A lot of us are lucky to get in one gaming day every month or two. Having to trade in that day with a bunch of buddies for a focused NDA-fuelled "working" hobby day is a genuine bummer.

[Galas]

Whats exactly the problem here? That playtesters play the game and... win games in tournaments?

[Meatgrinder]

For anyone confused about where this came from, because all context was left out of the OP, Team England in the ETC this year is made up almost entirely of playtesters.

If ETC had run this year Team England would have walked into the event with Lumineth lists that they have been working on for significantly longer than any other team and have the experiance against them that every other team lacks. Thats by their own admission on Bad Dice by the way. This has drawn criticism from other teams.

Nobody cares about singles events, only team events where team comp is important. 

[relic456]

2 hours ago, Meatgrinder said:

Team England in the ETC this year is made up almost entirely of playtesters.

Thanks for the context, that's absolutely whack. 

Some interesting points being made. To say that playtesters can't show measurable improvement is wild to me. Yes, if a playtester is winning 4 or 5 games a day at an event, they can't start winning 6 games. But not every 5 win player is a playtester, and a skilled playtester is going to have an advantage (however slight) over those other players. So while a playtester's record might looks the same, I'll reckon that the playtesting advantage would be pushing other players down. It would be interesting to look at the win rates between historically strong players against playtesters (who were also historically strong) before and after they started testing. There are a lot of other factors at play so it's probably difficult to draw conclusions, but it seems logical to me. I do agree that the people winning the most tournaments should be contributing to the playtesting, but I definitely don't like that they then get to turn that extra experience around and use it in a tournament. Keeping armies out of tournaments until their FAQ solves this problem as well as others. 

There's also something to be said about perception. Even if the advantage is slight, it's still going to be perceived by others. I mean we're all talking about it because of that perception. 

Having your "heads" in multiple metas. I'm sorry but this one is the most silly to me. I'm sure all playtesters are smart individuals, with extreme mastery of the system, so to say that it's hard to keep the rules straight really sells you guys short! Is it that different from just playing multiple tabletop games, lots of people can keep 40K, Kill Team, and AoS straight just fine despite being very similar systems. 

"If somebody wants to make a claim that playtesters gain a significant advantage, I believe it’s up to them to prove it via data, not you lads defending yourselves with data." I think this is a really easy position to take when it's functionally impossible to do lol. Where would you even begin? Is there a list of playtesters publicly available, or would you have to rely on people self-reporting? Is taking the results of a playtester prior to their playtesting feasible? How long do people playtest for? If you've been a playtester for 2 years, how to you separate out the advantage from playtesting from just your general improvement over time?

Playtesting is a privileged position, and like any other privileged postion there's always potential for abuse and maybe we should have systems in place to limit that possibility. If that means some people don't want to playtest anymore than so be it, I imagine there are plenty of people who would jump at the opportunity.

To summarize, I think @Dan_Elkington worded it really well, it's very difficult to argue that a game is fair when you're going up against someone who has been thinking about and playing with the army for weeks (months?) longer than anyone else in the room.

As a final thought for discussion, I wonder how the growth of the TTS scene will impact playtesters in the future. It would let them get significantly more games in and hopefully result in a larger sample size for GW to draw conclusions from. But would it make their advantage larger? Maybe. 

[rbstr]

As others have said  adding a bit of delay to how soon a new book can show up at an event will eliminate the potential for advantage regardless of if it actually exists or not.

Really that's just a good place to be ethically in general, I think.

[ChrisT]

I think this is an interesting debate, and having spent a fair amount of time as a playtester in the past i think it can really vary.

The main thing for any game is knowing it inside out, including your army and your opponents.  This comes with time spent playing the game, and yes the PT are often people that play the game lots.  

However, often the role, as it suggests, is to test items out.  This may be a new rule, a new model, a new ability etc does it work, is it clear, how does it play, does it break something, is it pointed correctly.

Now fast forward 6 months to a release date, do the PT still have the advantage?  Not only have they been looking at other books in the interim and new rules etc.  They have even more knowledge and yet they still have to remember how the "current" book plays.  In some ways this can be a disadvantage.  I distinctly remember in WFB there being a few times as a group or at an event I was like that's not how the rule works, only for my opponent to say yes it here here is the book / FAQ and then I just go quiet as i realise why I the rule was not as I remembered.

Can a PT think that unit is ace i will buy one of them when they are released?  Sure, but how long does the hive mind of the internet also take to find that unit?  0.0001 seconds?

So in summary being a PT means you are a good player already but imo it does not give you a real advantage anymore.

Caveat - if the internet did not exisit then i would change this opinion.  One of my fav things was to go to a tournament our group of 5-6 players would have theorised lists etc, but before all these forums existed we were in our own bubble, you attended a GT and boom 100 players and lots of new tactics and choices were found.  That was the internet forum of the day, so back then a PT would have had an advantage as they had longer to find that combo and although lots of bubbles may have found it the internet was not as it is today with the hive mind of sharing.   However, after 1 GT this was out in the world and so people then copied it.

 

[Ravinsild]

On 6/23/2020 at 5:32 AM, Tokyo Nift said:

4) At the last 6N, Slaanesh had just received a new book shortly before list submission. As I'm sure people are aware that Slaanesh book was one of the most powerful books AoS has ever seen. Despite this, the England team did not have a Slaanesh list in their roster, suggesting that despite insider knowledge they had not had sufficient time to prepare a Slaanesh build for the current competitive team meta.

Yeah or maybe they just aren't degenerate heretics and stay loyal to the God-King Sigmar!

[amysrevenge]

14 hours ago, ChrisT said:

Caveat - if the internet did not exisit then i would change this opinion.  One of my fav things was to go to a tournament our group of 5-6 players would have theorised lists etc, but before all these forums existed we were in our own bubble, you attended a GT and boom 100 players and lots of new tactics and choices were found.  That was the internet forum of the day, so back then a PT would have had an advantage as they had longer to find that combo and although lots of bubbles may have found it the internet was not as it is today with the hive mind of sharing.   However, after 1 GT this was out in the world and so people then copied it.

 

 

This is where I come down as well.  I think the advantage of seeing a version of the book within a small circle of people for a month is likely to be close enough to the advantage of seeing the real book with the combined input of hundreds of keeners on the internet for a week.  For people who are already in the top tier of players, I mean.

 

Side note:  making an argument like "I say X; prove me wrong" is pretty cheap.  A good argument is "I say X, and here is my evidence. Counter my evidence if you dare."

[Jack Armstrong]

12 hours ago, amysrevenge said:

Side note:  making an argument like "I say X; prove me wrong" is pretty cheap.  A good argument is "I say X, and here is my evidence. Counter my evidence if you dare."

Hopefully if you go back to my post I think that's exactly what I wrote/did!

 

(I don't think you were talking about me to this point)

[Theo Jansen]

Dear Jack,

Let me preface by saying: I prefer have playtesters (PTs) in the competitive environment, over the alternative of a group not allowed in events. I think the benefits far outweigh the downsides. So I hope you keep showing up and will feel welcome to do so in the future.

By extension of that, I hypothesize the Playtesters Paradox™: To achieve a balanced competitive scene by giving the best players an advantage

Ethics in playtesting

Having said that, it does become a question of ethics, how the PT group handles this, I believe. Whether or not there is an actual advantage in being a PT, is a matter of perception. In a lot of ways (like Jack said to me), it is irrelevant if the (perceived) advantage is actually there. Fact is, a part of the community perceives your status as PT as an being advantageous. It is impossible to prove whether it is or not.

With that in mind, I am surprised by the defensiveness and victimizing is see from the PT group. Whether it be on off-hand jokes or serious topics like these. Trivializing the advantage to the point of calling it a disadvantage (evidently) feels disingenuous to a part of the community.  It helps create a divide that I don’t believe needs to exist.

I believe the PT group should ask them selves how they want to handle their position within the community. I hope that this backlash (if we can call it that) causes a discussion of ethics, introspective and morality within the PT group. It would be disappointing if the conclusion would be “let’s not talk as openly about this again”, that would reduce the level of transparency even more.

ETC

I assume the best intention, so I believe it to be unintended, but Jack’s topic  feels like a straw man for the ETC discussion. Saying having half the England team being PT is a disadvantage, while at the same time using playtesting time to prep for ETC is ethically questionable in my opinion. You can’t have your cake and eat it.

Most people have a good moral compass. They will instinctively feel that there is something wrong with this situation. Whether they feel there is a problem, differs, but most will see an ethical dilemma. By denying the dilemma is there, you imply concern for the health of the competitive scene is hyperbolic. People do find it important and the PT team, to an extent, exists because we (and the PT team and GW) do.

I will say that I think the problem is way less pronounced in singled than it is in teams events. 

Solution

I hope you’ll allow me to indulge and propose a possible solution. Proactiveness and assertiveness are great tools in situations like this. You cannot be certain whether or not there is an advantage, but you can mitigate for a possible advantage. By saying, “we as a group, will self-restrict (or self-comp, to stay in jargon), by not using new tomes for 2 months after release for singles and 4 months after release for team events, to mitigate any unintentional advantage we may gained by foreknowledge”. Help people understand that it is not your intention to take advantage, communicate about what you do in testing and what the moral values of the PT group are.

My 2 cents.

 Theo

*Full transparency: I am the ETC captain for the Netherlands and have a stake in this discussion, as I feel we need a change in that event and have been advocating that.

 

Edited June 25, 2020 by Theo Jansen

[Jack Armstrong]

Nicely balanced and thought provoking post thanks Theo.

[Ravinsild]

Well, I don't really have a dog in this fight, but any playtester are welcome at Nashcon and they can bring any book they like.

Now whether or not you want to travel to Nashville, Tennessee for Nashcon and whether or not it's an event large enough to be on anyone's radar aside from those of us who live in the Southeast USA is another question.

Still welcoming you to Nashcon.

[Dan_Elkington]

Again, there seems to be a lot of subjective commentary about whether being a playtester materialises in an advantege. This is irrelevant.

I'd love to playtest new stuff, I've a real eye for detail and rules and do some technical copywriting on the side of my job, as well as my academic work in my professional sphere.

If I were honoured to be a playtester at some point in the future, would I use the new stuff; no.

Why; because I wouldn't want my opponent to think that the only reason I won was because I had the book for several months longer than they did.

In fact, I'd probably take some meta busting lists that I knew would ****** with the new book to help the community work out how to beat the new stuff.

@Jack Armstrong

Edited June 26, 2020 by Dan_Elkington
didn't really make the point I wanted to.

[PlasticCraic]

Very good post @Theo Jansen 

I think there's a lot of related but distinct discussions we could have around the playtesting process.  Some of these are (with my opinions italicised):

Is it a net benefit to the game?  Yes, massively so.

Is it perfect?  No, just as we can't expect any process to be perfect, but we should constantly strive to improve.

Do they do a good job?  Yes, I think so, looking at some of the things they have caught.  The relevant comparison is not a subjective, imaginary perfect version of a given book, but what that book looked like before playtesting.

Is it fair to criticise the process?  Yes with an asterisk.  From someone on the outside looking in, the UK scene seems very factional and I could well imagine that criticising the process is used as a proxy for criticising the individuals involved / scoring points in personal feuds.  Hopefully when there is well-reasoned, constructive criticism of the process, that doesn't get misconstrued as criticism of the individuals.

Do the playtesters get an advantage in individual events?  Already discussed extensively in this thread and beyond, with some good points raised on all sides.  Minor if any in my opinion.

Do the playtesters get an advantage in the ETC / other team events?  Yes, and it's probably quite significant.

It's this last one that is the one that I would argue deserves most focus, given that the reason this blew up as a topic was specifically because of comments made on the Bad Dice review of the Lumineth book, by a playtester who represents England at the ETC .

In the context of ETC, having a team of mostly / all playtesters is advantageous in several ways:

• You know which way the meta is likely to go before everybody else
• This can and does inform your portfolio of armies (confirmed on the podcast)
• You know whether to invest time and tournament reps in specific armies
• You know if an army has a likely hard counter looming

All of these things are huge in an event where pairings are critical.

I think that context is something that deserves specific discussion.  For ETC specifically I would suggest an earlier cut off date for what is valid.  It's not a perfect solution, because it means you're likely playing with an "old" ruleset, but it would solve the issue of big swings in the meta right before the event that some teams knew already knew about in detail, and most teams didn't.

Finally I'd like to say that there are several red herrings that come up whenever playtesting comes up that I would like to address in advance:

"They're great players" Yes they are, and sincere hats off to them for their many achievements.  But that is tangential to this specific discussion.  You could put Usain Bolt in rocket boots, and say he would have won the race anyway, and that would be true.

"The game would be worse without playstesting" Agreed.  Suggesting that playstesters have an advantage in team events is in no way advocating for the abandonment of the entire playtesting process.

Interested to hear everyone's thoughts on the above, but especially from playtesters on whether they agree / acknowledge that they have an advantage for the ETC specifically, and what if anything they would suggest as an outcome for that.

[Sleboda]

This whole discussion around playtesters and the seriousness of tournament competitive balance reminds me of one thing above all others:

I really, strongly, totally believe that the worst thing for this hobby is the availability of prizes, either cash or cash value, at events.

[Kramer]

2 hours ago, Sleboda said:

This whole discussion around playtesters and the seriousness of tournament competitive balance reminds me of one thing above all others:

I really, strongly, totally believe that the worst thing for this hobby is the availability of prizes, either cash or cash value, at events.

Prizes for winning I agree with. 
prizes for best hobby, nicest player, subjective goals I do support. 

[Sleboda]

7 hours ago, Kramer said:

Prizes for winning I agree with. 
prizes for best hobby, nicest player, subjective goals I do support. 

Agreed, though just a point or two:

1. I hate "participation culture" in general and support rewarding actual performance. Prizes in a competition where the rules are clear and well regulated I'm ok with. From spelling to track & field, yes, award cash to the winners. Unfortunately, games of toy soldiers are never tightly written and accurately judged enough to support the clear recognition of who was best.

2. Subjective cartegories based on artistic expressions are tough to get behind as well (and this coming from a winter of a significant number of hobby prizes), but due to the inherent nature of art, I think the vast majority of participants accept that the "best" award for art is not a direct reflection on only the artist, which makes "losing" easier to take. 

3. Any subjective award that can be gamed by, say, a club to give Best X to one of its members should be eliminated.

4. Lastly, I still say that cash or cash equivalents at hobby tournaments are a bad idea. Go with trophies in all cases.

[The World Tree]

10 hours ago, Sleboda said:

This whole discussion around playtesters and the seriousness of tournament competitive balance reminds me of one thing above all others:

I really, strongly, totally believe that the worst thing for this hobby is the availability of prizes, either cash or cash value, at events.

In the UK, where the majority of playtesters appear to be based, there doesn't tend to be prizes of cash value for winning.

[Sleboda]

7 minutes ago, The World Tree said:

In the UK, where the majority of playtesters appear to be based, there doesn't tend to be prizes of cash value for winning.

Not even boxes of models or other hobby supplies? That's good to hear!

[EccentricCircle]

Regardless of how it affects matched play, I think its pretty conclusive that playtesting gives you guys an unfair advantage in the most important of all Warhammer related competitive activities: Trying to guess what will be released next!

Everyone endlessly speculating in the rumour thread must look ridiculous when you're already playing with next years armies.

[Falandris]

What I am always asking myself is how does the playtesters influence the actual written rules and how does that affect the decisionmaking of GW?

Is it something like - here are our suggested rules, dear playtesters - what do you think about them?

And what is the general approach going into playtesting. Is it mainly from a competitive point of view in mind? Is there such thing as trying to avoid powercreep? And how is the general level of other armybooks considered while evaluating the new rules? If thats a thing that is considered in the whole process - how comes there are such big differences between books like Gloomspite Gitz and for example Tzeench possible. 

So if we look at the recent releases that had a huge impact on the meta like DOK, Slaanesh Tzeench, OBR and likely Lumineth - so in these cases the playtesters came to the conclusion that these set of rules were a great way of extending the at that time actual way of playing and increasing the overall enjoyment of the game? Or is it more like that a lot of suggested changes didn't make it into the book.

Im really interested to hear more about the whole thought process to get an overall better understanding of the decisionmaking. If some of the playtesters could give some insights on this i feel like this would avoid a lot of grudge and misunderstandings.

If it doestn't really fit into this discussion here I can open up an independet thread for this topic as well. I feel like a lot of people would be interested though.

[Sleboda]

1 hour ago, Falandris said:

What I am always asking myself is how does the playtesters influence the actual written rules and how does that affect the decisionmaking of GW?

I can't claim to know current stuff, but back when I tested, we were instructed to play 2000 points, 4x8 table, pitched battle for the vast majority of our games. We were to report our lists and the result, plus highlights if possible.

We were actively discouraged from sharing our thoughts on the clarity of rules.