AoS Statshammer: A tool for caclulating and comparing damage outputs for Warhammer Age of Sigmar units.

[Inquisitorsz]

Awesome stuff
Love your work

Here's some feedback: 

• I see you've already planned defensive abilities (like reroll saves or ethereal) which is awesome. 
• I don't see a way to rename the weapon profiles? But that's very minor
• Would like a toggle to turn modifiers on and off.... just like the whole weapon profile, but for modifiers, or for each modifier. Might get a bit too cluttered? 
• Would be great to see variance on the graphs.... like for things that do D6 mortal wounds on a 6 to hit with 1 attack.... the average isn't very useful. Perhaps the line graph can have variance bars? 
• Seems there's a limit for 5 units at a time. Makes sense for the graphing, but if you're trying to compare different units and different options, can reach that limit quickly (eg, Drycha and her 2 angry forms and various wargrove bonuses). Would be nice to maybe increase the limit to 8 or something? 
• Finally, this is probably a hard one, but would be cool to put in a monster damage table and see how their output varies vs wounds taken. I guess you could currently just have separate profiles for each damage level

Keep up the great work! 

Edited December 17, 2019 by Inquisitorsz

[Kadeton]

1 hour ago, Inquisitorsz said:

Would be great to see variance on the graphs.... like for things that do D6 mortal wounds on a 6 to hit with 1 attack.... the average isn't very useful. Perhaps the line graph can have variance bars? 

This is exactly what I was thinking. People focus on averages far too much in statistical discussions (because they're much more straightforward to calculate), and the importance of variance gets lost along the way. It would be awesome to see a statistical tool that made this information easily accessible!

[NoMaDhOoK]

Thanks for the feedback. Here are some responses / thoughts

2 hours ago, Inquisitorsz said:

• I don't see a way to rename the weapon profiles? But that's very minor

This would be very easy to add, so I will add an optional name field for them

2 hours ago, Inquisitorsz said:

Would like a toggle to turn modifiers on and off.... just like the whole weapon profile, but for modifiers, or for each modifier. Might get a bit too cluttered? 

I'll look into this, though I'll just have to make sure the UI doesn't get too cluttered

2 hours ago, Inquisitorsz said:

Seems there's a limit for 5 units at a time. Makes sense for the graphing, but if you're trying to compare different units and different options, can reach that limit quickly (eg, Drycha and her 2 angry forms and various wargrove bonuses). Would be nice to maybe increase the limit to 8 or something? 

I can look into increasing the limit. The limit was also there to avoid cluttering the table too much (especially for mobile). I may look a transposing the table (swap the rows and columns) to see if that helps. Essentially this means the table will grow in the number of rows as the number of units increases (as opposed to growing in the number of columns like it currently does)

2 hours ago, Inquisitorsz said:

Would be great to see variance on the graphs.... like for things that do D6 mortal wounds on a 6 to hit with 1 attack.... the average isn't very useful. Perhaps the line graph can have variance bars? 

CC: @Kadeton. Yeah, while average is the best metric to evaluate the performance of a unit over all the possible games it will ever be used in (as it will always eventually average out), you are right in that it doesn't give a good indication of the consistency of the unit within a particular game. After I have got through the other needed changes (essentially a 1.0 release), I do plan on looking at other statistical metrics (range, and std. dev.).

Just going to think out loud here in case someone has better ideas:

The difficulty with variance and standard deviation is that it contains the sum of all the possible outcomes within a given sample. This means that there are 2 options:

1. Simulate x number of outcomes (between 1 000 and 10 000) and use those outcomes as the sample.
2. Plot all of the possible outcomes (Essentially building a probability tree / event chain).

Both of these options are quite difficult (and computationally expensive) due to how modifiers can interact with the chain, especially considering the current method for the average calculation is not done through simulations.

It is something I am looking into, though it is probably a ways off, and most likely will not be in the auto-updating graph (like current average is), but rather a separate "Advanced statistics" button that will give you the extra metrics.

[Kadeton]

I'll see if I can come up with a less computationally-intensive option to give you the SD of a complex set of linked rolls and modifiers.

[NoMaDhOoK]

7 minutes ago, Kadeton said:

I'll see if I can come up with a less computationally-intensive option to give you the SD of a complex set of linked rolls and modifiers.

Awesome, thanks. I'll do some more thinking on it as well. I do agree that it would be a great addition (just going to get PDF export done first)

[Inquisitorsz]

15 hours ago, NoMaDhOoK said:

CC: @Kadeton. Yeah, while average is the best metric to evaluate the performance of a unit over all the possible games it will ever be used in (as it will always eventually average out), you are right in that it doesn't give a good indication of the consistency of the unit within a particular game. After I have got through the other needed changes (essentially a 1.0 release), I do plan on looking at other statistical metrics (range, and std. dev.).

Just going to think out loud here in case someone has better ideas:

The difficulty with variance and standard deviation is that it contains the sum of all the possible outcomes within a given sample. This means that there are 2 options:

1. Simulate x number of outcomes (between 1 000 and 10 000) and use those outcomes as the sample.
2. Plot all of the possible outcomes (Essentially building a probability tree / event chain).

Both of these options are quite difficult (and computationally expensive) due to how modifiers can interact with the chain, especially considering the current method for the average calculation is not done through simulations.

It is something I am looking into, though it is probably a ways off, and most likely will not be in the auto-updating graph (like current average is), but rather a separate "Advanced statistics" button that will give you the extra metrics.

Cheers for the responses. 
I think you're spot on in your thinking  here. 
In terms of variance I guess there's 2 main aspects. What is practically useful information, and what is mathematically correct information?  
It's pretty interesting to see the full maths behind all these stats/probability, but ultimately that's not what the average viewer needs to see. Knowing the mean/median/mode, SD, normal/non-normal distribution, variance and variability etc.... doesn't tell most gamers anything.   
We need some way to easily compare different weapon options or units.  

I wonder if doing something like displaying the min/max of any D6 attacks or D6 damage stat would be enough.   
For example. If I have 4 attacks, 3+,3+,-1,D6... you could work out the average as normal, up until the damage calculation.... and then display the results for both damage 1 and damage 6. That kind of gives you an "averaged min/max" right? 
And either display that as two separate lines, or as a single average line with min/max variance bars. 
Of course that might get too complicated if you have multiple variable stats like D6 attacks that cause D6 damage. But I guess that's kind of like the event tree that you mentioned.  

Simulating 10,000 outcomes is probably the easier way to do it though, but I'm not sure how that would work on the back end of the app/website. Perhaps have an "advanced stats" menu like you said, and have that run a calculation rather than a live updating table. 

ultimately I think it would be useful to see how reliable/consistent some of these monsters are (coz it's usually monsters that have these kind of random stats). 
it's going to be a fine balance between being too mathematically heavy (both in computation and explanation) and how the data is displayed. 
I'm sure you're aware that interpretation and correct display of data is just as important as actually calculating it in the first place. 

[NoMaDhOoK]

Thanks for the feedback

8 hours ago, Inquisitorsz said:

I wonder if doing something like displaying the min/max of any D6 attacks or D6 damage stat would be enough.   
For example. If I have 4 attacks, 3+,3+,-1,D6... you could work out the average as normal, up until the damage calculation.... and then display the results for both damage 1 and damage 6. That kind of gives you an "averaged min/max" right? 
And either display that as two separate lines, or as a single average line with min/max variance bars. 
Of course that might get too complicated if you have multiple variable stats like D6 attacks that cause D6 damage. But I guess that's kind of like the event tree that you mentioned.  

I was thinking of doing this, however, unfortunately there is also the matter of the Exploding, Mortal Wound, and Conditional Bonus modifiers that occur on a specific roll. This introduces another layer of variance. 

8 hours ago, Inquisitorsz said:

Simulating 10,000 outcomes is probably the easier way to do it though, but I'm not sure how that would work on the back end of the app/website. Perhaps have an "advanced stats" menu like you said, and have that run a calculation rather than a live updating table. 

Once the next update (PDF) is out (shouldn't be too long, I'd say I am around 60-70% done with it), I will mess around a bit and see if I can get it working. I agree that the simulation is probably the easiest, and I could try and use parallelism to speed up the generation of the outcomes (Since each generation is independent).

8 hours ago, Inquisitorsz said:

ultimately I think it would be useful to see how reliable/consistent some of these monsters are (coz it's usually monsters that have these kind of random stats). 

Definitely. Another good example for this requirement is the Celestar Ballista, which has a pitiful mean of 2.59 damage against a 4+ save. But this is because it scores D6 hits per hit, resulting in a massive variance (It's max damage is actually 24 damage). Nurgle suffers from this as well with a lot of their damage coming from D6 rolls.

8 hours ago, Inquisitorsz said:

it's going to be a fine balance between being too mathematically heavy (both in computation and explanation) and how the data is displayed. 
I'm sure you're aware that interpretation and correct display of data is just as important as actually calculating it in the first place. 

Another advantage of using simulations is I could then even plot a full probability curve if I wanted to (Giving a really good indication on the damage spread) 

Edited December 18, 2019 by NoMaDhOoK

[Ineffectual Clawlord]

I just wanted to congratulate you on your work, it's very impressive!

I normally do these calculations by hand, and I can definitely say that this is faster while at the same time giving an option to apply all the special modifications that a lot of other sites/apps don't provide. I'm looking forward to see how this work progresses.

[Boar]

Nice. I wanted to do something similar, kinda got sad for a moment that someone done this already. Still not going to surrender, best way to learn something new is doing something tied to your interests.

8 hours ago, NoMaDhOoK said:

Definitely. Another good example for this requirement is the Celestar Ballista, which has a pitiful mean of 2.59 damage against a 4+ save. But this is because it scores D6 hits per hit, resulting in a massive variance (It's max damage is actually 24 damage). Nurgle suffers from this as well with a lot of their damage coming from D6 rolls.

Doing variance with all those modifiers and steps everywhere, looks horrible. I was thinking of running simulation like 100-1000 times, always something because average is well, only average and example with ballista shows it perfectly

Certainly GW is not making things easy (either to analyse or actually play)  the way they design certain rules (like additional rend on 6 to hit - just horrible)

 

[NoMaDhOoK]

2 hours ago, Boar said:

Nice. I wanted to do something similar, kinda got sad for a moment that someone done this already. Still not going to surrender, best way to learn something new is doing something tied to your interests.

The repo is also public so you are welcome to contribute if you want. Either way, I wish you luck

2 hours ago, Boar said:

Doing variance with all those modifiers and steps everywhere, looks horrible. I was thinking of running simulation like 100-1000 times, always something because average is well, only average and example with ballista shows it perfectly

Yeah I think I have settled on the simulation route (100 is too small a sample size, so probably 1000). I am then going to do a full probability curve chart, as even variance / standard deviation metrics can be skewed heavily by outliers.

[NoMaDhOoK]

I have released the 0.5.0 version now. As always, please do let me know if there are any issues.

Changes

• Changed order of graphs from [Line, Bar, Radar] -> [Bar, Line, Radar]
• Added Beta version of PDF export (cc: @Planar)
• Allow you to add an optional weapon profile name (cc. @Inquisitorsz)

UI

• Added chart titles
• Added Reddit button to footer

Fixes

• Fix scroll "jumping" on reload of page

Notes

• The PDF is in Beta as I want to get feedback before finalising the design
• The PDF results are always on a separate page to ensure that they are grouped together
  • This will make screenshots / single page prints easier
• In the future I will add PDF export settings (e.g: exclude unit data)

[tom_gore]

This looks really cool. Do you think it would be possible in the future to use this as an actual dice roller, too? In short, it would calculate the amount of hits and wounds from an actual randomized roll according to the "specs" inserted? After that the ability to save your "favourite" profiles and you'd save a lot of players a lot of time at the gaming table

 

[NoMaDhOoK]

5 minutes ago, tom_gore said:

Do you think it would be possible in the future to use this as an actual dice roller, too? In short, it would calculate the amount of hits and wounds from an actual randomized roll according to the "specs" inserted?

Though that was not the original intention of the website, I am currently working on the ability to run a number simulations in order to plot probability curves, so I guess adding a little roll button to run 1 simulation would be an easy addition if you want it.

6 minutes ago, tom_gore said:

After that the ability to save your "favourite" profiles

Currently you can do this through the import / export function (saves as a .json file). In the future I may add the ability to register/login and save that way. Though that is a little further off.

[NoMaDhOoK]

@Inquisitorsz, @Kadeton, @Boar I have some good news. Doing simulations was a lot easier than I had anticipated. I have managed to get it working (as far as I have tested so far) and have run the following tests (ensure that the mean of 3000 simulations is within 3% of the calculated "population" mean)

I still have a lot more tests to run and frontend work to do, and I haven't even started looking at performance optimisations, but it is looking very promising.

[Kadeton]

I've also been putting together some tools for efficiently mapping out the complete possibility space for these complex chains of rolls, which similarly turned out to be easier than I expected. It will be significantly faster than a simulation, but I'm nowhere near as far along as you at this stage.

[NoMaDhOoK]

Still got lots more work to, but here is a sneak peak of the probability graphs that I have working in my dev environment

I am hopefully aiming for a release sometime near the end of next week.

[NoMaDhOoK]

[0.6.0] Christmas Update - Advanced Metrics and Probabilities

Changes

• Added a new screen called Advanced Stats which provides additional stats generated through simulations (cc: @Inquisitorsz, @Kadeton, @Boar)
  • Contains tables with mean, median, max, and variance
  • Contains full probability curve graphs
    • With the option to display reference lines for mean, median, or max
  • This feature is in Beta
• Add probability curves to PDF

UI

• Added the ability to hover (or touch on mobile) legend items in charts in order to highlight only that series
  • This will also highlight the relevant reference lines if there are any
• Move from a AppMenu to a Drawer
• Changed App Bar from fixed -> Hide on scroll

Architecture

• Spawn a cluster of workers for the express server (3 for now, may increase)
• Add gzip compression to API endpoints

Fixes

• Fixed Chart titles missing from PDF graphs

Notes

• This was a large release with a lot of new, beta, additions. Please do let me know if you have any problems, and Merry Christmas

[NoMaDhOoK]

FYI: I am releasing a quick fix for `Reroll Failed` and `Reroll Ones` with the new simulations (advanced stats)

Edit: **Fix Released**

Edited December 24, 2019 by NoMaDhOoK

[The_Dudemeister]

Can you make the probabilities in the tooltip cumulative? I'd like to know what the chances are to make at least 4 damage and not what the chances are to make exactly 4 damage 😄

[NoMaDhOoK]

2 minutes ago, The_Dudemeister said:

Can you make the probabilities in the tooltip cumulative? I'd like to know what the chances are to make at least 4 damage and not what the chances are to make exactly 4 damage 😄

Makes sense. That is definitely something I can look into doing 👍

[Inquisitorsz]

Awesome stuff mate. 

Looks fantastic. 

I tried putting in the stats for Durthu and a wounded version. 

The damage curve seems to make sense but the variability is like 4x higher for the unwounded version which doesn't make sense. D6 damage should be much more variable than 6 damage right?

I think it's got something to do with that the unwounded version has a much higher maximum.... Although you can still roll a 6 for D6 damage, so the maximums should be the same right? Even if incredibly unlikely. Maybe it's a rounding thing? 

Also when I put in add a second weapon profile, his impaling talons, things get weird. 

Finally, impaling talons seems to Max out at a bit over 2 damage (graph gets cut off) but the ability is D6 mortal wounds on a 6 to hit, so the Max should be 6 (or actually 7 with the winterleaf exploding 6s).

Also, you should add a donate link somewhere. Keep up the good work! 

[NoMaDhOoK]

10 hours ago, Inquisitorsz said:

Awesome stuff mate. 

Looks fantastic. 

I tried putting in the stats for Durthu and a wounded version. 

The damage curve seems to make sense but the variability is like 4x higher for the unwounded version which doesn't make sense. D6 damage should be much more variable than 6 damage right?

I think it's got something to do with that the unwounded version has a much higher maximum.... Although you can still roll a 6 for D6 damage, so the maximums should be the same right? Even if incredibly unlikely. Maybe it's a rounding thing? 

Also when I put in add a second weapon profile, his impaling talons, things get weird. 

Finally, impaling talons seems to Max out at a bit over 2 damage (graph gets cut off) but the ability is D6 mortal wounds on a 6 to hit, so the Max should be 6 (or actually 7 with the winterleaf exploding 6s).

Also, you should add a donate link somewhere. Keep up the good work! 

Thanks for the write up. I'll take a deep dive into it after Christmas. Simulations are a little harder to test due to the "random" aspect. So there could be something off in the chain

Edit: found the bug for this issue. Details in separate reply below

The wonky graph could be due to a multi wound profile. Many units get this weird "wavy" pattern when they have a multiwound profile (e.g: 3 damage) along with a modifier that can "sometimes" deal aditional damage.

Because it is not cumulative probability, the chance of getting a multiple of the profile (say 3 damage) is higher than a non multiple.

I think an option to switch the graphs to cumulative probability may be helpful so that's on my to do list for before the new year

Edited December 25, 2019 by NoMaDhOoK

[NoMaDhOoK]

10 hours ago, Inquisitorsz said:

I tried putting in the stats for Durthu and a wounded version. 

The damage curve seems to make sense but the variability is like 4x higher for the unwounded version which doesn't make sense. D6 damage should be much more variable than 6 damage right?

@Inquisitorsz Found it, it was a bug with the get damage portion of the simulation. It was still getting the average damage (3.5) instead of rolling. This resulting in the tests that I ran (comparing average of sample against population) still pass, even though it was not rolling.

When probabilities get way too small, it is possible for them to get lost (rounded out).

Looks like I have some more work to do after Christmas.

I am going to get the damage fix out so long

Edited December 25, 2019 by NoMaDhOoK

[NoMaDhOoK]

7 hours ago, NoMaDhOoK said:

I am going to get the damage fix out so long

FYI: The fix was released (I just forgot to put a message here at the time) 

Edited December 25, 2019 by NoMaDhOoK

[NoMaDhOoK]

[0.7.0] Add Cumulative Probability Graphs

Changes

• Add Cumulative Probability Graphs to Advanced Stats screen (cc: @Inquisitorsz)
• Add Cumulative Probability Graphs to PDF

UI

• Allow users to toggle the X Axis Syncing of graphs in the Advanced Stats screen

 

Next up: Target Modifiers