Evaluating Hockey Players – Three Primary Factors in Explaining Variance in Performance

Evaluating a hockey player (or team for that matter) seems like a relatively simply task: observe, measure, compare, repeat. There are, however, a litany of competing factors that can influence our observations, perceptions and even measurements of a player at any given position, meaning proper evaluation requires being aware of these factors and the manner in which they interact to influence how we gauge a given guy's performance.

The difference (or "variance) in results across hockey players is primarily dependent on three factors: skill, circumstances and randomness:

1.) Skill

This is the factor that most analysis and punditry is restricted to. Much of it focuses on the perceptually obvious: how fast a guy skates, how big he is, how hard his shot is, how frequently he hits, etc. Often referred to as "tools", a hockey players athleticism and physical prowess obviously go a good ways to determining his performance and output during his career.

Sometimes overlooked, especially by casual fans, is the “toolbox” portion of the skill factor – that is, how well a player thinks the game. Decision making is a major facet an NHL player’s overall ability (particularly skaters) given the speed of the game and the rapidity with which mistakes can turn into goals at the pro level. It’s an aspect that can turn a potential game breaker (Dion Phaneuf, Todd Bertuzzi) who has a lot of raw tools into a potential liability. Most NHL coaches are very good at rating players along these lines, although even they can be blinded by a nice package of abilities and the inevitable seduction of “if only…”. Just ask Mike Keenan.

The best hockey players are often both physically gifted and excellent decision makers. The gradation falls away as guys get less athletic or less mentally adept or both.

2.) Circumstances

Of course, results in hockey aren't just determined by skill alone. Different players play in vastly different environments, resulting in sometimes radically different production and, as such, perception of their abilities. Beyond the most casual fan, a majority of hockey pundits, poolies and such recognize that playing with skilled linemates or spending a lot of time on the power play is conducive to putting up points whereas playing on checking line, killing a lot of penalties or playing 7 minutes a night with goons on the "energy unit" is not. Things like playing through injury and various personal issues (disrupted home life, problems with the coach or in the dressing room) also fall in this category. 

With the advent of advanced statistical collection and analysis, we've also begun to recognize the influence of other, less obvious circumstances on a player's results. For example, it's been demonstrated that starting a lot more in the defensive zone than the offensive zone works to suppress a skaters outshooting (corsi) and scoring results (points, plus/minus). This influence is so subtle as to be close to undetectable to the naked eye in the short term (a single game or series of games), so it's usually ignored or missed by the average fan or talking head. It's also something that hadn't really been collected or measured until recently, meaning it hasn't been in the collective consciousness of general hockey analysis until now.

3.) Randomness

Skill and circumstances go long way to determining a players performance/results. In fact, they are often interrelated as coaches will seek to match players of certain skills to complimentary circumstances – or, at least, the best fit available given the roster he's working with.

The final, least intuitive and therefore most overlooked factor in evaluating player performance is good ol' fashioned randomness. Call it luck, call it the bounces, call it probability, more and more we're learning that the hockey gods can actually have a significant effect on counting stats in the NHL. 

Humans aren't really wired to understand randomness since our brains are apt to assign results to purposeful actions rather than, well…chance. That's where superstitions and rain dances come from, naturally, and it's seemingly embedded in our DNA. 

Randomness is less an influence when the spread between the other two factors (skill and circumstances) is large which is another reason it’s counter-intuitve. For example, a team of NHL players would murder the local rec league club, no matter how many favorable bounces the latter was gifted. Similarly, if one NHL team played an entire game 2 men up, there’s a 99.9% chance they would obliterate the opposition, even if it was an all-star team. However, the closer the other two get, the more chance there is that the difference between a couple of guys (or even teams) results could be the same force that causes a coin to land on heads three or four times in a row. These are the top 0.1% hockey players on the planet, being coached by some of the best hockey minds with the talent being relatively well spread out across the league. The difference between success and failure is therefore rather thin. As such, every year, some players and some clubs end up at either end of the percentages distribution curve even though they don’t really “deserve it”. You can see this in action when, say, Jordan Staal finishes his rookie season with 29 goals on 133 shots (22 SH%), making everyone think he’s the next 40 goal scorer….only to fall to 12 goals the following year (6.6 SH%) even though his skill and circumstances didn’t really appreciably change. Sometimes, the puck goes in more often than it should – sometimes it doesn’t. In the parlance of truck drivers everwhere…shit happens.

Randomness isn’t totally chaotic, however. Usually it can be predicted with a good amount of certainty that, eventually, outliers will regress to the mean. For example, we can say with a lot of confidence that the Flames won’t continue to score at a 37% rate on the PP because it’s well established that – in the long run – even elite power play units only score about 20% of the time with the man advantage. Recognizing the mean as well as standard deviation of a given population of scores is useful when trying to determine the degree to which players or teams results might be “skewed” by the bounces.

There are metrics which aid in the recognition as well as the measurement of each of the above factors. 'Round these parts we tend to discuss and focus on the latter two factors (circumstances, randomness), partially because the first (skill) is well covered in other corners of punditry and partially because some of us are interested in the bleeding edge of analysis. One of my expressed goals going forward is to not only investigate these factors (and, therefore the associated measures) but also to make them more meaningful and accessible to hockey fans and pundits that may be unaware of them or more qualitatively inclined.

by Kent Wilson