If You're Going to Mess with Statistics, Do It in Reality

Inside Lacrosse published a piece today on pace statistics in college lacrosse, which is cool. It's exciting that this stuff is gaining a little bit of ground, but there is a bit of a problem with the piece: The results are screwball-y.

I've been working with this stuff for about six years (here's the first piece), and there's one rule of thumb that I've kept in mind the entire time: This nonsense needs to pass the eyeball test or else it's probably ridiculous. When I see the assertion that Denver is scoring on 43.4 percent of its offensive possessions, that fails the eyeball test immediately, causing me to scold my dragon for punching silly numbers into a machine and letting it spit out results.

I really don't know how the author got to the results he came to. I've tried to back into them, but that appears to be an exercise in futility. The problem, I think, is that the author isn't using a sound foundation for his methodology:

In order to ascertain the number of possessions a program plays, several years of research were required using both team statistics and film analysis. What has been determined is that the most effective means of estimating a team’s number of possessions involved a formula which consists of a team’s number of goals along with the number of opponent clears. While the formula does not always produce the exact number of possessions, it has proven to be an effective estimate in the sport of lacrosse. What it helps to do is to compare top offenses and defenses, not by the traditional standards that have been used for years, but in terms of efficiency.

Do you see the potential flaw? No?

• The author is deflating the number of offensive possessions that a team plays. There are three simplistic ways to generate an offensive possession opportunity: Win a face-off; have a clearing attempt; and getting the ball on a failed clear. Now, this drops some offensive possession opportunities -- a team is awarded possession on a face-off infraction; a period starts without a face-off due to an infraction in the preceding quarter; etc. -- but it captures enough that the noise isn't deafening. If you only consider opponent clears -- which is cockamamie in and of itself -- you're really losing a lot of possessions and artificially inflating a team's offensive efficiency. It's just a huge flaw.

Again, I don't care if you build a methodology off of what is pre-existing -- my work was built off of others' work, and I frequently acknowledge that -- but at least do it correctly. It does no benefit to college lacrosse to utilize a methodology with such a potentially fractured foundation. For example:

• Bucknell has scored 105 goals this season. Under the methodology than I and others have developed, the Bison's adjusted offensive efficiency (goals per 100 offensive possessions adjusted for defenses faced) is 38.13. When you factor in that Bucknell has played about 270 offensive possessions this season, the math is strong: 38.13 (goals per 100 possessions) multiplied by 2.7 (the possession rate) gives you approximately 103 goals based on an efficiency look. It's off by two goals, which passes the eyeball test.
• Bucknell has scored 105 goals this season. Under the methodology that the author uses in the linked Inside Lacrosse article, the Bison would be expected to score 117 goals (270 possessions multiplied by the "offensive efficiency" rate of 43.4%). That's not reality. It's not even close and doesn't pass the eyeball test. There's too much artificial inflation.

So, kudos to Inside Lacrosse for pursuing this material for its readership. All I ask, though, is that the material look like reality. Right now, it's pretty poor and until it gets a tune-up, I'm going to pass on reading it.

And, because I'm a super nice guy, here's the updated efficiency workbook that works within the laws of reality. (Updated through Wednesday's games.)