On using data and stats to help prevent refereeing problems

Refereeing in football has transformed dramatically in recent decades. Even as far down as the National League, referees will have an assessor at the match who debriefs them afterwards, statistics are collected and compiled, and league tables determine promotions and relegations.

Those league tables are based on easily measured data – yellow and red cards among other things. They’re summed up or averaged over games, and the identities of the teams are discarded. Is this the right detail to omit though?
Every week, referees travel to a stadium and officiate between two teams – teams for which they have no declared connection. Club have a responsibility to provide facilities – showers, food, parking and so on – and there’s almost no difference in between clubs in what they provide.

In each match, players, managers, coaching staff and naturally fans, will interact – often in a hostile manner. All these interactions form associations, implicit or otherwise, positive or negative, which may influence officials, since referees are not machines.

Ideally, then, those supervising officials would keep track of such potential associations – whichever direction they go in. All the data exists almost in real time. The brilliant www.football-data.co.uk isn’t the fastest, but it covers England down to the National League and identifies who the referee was in a given match.
From this we can learn referee statistics: Michael Oliver has awarded, on average, 1.2 home yellow cards and 1.5 away yellow cards in 125 matches; while Elliot Swallow has awarded 2.1 home yellows and 1.9 away yellows in 101 matches. We can also get a sense of how “dirty” a team is by how many cards and fouls are given against them.

Much more detailed information could be used now that event data is commonplace, but arguably cards and fouls covers a huge amount of a referee’s imprint on a match.

We need an index of referee performance that consists of these three metrics. With www.football-data.co.uk we can do this for matches back to 2000-01 in England’s top five divisions – close to 80,000 matches. Fouls average about 11 to 12 per team per match, yellows about 1.5 to 2 per team per match, and reds about 0.09 per team per match, so we scale by the range of these variables. This implicitly upweights red cards, which makes sense given their salience. We subtract the average so that a negative referee index means the referee gave fewer of these key calls than normal, and a positive referee index means the referee gave more.
Our resulting index could be -0.31 if a referee awards zero fouls, yellows or reds against a team; 0.39 if 15 fouls, 2 yellows and 2 reds are awarded against a team and, in the extreme, 1.2 for the infamous match in 2005 when Rotherham had 24 fouls called against them and were given 5 yellows and 3 reds.

Then we use a “fixed effects” approach to evaluate how this index varies across referees and clubs. We could look at the difference in mean between referees, but it would be hard to control for team quality and home advantage that way, and would be more cumbersome. Fixed effects allow each referee to have their own average level.

Fixed effects for teams would tell us if particular teams have more decisions called against them than others. These are important controls to have in the method, but not our central concern: we care about the club-referee interactions.

We thus use club-referee fixed effects, which give a separate number for each club-referee combination. We do that, controlling for quality differences between the teams and home advantage, and we get a rating for each club-referee combination that is positive (negative) if there are more (less) than the average amount of fouls, yellow and red cards.

We have to consider club-referee combinations that have happened more than once – one is an outlier. Two would be too small to be reliable, but since on average there’s about eight months between a referee’s assignment to a team, then arguably waiting for 10 matches would be too long. If we arbitrarily select five matches, that yields a huge 7,797 combinations over our near 80,000 matches. A lot of data and information.

The biggest positive outlier is Stephen Martin refereeing Stoke City – six matches in which Martin has awarded an average of 16 fouls per game, four yellow cards per game and one red card throughout. The second largest outlier is Ben Toner with Salford, followed by David Rock and Oldham Athletic, Bobby Madly and Luton Town, then David Rock and Rochdale. If Pro Ref have a sense of humour, they'll appoint Rock for the next 409 derby.

We need also need a control for referees that in general are more prone to give fouls and cards, and also a control for teams that in general are a bit more likely to commit fouls – dirtier teams. We do that by using the earlier separate team and referee fixed effects. This allows us to look at each club-referee combination as an outlier relative to the baseline card/foul propensity of each referee and club.
We look at standardised residuals, which are a direct measure of the signal relative to the noise. For combinations of fewer (more) matches the noise is larger (smaller).

We find just seven statistically significant combinations, with quite a few just below that threshold. Of the significant ones, Stephen Martin and Stoke is the only one involving a currently officiating referee. David Rock and Oldham has a p-value of about 0.08, just short of the gold standard of 0.05.

The p-value is the probability of a false rejection of the null hypothesis that this particular referee-club combination is no different from either the club or the referee’s general patterns in fouls and cards. It points to an important principle: we can never be certain about a referee-club combination being different, we just keep doubt down to some reasonable or acceptable level. Importantly, that level needs to accord with any public perceptions.

There is huge potential for an index like this. It would need to be tested. Ideally, p-values should not fall below 0.05, that historic barometer of “statistical significance”, and p-values in the range from, say, 0.05 to 0.2 ought to be monitored when referee assignments are determined. Such an approach would ensure that Pro Ref will consider the tempestuous affairs that have been David Rock’s recent appointments to Oldham matches (Harrogate, Newport, Crawley, and Colchester as fourth official). The responsibility lies with each club to keep its employees in check, and bad fouls must be punished, hence some part of a procedure like this must involve the club. Memories can linger, and a longer spell before a follow up appointment of Rock to Oldham may will surely be helpful.

Why does this matter? It matters because football fans are tribal to their clubs, with an ultra-strong focus on their club, and what happens to their club. They aren’t focused on general statistics. They only care about patterns pertaining to their club, and will spot perceived patterns well before they become anything that can be established by any empirical method.

A referee can have a solid set of statistics in general over a season, and can be evaluated by whoever assesses at whichever match, all while patterns can emerge potentially even unbeknown to the officials themselves. Given the scrutiny referees now face, both on social media and through their weekly data-led assessment, it seems implausible that any referee operates differently with respect to any particular team. After all, a football match is thousands of individual decisions over nearly two hours, and a referee may go many years between officiating any particular team. This is unlikely to stop conspiracy theories developing among fans who focus only on their team.

The method outlined here is a sketch. A more sophisticated system could be developed, and one that better allows for variation in how dirty teams are. A more in-depth academic paper would need to be written to properly ground the method in the established literature, and subject to peer review. Even after that, the method here would need to be operated with substantial care and discretion. There is no reason to suppose anything at all is untoward in these patterns. Equally, it would arguably be good practice for Pro Ref to be keeping themselves appraised of potential patterns emerging in this area.

Such an approach seems rather one-sided. Why should referees be the primary focus here, when there are two clubs in each match, and 116 in England’s top five divisions? If an issue is developing between a club and a referee, why should the action only be on the referee side and not the club? Should not the club accept more responsibility for their playing staff behaviour? There is no reason why any low-key investigation into any emerging outlier identified by this approach shouldn’t involve both club and referee but, at the same time, by allowing knowledge of such potential associations to influence future referee assignments to matches may reduce the potential for flashpoints to emerge, as they did at Boundary Park on Halloween.