The accuracy of rumor reporting

I just read an article from Dylan Murphy at looking at the accuracy rate of NBA reporters for reporting rumors at the trade deadline. He looked at the rumors reported by Adrian Wojnarowski, Chris Broussard, Marc Spears, Marc Stein, Sam Amick and Ken Berger, comparing the number of reports to the number of deals that actually came to fruition.

The results, unsurprisingly, were dismal. Spears had the best hit ratio at 40 percent, but he also had the smallest sample size — with two of the five rumors he reported actually going down.

In fact, the hit rate was pretty much a function of the sample size — there’s a statistical function called r-squared that measures the extent to which the y values in a set of x-y data pairs are influenced by the x values. Put the number of predictions on the X axis and the hit rate on the Y axis and calculate r-squared — the result in this case is 0.96596, or close to 97 percent. This is very high indeed — a perfect correlation is 1.0 or 100 percent.

So Murphy’s article doesn’t measure how accurate the reporters are at all — it measures how many rumors they report. Murphy recognized this, writing:

Except there’s something more seismic in there – it’s not that reporters are peddling misinformation; multiple teams were probably courting Josh Smith, at least on some level – but that somewhere along the line, “moderately revelatory potential event” became breaking news, and the reporters are there every step of the way to catalogue any minor shift: new teams in pursuit, new players floated, likelihoods increasing and decreasing.

Murphy correctly surmised that there aren’t enough morsels to feed the hungry beast that is the Internet. Some resort to throwing whatever wood they can find into the fire, while others are more discriminating.

The fundamental problem is that we’re using the wrong measuring stick to judge reporters’ accuracy. It’s almost certainly not wrong to have reported that the Bucks were pursuing Josh Smith — I haven’t personally spoken to John Hammond or Danny Ferry about it to get 100 percent confirmation, but I think we can put the accuracy rate of this one in the high 90 percent range.

So if the two teams try to work out a deal, a scribe reports they are talking, and the deal ultimately falls through, was the reporter right or wrong? He was 100 percent right — they were talking about a deal when he reported it. If the deal doesn’t actually go down it’s not the reporter’s fault.

Another problem is that there are too many variables here. In addition to “did they have discussions?” we also need to assess the “how serious were the discussions?” question. Did Ferry call every GM in the league to gauge Smith’s value, and in the process have a mundane, ten-minute exploratory conversation with Hammond before deciding he wasn’t going to deal his player? Or did they have serious negotiations that went right down to the wire before falling through?

Most of the time the answer to the above question is “somewhere in the middle.” So is a reporter right or wrong if he says the two teams had a discussion? A serious discussion? Are working hard on a trade? That a deal is imminent? The answer (as long as the discussion actually took place) lies in how the rumor is presented, and is not based on whether it actually results in a deal.

As I read these types of articles I keep asking myself “what’s the baseline?” In other words, what’s the true ratio of deals discussed to deals completed? A dozen-or-so deals were completed around the trade deadline, but how many deals were actually discussed? I don’t know the answer to this question, but I know it’s high — certainly in the hundreds, and maybe in the thousands.

Let’s be conservative and say 100 deals were seriously discussed (i.e., beyond simply gauging interest) prior to the deadline. If a dozen of these deals actually went through, that means we’d expect — with perfect reporting — to see the reporters’ accuracy rate hovering around 12 percent — twelve. And that’s using a conservative estimate of the number of deals that were discussed.

The low discussion:completion ratio also means that the GMs themselves don’t know which deals will be completed and which won’t — they have to do their due diligence and sift through dozens of potential trades to find the needle in the haystack — the trade proposal that actually has merit. I talked to one team executive a week-or-so before the deadline who said their basketball operations staff was swamped working the phones — and this was a team that wasn’t strongly pursuing anything.

So while I don’t know the true baseline rate of deals discussed to deals completed, I’m confident it’s pretty low. And since these discussions take place in private it’s usually impossible to know whether a discussion actually took place, and if it did, how serious it was. But one thing for sure — we’re using the wrong yardstick if we judge a report based on whether the deal actually was completed.

Collectively, the reporters named in the Murphy article were 21-for-91, or a little over 23 percent. Compare that to my conservative baseline estimate of 12 percent. While an uninformed fan could see that 23 percent completion rate and conclude that NBA reporters are doing a pretty lousy job, I think it’s pretty impressive.