Read any IT trade journals or their online kin and you will be inundated with the work of analyst firms like Gartner Research and JupiterResearch. Their contributions typically appear as short, assured statements of relative technology maturity (”virtualization to rule server room by 2010“), often joined with fearless statistical predictions (”worldwide PC shipments to increase 10 percent“) and the ever-useful bar chart or line graph showing projected adoption curves. They serve as the stolid benchmark of IT punditry, like the AP and Reuters of tech opinion.

Their business model for providing these ubiquitous quotations and summary graphs works like supermarket free samples, gaining virtue by association through regular appearances in public media. This creates a virtuous cycle - a Gartner analyst gets quoted in article on trends in virtualization, which establishes their credential as someone reputable on the topic, which leads to more citations - which raises their brand equity, and in turn attributes apparent value and credibility to their for-profit proprietary reports sold to corporate subscribers. But is this reputation duly earned by the data itself? In the wider field of futurism, as in stock picking, the value of your product ought to be tied directly to your record. Yet as the Roman poet Juvenal asked, “Quis custodiet ipsos custodes?” - or in this case, “who analyzes the analysts?”

JupiterResearch bills itself as offering “unbiased research, analysis and advice, backed by proprietary data, to help companies profit from the impact of the Internet and emerging consumer technologies on their business.” Even putting aside for a moment the inherent potential for conflict of interest in data sold for profit contributing to research bias (where the Internet age has a particularly spotty record a la “study commissioned by Microsoft”), is there any body of objective research on the research conducted, particularly the type tied to adoption rates (such as Gartner’s ‘hype cycles,’ itself a provocative phrase of the primacy of hype in research results)? For all of the charts showing stratospheric take-up of 3G mobile phones, average size of Storage Area Networks, or even dead-and-back-again concepts like “push technology” with clean graphs showing extrapolated growth out 3 to 5 years, is there a corresponding set of data showing their hit rate in ‘guessing the market’?

While a common-sense rejoinder is that the market would inevitably correct if they were lousy prognosticators, in a media-saturated environment and consistently shortening time-spans for evaluation, I wonder if the market is so caught up in new predictions and short-term expectations to even pay heed to longer cycles of accuracy. And since prediction is more likely to bear out in the shortest term, you can look right on the 6-month field while failing miserably at the 3-year mark, without possibly being caught out for it. Even in market analysis, which would seem by its very nature to live and die by Darwinian selection of the best guessers, the prevalence of puffed predictions without corresponding punishment raises the same concern. For example, TheStreet’s speculation today that Apple’s iPhone could net the company $216 million come Friday based on simple extrapolation (1962 potential store outlets x 200 phones in inventory x 100% sell-through = 392,000 iPhones x $550 average price = $216 million revenue) hardly seems like deep insight, yet come Saturday morning, what downside is there to guessing wrong? (Other than for Apple’s stock valuation, which has already lost over $4 billion in an afternoon over Engadget’s irresponsible posting of a fraudulent notice that the iPhone would be delayed a few months.) TheStreet got pageviews, and thus ad revenue for making the quotable claim. On Saturday (or whenever Apple releases official first-day sales), they have a built-in story regardless of how the launch goes. This McLuhan-esque play on meaning becoming secondary to the news cycle has ramifications for all news reporting in modern media, with particular effects already seen in law, politics, public policy, and the shape of entertainment.

Meanwhile, if a watchdog on the players in ‘unbiased research for sale’ does not yet exist, it certainly seems a valuable role to fill in media accuracy, just as CharityNavigator has added some rigor to evaluating the efficiency of non-profit organizations.