The Wall Street Journal reported that our government is using Big Data, a revolutionary methodology to accumulate information about our economy in order to track our results and report their findings.

Big data is a term that describes the large volume of data generated daily. Our government and businesses then analyze voluminous data banks to obtain insights that lead to better decisions and strategic business initiatives.

The industry-standard way to describe big data is with the “three V’s.”

  • Volume: The term big data refers to very large quantities of data.
  • Velocity: New data is being created at a very rapid pace and needs to be processed very quickly.
  • Variety: Big data comes from a wide variety of sources and resides in many different formats such as text files, images, video, audio files, presentations, spreadsheets, email messages and databases.

Government statisticians historically relied largely on surveys of households or businesses and store visits to gather data. From this information repository, they subsequently made estimates for the overall population. These techniques, pioneered in the 1940s, had shortfalls. The process was time consuming. Changing habits also undermined this methodology. That is, response rates are falling, and Americans no longer shop for the same things in the same places as they did 80 years ago.

Fortunately, the employment of big data provides a better mousetrap. Our government now collects massive amounts of information generated by firms on a real time basis and then publishes their findings in official economic releases.

Last month, the Labor Department’s finding from Big Data was extraordinary. That is, government economists uncovered the largest monthly drop in apparel prices on record. Their finding had far reaching results. First of all, the Commerce Department subsequently reported a decline in overall inflation estimates. Secondly, in recognition of these findings, the Federal Reserve indicated that they would keep overnight rates constant rather than lowering or raising them.

Within the next five years, the government will use Big Data gathering to replace much of its existing price-measurement practices. They intend to start publishing an experimental measure of new vehicle prices this year using information from J.D. Power, a private data provider. Statisticians are also working on new price measures for medical services, cellphone plans, cable and Internet services and have considered new measures for groceries and rents.

In February, the Census Bureau announced that the NPD Group, a private data service, would begin providing regular data feeds to the agency to help it put together its retail sales report. Advances in computing technology allow interested parties to track better both the prices and quantities of goods that are sold.

Government data-keepers have found it difficult to keep, up as the economy has changed over the past several decades both in size and scope.

In 1940, almost a quarter of Americans’ spending went toward food, prices of which are relatively easy to measure, and 3.1% toward health care, whose prices are difficult to ascertain. Today, those expenditures have almost reversed: 7.2% of spending is devoted to food, and 15% to health care.

Fewer Americans work in factories, and more work in offices now than in 1940. It is a lot harder to measure the output and productivity of a law firm than of a factory producing widgets.

Welcome to a statistical revolution. The use of big data provides government and business with insights that are unimaginable. This change in perspective prevents any return to status quo ante. Sending hundreds of statisticians to collect information is bush league. We need to capture the relevant data that provides insights into our $20 trillion economy.

Originally published in the Sarasota Herald-Tribune