Comment on ‘What Protects the Autonomy of the Federal Statistics Agencies? An Assessment of the Procedures in Place That Protect the Independence and Objectivity of Official Statistics” by Pierson et al.

This article refers to:

What Protects the Autonomy of the Federal Statistical Agencies? An Assessment of the Procedures in Place to Protect the Independence and Objectivity of Official U.S. Statistics

I very much welcome this important paper, which is timely and has impressive authorship. Their experience adds weight and authority to its analysis and conclusions.

The authors conclude that a lack of professional independence exposes the U.S. statistical system to efforts to undermine the objectivity of official statistics that the statistical agencies cannot completely rebuff. To an outside observer such a conclusion is not surprising, and it has had parallels in the United Kingdom.

In these comments I want to focus on one aspect of professional independence: governance, by which I mean the decision making and accountability framework within which a statistics office operates.

The U.S. statistical system is a noted example of a decentralized system, with very many statisticians located in policy departments. The pros and cons of decentralized and centralized systems have been expounded elsewhere (see the UN Handbook, sec 4.3), (UNSD 2022) and it is noted (see PARIS21 2005) that the value of strong policy relevance of statistics produced by a decentralized system can be outweighed by a lack of professional independence, undermining the trust in those statistics.

The authors define professional independence as an agency’s ability to act independently from political or other undue external influence with regard to its operations, such as data collection and analysis, staffing and publication. This is in accordance with the UN Fundamental Principles of Official Statistics (UN 2014) and the UN Handbook. I interpret staffing as decisions around individual staff appointments, or headcounts. The overall level of staffing depends on budget decisions, which must remain a question for the administration, as is acknowledged by the paper.

The paper raises the important question of who decides which statistics government should produce. Clearly governments are a major user of official statistics, but not the sole user. Official statistics are needed to hold governments to account, and they are also needed by businesses and citizens to inform their own decision making, and the nature of these statistics is such that only government can produce them. This is often described (as in the paper) as statistics for the public good. In the UK, the Statistics and Registration Services Act 2007 (HMSO 2007) requires the UK Statistics Authority (UKSA) “to have the objective of promoting and safeguarding the production and publication of official statistics that serve the public good”. Indeed, the current strategy for the UKSA (UKSA 2020) is headed Statistics for the public good.

It was not always this way. Römer (2022) detailed, using recently available official sources, how the UK government in the 1980s consistently reduced the publication of statistics on income distribution and introduced new measures to steer public debate away from the increasing poverty levels. At the same time there had been many changes to the unemployment series, which was derived from administrative systems.

The UK statistical system had been developed after the 1940s as a decentralized system co-ordinated by the relatively small Central Statistical Office. Sir Claus Moser had set up the Government Statistical Service in 1968, including all professional statisticians across government, with common recruitment, training and career development, and a common statistical policy framework managed by a set of interdepartmental committees. This was accompanied by an expansion of official statistics in the 1970s.

The resource landscape for the public sector changed with a new administration in 1979. The Rayner review (HMSO 1980) led to cuts in funding for official statistics and propounded the doctrine that official statistics should be collected to meet primarily the needs of government.

Widespread criticism of U.K. official statistics followed throughout the 1980s. In 1990 the Royal Statistical Society published a report Official Statistics: Counting with Confidence (RSS 1990). It criticized the impact of the Rayner cuts on statistical quality, and the poor protection in statute of the autonomy of the statistical system. The report rejected the Rayner doctrine and was critical of the arrangements for government access ahead of publication. It argued that the decentralized U.K. system needed strong safeguards.

The RSS report also argued that a lack of methodological resource had led to a slowness in taking up new statistical developments, and that the GSS should be free to comment on misuse of their statistics. The report concluded that a decentralized service was incompatible with public confidence unless control of all operational issues, including methodology, definitions, publication and staffing, were under the Chief Statistician.

Their four major recommendations included greater centralization and control of statistical activities, strengthened methodological research with a programme of evaluation of important statistics, an advisory commission reporting to Parliament on objectivity, integrity, timeliness and scope, and an Official Statistics Act to safeguard the autonomy and constitutional position of official statistics.

John Pullinger (1997) set out the steps leading up to the creation of the Office for National Statistics in 1996. His article covers the concerns noted by the RSS, and chronicles the steps taken in 1989, following the Pickford Review (HMSO 1989), to centralize the collection and publication of economic statistics in an expanded Central Statistical Office. This was followed by two packages of measures to improve the scope and quality of economic statistics. In 1991, the CSO was given Executive Agency status, distancing the office from government departments and setting out the roles of the head of the office and of ministers. The new arrangements also formally revoked the Rayner doctrine that official statistics should be collected primarily for the needs of government.

These reforms were insufficient to restore public trust and by the mid-1990s it was clear that further centralization of the statistical system was needed: in particular to bring together the statistical functions of the CSO with those of the separate Office for Population Censuses and Surveys, which had responsibility for the population census and social surveys, reporting to the Department of Health. The new merged Office for National Statistics came into being in 1996 bringing together practically all data collection and survey publication work into one office and creating a strong center for statistical methodology.

However, concern remained about public trust. The Government published two policy papers (HMSO 1998, 1999), which defined the concept of National Statistics, that is the work supporting the production of statistics intended for public use, no matter where in government this took place. A National Statistician was appointed with overall professional responsibility for National Statistics outputs. He was supported by new governance arrangements including a strengthened Code of Practice and an independent Statistics Commission. These new arrangements were set up from June 2000, without any changes to legislation.

The lack of a statutory basis for these reforms continued to cause concern—they could be changed by ministers without recourse to parliament. The statutory basis of statistics in the United Kingdom had been built up over many decades and was fragmentary. There was no comprehensive legislation covering the role of the National Statistician, the Statistics Commission, the Code of Practice etc.

The Treasury Committee (a Select Committee of the House of Commons) undertook an inquiry into the new arrangements and reported in January 2001 (House of Commons 2001). In that report, the Committee called for legislation on statistics to “establish more clearly the specific responsibilities of ministers, the National Statistician, the Statistics Commission and others in relation to National Statistics” and to “guard against political interference and backsliding in the future”. The Committee concluded that it was “absolutely essential that the new arrangements for National Statistics should be enshrined in a Statistics Act” and said that, if the Statistics Commission concurred with its view in two years’ time when it carried out its review of the Framework for National Statistics introduced in 2000, then it expected the Government to “bring forward legislation as a matter of priority.”

Following continued pressure from the RSS and Parliamentary Committees, the new Statistics and Registration Service Act was adopted in 2007, and is widely recognized as having improved trust in U.K. official statistics: setting up a model with the statutory post of National Statistician, and an independent U.K. Statistics Authority reporting directly to Parliament, which has the Office for National Statistics as its executive arm, and also a separate Office for Statistics Regulation.

It took 17 years, from 1990 to 2007, for the United Kingdom to move from a largely decentralized system with little statutory backing for professional independence, to a more centralized system with roles underpinned by legislation. This change happened in stages, through a combination of pressure applied by the Royal Statistical Society and Parliament. If the United States follows this path, it could be a similarly long journey, but a journey does not start until the first step is taken.