A Full-Scale Implementation of the NAPP 1880 U.S. Census Data Set Using Dimensional Modeling and Data-Warehousing Technology

Abstract

Data warehousing is a high-performance database technology for the processing of complex data cubes that can be aggregated and crosstabulated in a wide variety of ways. Individual-level U.S. 1880 census records can be analyzed using this technology to extract information not available from printed census volumes, if hierarchically structured code lists for numerical census variables can be converted into analytical dimensions that drive the crossclassification methods. Problems encountered in the construction of a full-scale warehouse, containing millions of records for Northeastern industrial states, are examined and resolved. Also, deficiencies in census-supplied information on occupations are identified, and a new occupational-coding system is implemented to facilitate remedying of these deficiencies in future, as new information from noncensus data sources becomes available.

Keywords:

• 1880 census
• data warehousing
• dimensional modeling
• occupational coding

Notes

The author is grateful for the support of the Economic and Social Research Council from grant RES-000-22-2420 (“Migration, Economic Opportunity and the Railroads: Movement of Heavy Industrial Workers in the North-East USA 1850-1900”). William G. Thomas, Department of History; and the students, staff, faculty, and directors at the Center for Digital Research in the Humanities—all at the University of Nebraska–Lincoln—originally provided the raw digitized data file for the Baltimore and Ohio RR 1857 payroll, from which a large number of railroad job titles were drawn for the new Occupational dimension. Katie Dooley contributed her expertise on nineteenth century European geopolitical boundaries to the correction of the geography dimension. Michael Haines originally suggested the idea of the “Digital 1880 Census” in PDF file format. The paper has benefited generally from the comments of participants in the special NAPP sessions at the SSHA Conference, Long Beach, CA, in November 2009.