Integrating memory-mapping and N-dimensional hash function for fast and efficient grid-based climate data query

Figures & data

Figure 1. Chunked storage and unified storage (a) chunked storage with multilayer array indexing, (b) unified storage with no extra indexes

Figure 2. Chunked storage and unified storage indexing, (a) current index, (b) proposed index

Figure 3. Architecture of LotDB

Figure 4. MERRA-2 in Panoply: (a) gridded dataset in plot view, (b) gridded dataset in array view

Table 1. Spatiotemporal queries

Index	Query Content	Number of Grid Points Retrieved
Q1	What’s the precipitation in D.C. at 9:30 a.m. on August 1st, 2017?	1
Q2	What’s the precipitation in D.C. from 9:30 a.m. to 21:30 p.m. for each day in June 2017?	403
Q3	What’s the precipitation in D.C. from 9:30 a.m. to 21:30 p.m. for each day in 2017?	4,745
Q4	What’s the precipitation in the Chesapeake Bay at 9:30 a.m. on August 1st, 2017?	72
Q5	What’s the precipitation in the Chesapeake Bay from 9:30 a.m. to 21:30 p.m. for each day in June?	29,016
Q6	What’s the precipitation in the Chesapeake Bay from 9:30 a.m. to 21:30 p.m. for each day in 2017?	341,640
Q7	What’s the precipitation in the U.S. at 9:30 a.m. on August 1st, 2017?	4,753
Q8	What’s the precipitation in the U.S. from 9:30 a.m. to 21:30 p.m. for each day in June 2017?	1,915,459
Q9	What’s the precipitation in the U.S. from 9:30 a.m. to 21:30 p.m. for each day in 2017?	22,552,985

Table 2. Pre-processing data (from NetCDF to CSV)

Raw Data (NetCDF format)	10 MB	100 MB	1 GB	3.45 GB	10 GB
Intermediate Data (CSV format)	120 MB	1.4 GB	16.3 GB	52 GB	168 GB
Pre-processing Time (hours)	0.01	0.13	1.4	4	13.4

Figure 5. Data uploading time for PostgreSQL, MongoDB, SciDB, and LotDB

Figure 6. Data volume in different containers

Table 3. Data pre-processing & uploading time and data size in different containers

Container	Preprocessing Time (hours)	Uploading Time (hours)	Data Size (GB)
Raw Data (NetCDF format)	N/A	N/A	3.45
PostgreSQL 9.3 (SQL)	4	1.97	89
MongoDB 4.0.4 (NoSQL)	4	7.3	55
SciDB 18.1 (Array Database)	4	3.3	12.06
LotDB	0	0.03	6.78

Figure 7. Spatiotemporal query run-time of PostgreSQL, MongoDB, SciDB and LotDB