A mediation system for continuous spatial queries on a unified schema using Apache Spark

Figures & data

Figure 1. Phases of query planning in SparkSQL (Armbrust et al., 2015).

Table 1. Comparison of architectural characteristics of Spark, Storm and Flink (Chintapalli et al., 2016; Inoubli et al., 2018).

	Apache Spark (Zaharia et al., 2012)	Apache Storm (Storm, Apache, 2014; Toshniwal et al., 2014)	Apache Flink (Carbone et al., 2015)
Processing approach	Real-time, Micro Batch Streaming	Streaming - Trident	Run time streaming
Streaming engine	Spark streaming processes data streams in micro-batches	Designed as DAG with spouts, bolts and streams used to process data	A streaming engine for such workloads; micro-batch, and batch
Data format	Discretized Stream or also called DStream (it is a continuous stream of data), DataFrames	Tuples	DataStream
Programming languages	Python, Java, Scala	Java, Scala	Python, Java
Cluster manager	Hadoop YARN (Vavilapalli et al., 2013), Kubernetes, Standalone, Apache Mesos	Zookeeper	Hadoop YARN, Kubernetes
Streaming query	SparkSQL, Structured Streaming	No	No
Latency	Few seconds	Sub-second	Sub-second
Sliding window	Time based	Time based and count based	Time based
Message	Exactly-once	At-least-once	Exactly-once
CPU consumption	**	*	***
RAM consumption	***	**	***
Processing power	100x faster than Storm	Millions of tuples per second per nodes	1000s per nodes

Table 2. The four popular big spatio-temporal systems (Alam et al., 2021).

	System Type	Underlying System/Architecture	Spatial data types	Partitioning	Indexing	Query Language	Supported Queries
Tornado (Mahmood et al., 2015)	Spatio-textual Stream	Apache Storm	{srcid, oid, (x, y), t, text} A-Grid	A-Grid	Adaptive Indexing Global: Spatial (A-Grid) Local: Spatio-textual (KD-Tree)	Atlas (SQL-Like)	Snapshot, Continuous (Range, kNN, Join)
SSTD (Chen et al., 2020)	Spatio-textual Stream	Apache Storm	Point	QT-tree (Spatial, Textual)	Global: QT-tree Local: Object, Query	N/A	Snapshot, Continuous (Range, kNN, Top-k)
GeoFlink (Shaikh et al., 2020)	Spatial Stream	Apache Flink	Point	Grid	Grid-based	N/A	Continuous (Range, kNN, Join)
GeoSpark (Apache Sedona) (Yu et al., 2015)	Spatial Stream	Apache Spark	Point, LineString, Polygon, Rectangle	Uniform-Grid Voronoi, R-Tree, Quad-Tree, KDB-Tree	R-Tree, Quad-Tree	Extended Spark SQL	Range, kNN, Spatial Join, Distance Join

Figure 2. Local and integrated schema.

Table 3. The relations in the integrated schema.

Global schema		Relations
G.region(regionName, geometry)	⊇	S1.department(departmentName, departmentCode, regionName)
G.region(regionName, geometry)	⊇	S1.commune(communeName, communeCode, communeShape, regionName, regionShape, departmentCode)
G.commune(zipcode, name, geometry)	⊇	S1.commune(communeName, communeCode, communeShape, regionName, regionShape, departmentCode)
G.building(geometry, type)	⊇	S2.building(boundaries, type)
G.sensor(deviceID, sensorName, applicationID, applicationName, data_airHumidity, data_temperature, location, timestamp)	⊇	S3.observation(measureID, measureTime, value, location, deviceID) ∧ S3.device(deviceID, deviceName, applicationID, applicationName) ∧ S3.measure(measureID, measureName), measureName=”temperature”
G.sensor(deviceID, sensorName, applicationID, applicationName, data_airHumidity, data_temperature, location, timestamp)	⊇	S3.observation(measureID, measureTime, value, location, deviceID) ∧ S3.device(deviceID, deviceName, applicationID, applicationName) ∧ S3.measure(measureID, measureName), measureName=”airHumidity”
G.sensor(deviceID, sensorName, applicationID, applicationName, data_airHumidity, data_temperature, location, timestamp)	⊇	S4.observation(measureName, measureTime, value, location, deviceID, deviceName, applicationID, applicationName), measureName=”temperature”
G.sensor(deviceID, sensorName, applicationID, applicationName, data_airHumidity, data_temperature, location, timestamp)	⊇	S4.observation(measureName, measureTime, value, location, deviceID, deviceName, applicationID, applicationName), measureName=”airHumidity”

Figure 3. Running query example Q.

Figure 4. System architecture.

Figure 5. Procedure of rewriting queries.

Figure 6. Running query example Q: query syntax tree.

Table 4. Dataframe transformation description.

Dataframe tranformation	Description
Select	projects a set of columns. Column names are obtained from <AttributeList> and <AggAttributeList>.
filter	filter rows by the given condition. The condition is set using the criteria defined in both <WCondition> and <HavingCondition>.
join	joins two DataFrames those columns are presented in a predicate or a function in <Wcondition>.
groupby	groups the DataFrame using a set of columns, which are defined in <GroupbyExpressionList>. It also handles windowed grouping with elements in <WindowList>.
agg	computes aggregates for columns. The aggregation function and aggregated column are specified in <AggAttributeList>.
sort	sort the DataFrame by the column specified in OrderBy.

Figure 7. Running query example Q: Spark application DAG representation (A) and optimizer Spark application DAG representations (B).

Table 5. Dataset of building and commune.

Dataset name	Geometry	Number of geometries	Size
Building	polygon	423,284	74 MB
Commune	polygon	35,228	40 MB

Figure 8. Snippet of queries.

Figure 9. $\mathit{\lambda }=1\mathit{\hspace{0.17em}}$ with four worker nodes.

Figure 10. $\mathit{\lambda }=1$ with eight worker nodes.

Figure 11. $\mathit{\lambda }=10$ with eight worker nodes.

Figure 12. $\mathit{\lambda }=100\mathit{\hspace{0.17em}}$ with eight worker nodes.

Table 6. Amount of shuffled data for three queries.

Method/Query	C	D
Q1	160 MB	142 MB
Q2	164 MB	148 MB
Q3	5 MB	620 KB

Armbrust, M., Xin, R. S., Lian, C., Huai, Y., Liu, D., Bradley, J. K. & Zaharia, M. (2015, May). Spark sql: Relational data processing in spark. In Proceedings of the 2015 ACM SIGMOD international conference on management of data (pp. 1383–1394).

 Google Scholar

Chintapalli, S., Dagit, D., Evans, B., Farivar, R., Graves, T., Holderbaugh, M. & Poulosky, P. (2016, May). Benchmarking streaming computation engines: Storm, flink and spark streaming. In 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) (pp. 1789–1792). IEEE.

 Google Scholar

Inoubli, W., Aridhi, S., Mezni, H., Maddouri, M., & Nguifo, E. M. (2018, August). A comparative study on streaming frameworks for big data. In VLDB 2018-44th International Conference on Very Large Data Bases: Workshop LADaS-Latin American Data Science (pp. 1–8).

 Google Scholar

Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauly, M. & Stoica, I. (2012). Resilient distributed datasets: A {Fault-Tolerant} abstraction for {In-memory} cluster computing. In 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12) (pp. 15–28).

 Google Scholar

Storm, Apache. 2014. ApacheStorm. Retrieved October , 2022. https://storm.apache.org/.

 Google Scholar

Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S. & Ryaboy, D. (2014, June). Storm@ twitter. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (pp. 147–156).

 Google Scholar

Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2015). Apache flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering, 38(4).

 Google Scholar

Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R. & Baldeschwieler, E. (2013, October). Apache hadoop yarn: Yet another resource negotiator. In Proceedings of the 4th annual Symposium on Cloud Computing (pp. 1–16).

 Google Scholar

Alam, M. M., Torgo, L., & Bifet, A. (2021). A survey on Spatio-temporal data analytics systems. arXiv E-Prints, arXiv–2103.

 Google Scholar

Mahmood, A. R., Aly, A. M., Qadah, T., Rezig, E. K., Daghistani, A., Madkour, A., Abdelhamid, A. S., Hassan, M. S., Aref, W. G., & Basalamah, S. (2015). Tornado: A distributed spatio-textual stream processing system. Proceedings of the VLDB Endowment, 8(12), 2020–2023. https://doi.org/10.14778/2824032.2824126

 Google Scholar

Chen, Y., Lu, Y., Fang, K., Wang, Q., & Shu, J. (2020). uTree: A persistent B±tree with low tail latency. Proceedings of the VLDB Endowment, 13(12), 2634–2648. https://doi.org/10.14778/3407790.3407850

 Google Scholar

Shaikh, S. A., Mariam, K., Kitagawa, H., & Kim, K. S. (2020, October). GeoFlink: A distributed and scalable framework for the real-time processing of spatial streams. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 3149–3156).

 Google Scholar

Yu, J., Wu, J., & Sarwat, M. (2015, November). Geospark: A cluster computing framework for processing large-scale spatial data. In Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems (pp. 1–4).

 Google Scholar

Data availability statement

The data that support the findings of this study are openly available in GitHub at https://github.com/AnnaNgo13/streamgeomed.