A Scalable and Flexible Repository for Big Sensor Data

Abstract

Data generation rates of sensors are rapidly increasing, reaching a limit such that storage expansion cannot keep up with the data growth. We propose a new big data archiving scheme that handles the huge volume of sensor data with an optimized lossy coding. Our scheme leverages spatial and temporal correlations inherent in typical sensor data. The spatio-temporal correlations, observed in quality adjustable sensor data, enable us to compress a massive amount of sensor data without compromising distinctive attributes in sensor signals. Sensor data fidelity can also be decreased gradually. In order to maximize storage efficiency, we derive an optimal storage configuration for this data aging scenario. Experiments show outstanding compression ratios of our scheme and the optimality of storage configuration that minimizes system-wide distortion of sensor data under a given storage space.