Real-Time Data Offloading for Super Resolution in Mobile Devices

Abstract

The rapid enhancement of camera performances in smartphones has allowed users to take high quality pictures without high-end digital cameras. However, there still remains a large gap between smartphone cameras and digital cameras when in comes to zoom-in functionality. Most smartphones provide only digial zoom-in functionality, where image quality degradation is inevitable when the user enlarges the image. Even the high-end smartphones embedded with optical lens provide limited optical zoom-in capabilities, leaving users with great inconvenience. While users can employ an external optical lens to utilize the optical zoom-in functionality, having to carry around an extra hardware incurs great overhead, not to mention its price.

Image Super Resolution (SR) can be a solution to overcome this limitation by recovering the quality degradation caused by digital zoom-in. Image SR, a technique to restore high frequency details from a Low Resolution (LR) image to obtain a High Resolution (HR) image, has been a traditional field of research in computer vision. As deep learning based, especially Convolutional Neural Network (CNN) based, methods have shown to outperform traditional methods, and have been actively researched in recent years.

In this paper, we exploit deep learning based image SR to replace the optical zoom-in functionality in smartphones without embedded optical lenses. As there are several resource constraints in smartphones~(e.g., computing power, energy, memory), challenges occur when aiming to provide a real-time performance relying solely based on local execution. To tackle the challenge, we propose a server offloading based approach to provide higher frame rate. Through a prototype implementation on Android and extensive experiments in real world environments, we show that our proposed system can provide at least 10~fps.