A scalable HW-based inline deduplication for SSD arrays

Abstract

SSD arrays are becoming popular in modern storage servers as a primary storage, and they aim to reduce the high cost of the devices by performing inline deduplications. Unfortunately, existing software-based inline deduplications cannot achieve the devices' maximum throughput due to their high CPU utilization and power overhead. A recently proposed approach to perform device-wide deduplications inside each SSD can distribute the CPU overhead among multiple SSDs, but it also suffers from severely decreasing deduplication opportunities with the increasing number of SSDs deployed per node. Therefore, we propose a node-wide deduplication engine that relies on specialized hardware to perform two key steps of deduplication; data signature generation and table management. Our FPGA-based prototype detects all duplicates, and compared to software-based inline deduplication, it reduces the overall CPU utilization and power consumption by 93.6 and similar to 20 percent respectively for a slow baseline and more for faster baselines.