Automated outlier detection and estimation of missing data

Abstract

The majority of algorithms used for data imputation are based on latent variable methods. The presence of outliers in process data, however, misleads the latent relations among variables, resulting in an inaccurate estimation of missing values. This article proposes an approach for automatically detecting outliers using T2 and Q contributions and estimating missing data using various general-purpose algorithms while reducing the impact of outliers. The software is validated using biomanufacturing data from the production of a monoclonal antibody produced by Chinese hamster ovary cells in a perfusion bioreactor for five missingness cases including missing completely at random, sensor drop-out, multi-rate, patterned, and censoring. Based on the normalized root mean squared error and the three proposed metrics corresponding to feasibility, plausibility, and rapidity, respectively, matrix completion methods are the most effective, except for the censoring case in which probabilistic principal component analysis-based methods are the most effective.