Anomaly detection for controlling data accruracy in service industry

Abstract

The purpose of this project is to investigate the application of anomaly detection, particularly control charts for individual sample, to control data quality of a risk management system in a financial industry. Four control charts are investigated, namely individual control chart, moving range (MR) control chart, moving average (MA) control chart and exponentially weighted moving average (EWMA) control chart. The quantitative and qualitative detection performance of these control charts is analyzed on two scenarios: live stream and data profiling. Results are compared with expected anomalies determined by system experts. It is discovered that individual control chart performed best for live stream scenario, while MR control chart performed best for data profiling scenario. Qualitatively control charts are simple, user-friendly and easy to fully automate and implement when compared with other detection methods available in literature. In addition, a suitable data quality assurance and control program using the two control charts is suggested