Error magnitude and directional accuracy for time series forecasting evaluation

Abstract

Evaluation of forecast accuracy is very much influenced by the choice of accurate measurement since it can produce different conclusion from the empirical results. Thus, it is important to use appropriate measurement in accordance to the purpose of forecasting. Commonly, accuracy is measured in terms of error magnitude. However, directional accuracy is as important as error magnitude especially in economics since it considers directional movement of the data. This research attempted to combine the two types of measurements by introducing a new element, the slope value. This proposed measure is known as square error modified of directional accuracy (SE-mDA). Before that, the existing directional change error measurement was modified by comparing the direction of two subsequent forecasts data with two subsequent observed data. Empirical application utilizing the monthly data of Malaysia and Bali tourism demand was used to compare the forecast performance between SARIMA, time series regression, Holt-Winter, intervention neural network and fuzzy time series. The root mean square error, mean absolute percentage error, mean absolute deviation, Fisher’s exact test, Chi-square test, directional accuracy, directional value and the modified of directional change error were used in forecast accuracy evaluation. The best forecast model in terms of SE-mDA for the data of Malaysia and Bali are Holt-Winters and neural network, respectively. The main conclusion from this study is that SE-mDA is able to improve the forecasting performance assessment of error magnitude measurement by considering the directional movements. At the same time it also enhances the available directional accuracy measurement by taking into account the difference between slopes of forecast data and observed data. These improvements will help forecaster to choose the best forecasting method or model so as to produce the most accurate forecast.