Study of cost functions in three term backpropagation for classification problems

Abstract

Three Term Backpropagation was proposed in 2003 by Zweiri, and has outperformed standard Two Term Backpropagation. However, further studies on Three Term Backpropagation in 2007 indicated that the network only surpassed standard BP for small scale datasets (below 100 instances) but not for medium and large scale datasets (above 100 instances). It has also been observed that by using Mean Square Error (MSE) as a cost function in Three Term Backpropagation network, has some drawbacks such as incorrect saturation and tend to trap in local minima, resulting in slow convergence and poor performance. In this study, substantial experiments on implementing various cost functions on Three Term BP are executed to probe the effectiveness of this network. The performance is measured in terms of convergence time and accuracy. The costs functions involve in this study include Mean Square Error, Bernoulli function, Modified cost function and Improved cost function. These cost functions were introduced by previous researchers. The outcome indicates that MSE is not an ideal cost function to be used for Three Term BP. Besides that, the results have also illustrated that improve cost function’s converges faster, while modified cost function produces high accuracy in classification