Weighted kernel regression in solving small sample problems for regression framework and its variants

Abstract

In real engineering problems, regression plays an important role as a tool to approximate an unknown target function. One of the primary regression problems is the insufficient number of training samples during regression of a model to find the relationship between input and output of samples. These samples do not provide enough information which leads to difficulty in the regression. Current techniques focusing on the selection of appropriate model parameters use several data-driven techniques and none of these techniques offer generalized and better solutions to the problems. Incorporation of prior knowledge is a plausible technique to improve the quality of the regression for data-driven techniques but this technique is based on preselection of coefficients in solving the problem, thus ignoring the consequence of the selection. To improve regression quality for small training samples problems, this thesis explored a new technique known as Weighted Kernel Regression (WKR) based on the Nadaraya-Watson Kernel Regression (NWKR) when solving small training samples problems. Besides WKR, the study introduced the incorporation of prior knowledge based on Pareto-optimality concept that utilizes the genetic algorithmbased multi-objective optimization technique to compute the Pareto optimal solutions. Instead of relying on pre-selection of coefficients to incorporate the knowledge, the proposed technique uses post-selection of solutions as it is less subjective, especially when used for solving small sample problems. The experimental results of the proposed technique using several benchmark problems were evaluated and compared with existing techniques. The findings showed that the proposed technique not only can offer better regression accuracy but also flexibility in generalizing the solution, especially when the available training samples are insufficient.