Restoration of hydrological data in the presence of missing data via kohonen self organizing maps

Abstract

The Malaysia National Network system utilises three methods of rainfall data collection, namely manual, chart recording and data logger method. These methods are simultaneously used at most rainfall stations. This leads to, where occurrence of missing data exists, the possibilities of missing data taken shape of three predictable patterns. The missing data patterns identified are either in the form of missing data from one recording method or two recording methods or all three recording methods. It is also noted that, where data is available, there are prevalent measurement inconsistencies between the three methods, even though all apparatus are placed at the same rainfall station. Through data exploration exercise, it is found that the discrepancy between one method of measurement and another may range between 0% - 100%, indicating a relatively unstable data to be replied upon. The current practice to resolve the problem of missing data from one recording method is to substitute the missing data with the remaining recorded data. Similarly, if data from any two of the three recording methods are missing then the available data from the third method is used as a reference. In statistical terminology, this method of substitution is referred as "Hot-deck Imputation". While easily applied, the obvious drawbacks of this method, is the fact that it is not supported by any scientific rationale and it cannot be applied when data is not available from all three recording methods.