The development of a new data migration model for NoSQL databases with different schemas in environment management system

Abstract

Data migration transfers data from one database to another database. The motivations of data migration are, for example, transferring data from a legacy database to a modern one and maintaining data up-to-date and consistent in a distributed system. Compared to data migration between traditional databases, data migration between heterogeneous NoSQL databases is more challenging due to the characteristics of NoSQL database such as flexible schema, different supporting features, and different storage paradigms. The differences may cause data quality problem after data migration, especially for environment management system where data are required to predict or to convey accurate information. Therefore, the migration of data between heterogeneous NoSQL databases requires not only to overcome the differences of these databases, but also to ensure the quality of the migrated data. In this paper, we proposed a data migration hub, a model that uses a record to record migration style to transfer data between different NoSQL database schemas. The proposed hub is applicable to the environment management system with data validation and fault tolerance in migration process. As confirmed by the pilot study, our method is able to migrate full set of fields to the destinated database in MongoDB.