Improvement in deep reinforcement learning controller for buck-boost converter with constant power load

Abstract

DC-DC converter has been used in the commercial and industrial sectors to step up and down the DC voltage. The increasing development of renewable energy technology, battery storage, and DC microgrids has stressed the importance of the usage of the DC-DC converter. However, DC-DC converters with constant power load (CPL) have experience instability issues such as voltage and frequency fluctuation. This negative effect is prominent to such an extent that it will cause negative input impedance characteristics that cause destabilize effects in the DC system and sensitive electronic components to be damaged. Many techniques are proposed to mitigate the issue, such as using passive and active damping, but they are limited to cost and physical constraints. Therefore, using an intelligent controller to manage the output of the DC-DC converter is a more attractive solution to the issues. The methods have been implemented in the controller using proportional integral derivative (PID), model predictive control (MPC), machine learning, and deep learning. As the system becomes more complex, methods that used PID and MPC controller have become infeasible to be implemented. Therefore, using machine learning and deep learning is an attractive alternative to solve the control issue. Reinforcement learning (RL) and deep reinforcement learning (DRL) have been used to solve complex control problems such as Proximal Policy Optimization (PPO). This project's purpose is to improve the DRL performance via improving reward function and compare both PPO and AC DRL controllers to analyse the performance during induced CPL by using MATLAB for the simulation. The project has shown that by improving the PPO DRL based long short-term memory (LSTM) network for actor and critic agents with an improved reward system will provide overall all improved performance when compared to the benchmark PID controller. By setting an environment in which the DRL controller is able to train properly, the performance of the buck-boost converter controller by AC and PPO DRL algorithm is compared. PPO DRL showing greater performance in converging to the reference point and a faster training period. Moreover, PPO DRL can demonstrate more robustness and improved voltage stability and settling time of the buck-boost converter without the need to further tune its training parameters.