Stochastic Optimal Energy Management based on Q-Learning for Hybrid Electric Vehicles

Abstract

Hybrid Electric Vehicles(HEVs) have been widely studied recently with growing concern over sustainability development of the global environment. HEVs use multiple power sources of internal combustion engine and electric battery generally, thus Energy Management Strategy (EMS) determining the split between two power sources needs to be coordinated to maximize the entire vehicle system efficiency. In this study, an EMS based on Stochastic Dynamic Programming(SDP) and reinforcement learning is developed. <br/> SDP is an approaches based on probability, in which optimization problem is defined infinite time horizon, therefore obtained control policy can be used as real-time controller of the vehicle. In order to apply SDP to vehicle control, the characteristics of the driving cycle are expressed as a Transition Probability Matrix (TPM) through the Markov process, in which the vehicle speed and the power demand are discretized. The objective function of the optimization problem in this study is defined as minimizing the expected value of vehicle fuel consumption, deviation of battery state of charge, and frequent engine on / off. As result power split ratio is given as function of vehicle speed, battery SOC, power demand and engine on/off status. <br/> However, SDP has limitation that it is offline policy considering it required TPM, thus in this study, reinforcement learning technique is used to compensate this problem. In the newly proposed control strategy, based on the Q-learning algorithm, the probability information of the driving cycle is updated to the Q value, that is expected cost value of each state variable and control input, and the control rule is updated by calculating the cost function for the all admissible control input at each time step using vehicle model. <br/> To verify control strategy, backward-looking is developed for parallel type HEVs. Simulation results show that fuel economy is improved compared to rule-based strategy and near optimal solution is obtained. <br/>