Deep Reinforcement Learning with LSTM-based Exploration Bonus

Abstract

Deep learning is dominant method in recent machine learning community. Deep learning outperforms traditional methods in various tasks such as image classification, object recognition, speech recognition, and natural language processing. However, most deep learning algorithm is focused on supervised learning task. Supervised learning only considers static environment therefore it is not proper to dynamic environment.

To solve this problem, reinforcement learning method combined with deep learning called deep reinforcement learning is proposed. Deep reinforcement learning is composed of two parts. First part is feature extraction using deep learning method. Second part is learning proper action for agent using reinforcement learning through trial and error.

However, reinforcement learning algorithm has exploration and exploration trade-off problem. It is hard to find optimal exploration and exploitation ratio for reinforcement learning. To solve this problem, we propose novel deep reinforcement learning algorithm with LSTM-based exploration bonus. LSTM-based exploration bonus uses Long-Short Term Memory (LSTM) networks as predictor and makes exploration bonus based on prediction error. LSTM-based exploration bonus guides agent play more daring action. As a result, agent could find optimal solution in more short time.

We test our method playing various Atari games. Experimental results shows our method outperforms plain deep reinforcement learning method. This shows the effectiveness of our method.