S-Space College of Engineering/Engineering Practice School (공과대학/대학원) Dept. of Computer Science and Engineering (컴퓨터공학부) Theses (Master's Degree_컴퓨터공학부)
Deep Bayesian Neural Networks for Continual Learning
지속학습을 위한 심층 베이지안 신경망
- 공과대학 컴퓨터공학부
- Issue Date
- 서울대학교 대학원
- deep learning; bayesian neural network; continual learning; catastrophic forgetting; uncertainty estimation
- 학위논문 (석사)-- 서울대학교 대학원 : 공과대학 컴퓨터공학부, 2018. 2. 장병탁.
- In the problem of continual learning, tasks are given in sequence and the goal of the machine is to learn new tasks while retaining previously learned tasks' performances. Even though deep neural networks have become prevalent among machine learning techniques recently, they may fail to deal with continual learning environment. As the model learns new tasks, parameters contributing largely to previous tasks' performances may change and result in poor performance. This phenomenon is called catastrophic forgetting, and researchers tackled this problem using regularizers and structure optimization techniques.
We aim to show that applying Bayesian modelling to deep neural network is beneficial for continual learning. Not only does Bayesian framework provide systematic way of performing online learning, it also provides uncertainty estimates for measuring each parameter's contribution to previously learned tasks' performances. By rescaling gradients applied to parameters with large performance contribution, the model can retain performances to previous tasks longer. This thesis shows how deep Bayesian neural networks utilize model uncertainty, which leads to alleviation of catastrophic forgetting in continual learning environment.