Browse

Deep Network Regularization with Representation Shaping

Cited 0 time in Web of Science Cited 0 time in Scopus
Authors
최대영
Advisor
Rhee, Wonjong
Major
융합과학기술대학원 융합과학부(디지털정보융합전공)
Issue Date
2019-02
Publisher
서울대학교 대학원
Description
학위논문 (박사)-- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(디지털정보융합전공), 2019. 2. Rhee, Wonjong.
Abstract
The statistical characteristics of learned representations such as correlation and representational sparsity are known to be relevant to the performance of deep learning methods. Also, learning meaningful and useful data representations by using regularization methods has been one of the central concerns in deep learning. In this dissertation, deep network regularization using representation shaping are studied. Roughly, the following questions are answered: what are the common statistical characteristics of representations that high-performing networks share? Do the characteristics have a causal relationship with performance? To answer the questions, five representation regularizers are proposed: class-wise Covariance Regularizer (cw-CR), Variance Regularizer (VR), class-wise Variance Regularizer (cw-VR), Rank Regularizer (RR), and class-wise Rank Regularizer (cw-RR). Significant performance improvements were found for a variety of tasks over popular benchmark datasets with the regularizers. The visualization of learned representations shows that the regularizers used in this work indeed perform distinct representation shaping. Then, with a variety of representation regularizers, a few statistical characteristics of learned representations including covariance, correlation, sparsity, dead unit, and rank are investigated. Our theoretical analysis and experimental results indicate that all the statistical characteristics considered in this work fail to show any general or causal pattern for improving performance. Mutual information I(z
x) and I(z
y) are examined as well, and it is shown that regularizers can affect I(z
x) and thus indirectly influence the performance. Finally, two practical ways of using representation regularizers are presented to address the usefulness of representation regularizers: using a set of representation regularizers as a performance tuning tool and enhancing network compression with representation regularizers.
Language
eng
URI
http://hdl.handle.net/10371/152565
Files in This Item:
Appears in Collections:
Graduate School of Convergence Science and Technology (융합과학기술대학원)Dept. of Transdisciplinary Studies(융합과학부)Theses (Ph.D. / Sc.D._융합과학부)
  • mendeley

Items in S-Space are protected by copyright, with all rights reserved, unless otherwise indicated.

Browse