Publications

Detailed Information

Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings

DC Field Value Language
dc.contributor.authorYu, Sangwon-
dc.contributor.authorSong, Jongyoon-
dc.contributor.authorKim, Heeseung-
dc.contributor.authorLee, Seong-min-
dc.contributor.authorRyu, Woo-Jong-
dc.contributor.authorYoon, Sungroh-
dc.date.accessioned2022-10-11T02:11:23Z-
dc.date.available2022-10-11T02:11:23Z-
dc.date.created2022-09-30-
dc.date.issued2022-05-
dc.identifier.citationPROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS), pp.29-45-
dc.identifier.urihttps://hdl.handle.net/10371/185785-
dc.description.abstractRecent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. This phenomenon, called the representation degeneration problem, facilitates an increase in the overall similarity between token embeddings that negatively affect the performance of the models. Although the existing methods that address the degeneration problem based on observations of the phenomenon triggered by the problem improves the performance of the text generation, the training dynamics of token embeddings behind the degeneration problem are still not explored. In this study, we analyze the training dynamics of the token embeddings focusing on rare token embedding. We demonstrate that the specific part of the gradient for rare token embeddings is the key cause of the degeneration problem for all tokens during training stage. Based on the analysis, we propose a novel method called, adaptive gradient gating (AGG). AGG addresses the degeneration problem by gating the specific part of the gradient for rare token embeddings. Experimental results from language modeling, word similarity, and machine translation tasks quantitatively and qualitatively verify the effectiveness of AGG.-
dc.language영어-
dc.publisherASSOC COMPUTATIONAL LINGUISTICS-ACL-
dc.titleRare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings-
dc.typeArticle-
dc.citation.journaltitlePROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS)-
dc.identifier.wosid000828702300003-
dc.citation.endpage45-
dc.citation.startpage29-
dc.description.isOpenAccessN-
dc.contributor.affiliatedAuthorYoon, Sungroh-
dc.type.docTypeProceedings Paper-
dc.description.journalClass1-
Appears in Collections:
Files in This Item:
There are no files associated with this item.

Altmetrics

Item View & Download Count

  • mendeley

Items in S-Space are protected by copyright, with all rights reserved, unless otherwise indicated.

Share