Affinity Prober를 이용한 언어 모델의 문장 수용성 판단 결정의 경계 요인 분석

Abstract

Recently, many studies endeavor to reveal the intrinsic nature of BERT, since transformer-based language models have achieved the state-of-the-art in many natural language understanding tasks. However, it is still hard to probe BERTs universal linguistic properties since different probing methods lead to fluctuating results between tasks and models. Thus, this paper suggests Affinity Prober, which is a flexible task- and model-agnostic probing method to investigate transformer-based language models decision boundaries when processing linguistic phenomena. Affinity Prober is designed to investigate potential linguistic knowledge from the Self Attention Mechanism. Using Affinity Prober, the study examines whether a bert-base-cased model has any explainable decision boundaries on sentence acceptability in terms of a lexical category in English. The results in syntactic phenomena show that the Affinity Relationship between function words is reinforced in the upper layers, while the Affinity Relationship between content words plays a key role in semantic phenomena. The study concludes that Affinity Prober is helpful to analyze the models decision boundary with lexical categories on specific linguistic phenomena.