Duplicate news filtering using sentence level heterogeneous graph

Abstract

With the flourishing development of the media of the network, dealing with the abusing news is becoming an essential requirement for portal news websites. However, previous research has only been attempting to improve the detecting efficiency or accuracy during finding near-duplicate news. Most of them rarely think about which news should be deleted or retained. Thus, we propose a heterogeneous graph-based news filtering framework using novel sentence level graph model for a new generation of duplicate news filtering, which is composed of two basic algorithms. First, extract and identify more duplicate news pairs by using sentence-level near-duplicate news detection algorithm

and second, calculate an accurate representative score by using the graph-ranking based on representative news selection algorithm. The proposed framework has been tested using real world dataset and the experimental result show that the proposed algorithms can improve the accuracy of descriptive news selection effectively.