Entity-level Factual Consistency of Abstractive Text Summarization (Research Paper Walkthrough)
#textsummarization #nlp #metrics
This research proposes a set of simple metrics to quantify factual consistency at the entity-level. Authors' analyze the factual quality of summaries produced by the state-of-the-art BART model on three news datasets. They also propose several techniques including data filtering, multi-task learning and joint sequence generation to improve performance on these metrics.
⏩ Abstract: A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.
Please feel free to share out the content and subscribe to my channel :)
⏩ Subscribe - / @techvizthedatascienceguy
⏩ OUTLINE:
0:00 - Abstract & Background
02:07 - Entity-level factual consistency metrics
03:24 - Precision-source
04:44 - Entity-based data filtering
06:30 - Precision-target, Recall-target
07:15 - Multi-task learning
11:00 - Joint entity and summary generation
12:08 - Results and Wrap-up
⏩ Paper Title: Entity-level Factual Consistency of Abstractive Text Summarization
⏩ Paper: https://arxiv.org/pdf/2102.09130.pdf
⏩ Author: Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
⏩ Organisation: Amazon Web Services, Columbia University
Text Summarization Papers Playlist: • Text Summarization of COVID-19 Medica...
Evaluating Text Generation Systems Playlist: • Evaluation of Text Generation: A Surv...
Blog Summary: / bsoxrcuykgb
Enjoy reading articles? then consider subscribing to Medium membership, it just 5$ a month for unlimited access to all free/paid content. Subscribe now - / membership
*********************************************
If you want to support me financially which totally optional and voluntary :) ❤️
You can consider buying me chai ( because i don't drink coffee :) ) at https://www.buymeacoffee.com/TechvizC...
*********************************************
⏩ Youtube - / techvizthedatascienceguy
⏩ Blog - https://prakhartechviz.blogspot.com
⏩ LinkedIn - / prakhar21
⏩ Medium - / prakhar.mishra
⏩ GitHub - https://github.com/prakhar21
⏩ Twitter - / rattller
*********************************************
Tools I use for making videos :)
⏩ iPad - https://tinyurl.com/y39p6pwc
⏩ Apple Pencil - https://tinyurl.com/y5rk8txn
⏩ GoodNotes - https://tinyurl.com/y627cfsa
#techviz #datascienceguy #bart #nlg #metrics #researchpaper