Loading...

Evaluating LLMs in persian news summarization

VarastehNezhad, A ; Sharif University of Technology | 2024

433 Viewed
  1. Type of Document: Article
  2. DOI: 10.1109/IKT65497.2024.10892758
  3. Publisher: 2024
  4. Abstract:
  5. This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications. © 2024 IEEE
  6. Keywords:
  7. Large Language Models ; LLM Evaluation ; Text Summarization ; Economic and social effects ; Economics ; Language model ; Language processing ; Large language model ; Large language model evaluation ; Model evaluation ; Natural language processing ; Natural languages ; Persian news ; Persians ; Text Summarisation ; Natural language processing systems
  8. Source: 15th International Conference on Information and Knowledge Technology, IKT 2024 ; 2024 , Pages 195-201 ; 979-833152225-4 (ISBN)
  9. URL: https://ieeexplore.ieee.org/document/10892758