PERFORMANCE COMPARISON OF ZERO-SHOT AND TWO-SHOT PROMPTING IN DETECTING FAKE NEWS USING LARGE LANGUAGE MODELS

Abstract

Fake news detection is a highly crucial challenge in Natural Language Processing (NLP), particularly during significant social events like elections and national crises. This study uses the GPT-3.5-Turbo model to test the effectiveness of zero-shot and two-shot prompting in detecting fake news on the PolitiFact and Liar datasets. Zero-shot prompting consists of task instructions without examples, whereas two-shot prompting contains a few task-related examples. The methodology includes dataset preparation, Large Language Models (LLMs) response collection, encoding, and evaluation using metrics such as accuracy, precision, recall, and F1-score. The results show that two-shot prompting increases performance marginally across all parameters when compared to zero-shot prompting. PolitiFact’s accuracy improved from 0.286 to 0.293, while Liar’s improved from 0.220 to 0.226. Precision, recall, and F1-score also showed minor gains. However, these advances were not statistically significant and highlight the model’s difficulty with handling multi-class classification in the political domain. The GPT-3.5-Turbo model performed better on the PolitiFact dataset, suggesting variability in performance across different datasets. In conclusion, although two-shot prompting provides a slight advantage, the GPT-3.5-Turbo’s overall performance remains limited, indicating the need for more sophisticated techniques (such as advanced prompting methods or more powerful LLMs) to enhance fake news detection.