رسالة جامعية
Towards Interpretable Machine Reading Comprehension with Mixed Effects Regression and Exploratory Prompt Analysis
| العنوان: | Towards Interpretable Machine Reading Comprehension with Mixed Effects Regression and Exploratory Prompt Analysis |
|---|---|
| المؤلفون: | Del Signore, Luca |
| المصدر: | Dissertations, Theses, and Capstone Projects |
| بيانات النشر: | CUNY Academic Works |
| سنة النشر: | 2023 |
| المجموعة: | City University of New York: CUNY Academic Works |
| مصطلحات موضوعية: | language modeling, statistics, AI, Computational Linguistics |
| الوصف: | We investigate the properties of natural language prompts that determine their difficulty in machine reading comprehension tasks. While much work has been done benchmarking language model performance at the task level, there is considerably less literature focused on how individual task items can contribute to interpretable evaluations of natural language understanding. Such work is essential to deepening our understanding of language models and ensuring their responsible use as a key tool in human machine communication. We perform an in depth mixed effects analysis on the behavior of three major generative language models, comparing their performance on a large reading comprehension dataset, and draw some counterintuitive conclusions on the relationship between different prompt features and model accuracy and how that relationship varies between different models. Firstly, we observe a divergence in model accuracy as the prompt’s token count grows with overall stronger models increasing in accuracy and overall weaker models decreasing. Secondly, all models unexpectedly exhibit accuracy gains as they are faced with increasing syntactic complexity – a metric derived from a prompt’s constituency parse tree. Lastly, a post hoc analysis revealed that the overall most difficult prompts had the greatest ability to discriminate between different language models, suggesting their outsized usefulness in MRC evaluation methodologies. These findings raise fascinating questions about the nature of language model understanding and suggest new, more interpretable approaches to their evaluation. |
| نوع الوثيقة: | thesis |
| وصف الملف: | application/pdf |
| اللغة: | English |
| العلاقة: | https://academicworks.cuny.edu/gc_etds/5578Test; https://academicworks.cuny.edu/context/gc_etds/article/6681/viewcontent/Thesis_Final_Draft_Fixed.pdfTest |
| الإتاحة: | https://academicworks.cuny.edu/gc_etds/5578Test https://academicworks.cuny.edu/context/gc_etds/article/6681/viewcontent/Thesis_Final_Draft_Fixed.pdfTest |
| رقم الانضمام: | edsbas.3AC35678 |
| قاعدة البيانات: | BASE |
| الوصف غير متاح. |