基于T5的多项选择题自动生成模型研究

曲靖师范学院学报 ›› 2021, Vol. 40 ›› Issue (6): 36-42.

基于T5的多项选择题自动生成模型研究

徐坚^1,2, 孙瑜³, 张利明⁴

1.云南师范大学民族教育信息化教育部重点实验室,云南昆明 650500;
2.曲靖师范学院信息工程学院,云南曲靖 655011;
3.云南师范大学信息学院,云南昆明 650500;
4.曲靖师范学院文化旅游学院,云南曲靖 655011

收稿日期:2021-09-21 出版日期:2021-11-26 发布日期:2021-12-10
通讯作者: 孙瑜,云南师范大学信息学院教授,主要从事知识工程研究.
作者简介:徐坚,曲靖师范学院信息工程学院副教授,主要从事知识图谱、自然语言处理和智慧教育研究.
基金资助:
国家自然科学基金项目“民族教育信息资源多模态知识融合与服务研究”(62166050).

An Automatic Generation Model of Multiple-Choice Questions Based on T5

XU Jian^1,2, SUN Yu³, ZHANG Liming⁴

1. Key Laboratory of Educational Informatization for Nationalities,Yunnan Normal University, Kunming Yunnan 650500;
2. School of Information Engineering,Qujing Normal University, Qujing Yunnan 655011;
3. School of Information,Yunnan Normal University,Kunming Yunnan 650500;
4. School of Culture and Tourism,Qujing Normal University,Qujing Yunnan 655001,China

Received:2021-09-21 Published:2021-11-26 Online:2021-12-10

摘要/Abstract

摘要： 多选题自动生成属自然语言处理领域难题,其子任务有问题生成、答案生成、干扰项生成等尖端任务,虽前人对三者均有研究,但鲜有文献将其整合.提出多选题自动生成模型,对模型的问题生成、答案生成和干扰项生成等开展理论分析,并基于T5开展自动问题生成和答案生成的实验.实验表明,本模型能根据用户输入文本生成多项选择题,问题质量比纯T5模型的BLEU指标高约1个点,其它指标也略有提升.鉴于干扰项生成属前沿难题,文章仅作理论分析.

关键词: 多项选择题, 问题生成, T5模型

Abstract: The automatic generation of multiple-choice questions, whose subtasks include cutting-edge tasks such as question generation, answer generation, and distractor generation, is a new difficulty in the NLP field. Although previous studies have done on these three subtasks, few studies have tried to unify them. A multiple-choice question automatic generation model was proposed, theoretical analysis was made on the model’s question generation, answer generation, and distractor generation modules, and experiments on automatic question generation and answer generation based on T5 was conducted. The experiments showed that this model can generate multiple-choice questions based on user input text, and the question quality was about 1 point higher than the BLEU against the pure T5 model, and other metrics are also slightly improved. A theoretical analysis was made on the generation of distractor.

Key words: multiple-choice questions, question generation, T5 model

中图分类号:

TP391.1

徐坚, 孙瑜, 张利明. 基于T5的多项选择题自动生成模型研究[J]. 曲靖师范学院学报, 2021, 40(6): 36-42.

XU Jian, SUN Yu, ZHANG Liming. An Automatic Generation Model of Multiple-Choice Questions Based on T5[J]. Journal of Qujing Normal University, 2021, 40(6): 36-42.

参考文献

[1]张海,崔宇路,余露瑶,等.人工智能视角下深度学习的研究热点与教育应用趋势——基于2006～2019年WOS数据库中20708篇文献的知识图谱分析[J].现代教育技术,2020, 30(01):32-38.
[2]王连柱.机器学习应用于语言智能的研究综述[J]. 现代教育技术, 2018, 28(09):66-72.
[3]XIAO D, ZHANG H, LI Y, et al. Ernie-gen: an enhanced multi-flow pre-training and fine-tuning framework for natural language generation[J]. arXiv, 2020 (1):1-8.
[4]GAO Y, BING L, LI P, et al. Generating distractors for reading comprehension questions from real examinations[C]// Proceedings of the AAAI Conference on Artificial Intelligence.Hawaii: AAAI-press, 2019: 6423-6430.
[5]杜鹏东,孙涛,田振清.信息熵在多项选择题质量分析中的应用[J].现代教育技术, 2006 (3):36-39.
[6]KURDI G, LEO J, PARSIA B, et al. A systematic review of automatic question generation for educational purposes[J]. International Journal of Artificial Intelligence in Education,2020, 30(1): 121-204.
[7]RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. arXiv, 2019 (10):1-67.
[8]VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Advances in neural information processing systems. California: NIPS, 2017: 5998-6008.
[9]RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of Machine Learning Research, 2020,21(140): 1-67.
[10]SATRIA A Y, TOKUNAGA T. Automatic generation of english reference question by utilising Nonrestrictive Relative Clause[C]// the 7th International Conference on Computer Supported Education. Changsha: Springer, 2017: 379-386.
[11]SATRIA A Y, TOKUNAGA T. Evaluation of automatically generated pronoun reference questions[C]// Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. Copenhagen: ACL, 2017: 76-85.
[12]JIANG S, LEE J. Distractor generation for chinese fill-in-the-blank items[C]// Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. Copenhagen: ACL, 2017: 143-148.
[13]LIANG C, YANG X, WHAM D, et al. Distractor generation with generative adversarial nets for automatically creating fill-in-the-blank questions[C]// Proceedings of the Knowledge Capture Conference. Texas: Association for Computing Machinery, 2017: 1-4.
[14]LIANG C, YANG X, DAVE N, et al. Distractor generation for multiple choice questions using learning to rank[C]// Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications. New Orleans: ACL, 2018: 284-290.
[15]LIU M, RUS V, LIU L. Automatic chinese factual question generation[J]. IEEE Transactions on Learning Technologies, 2016, 10(2): 194-204.
[16]LIU M, RUS V, LIU L. Automatic chinese multiple choice question generation using mixed similarity strategy[J]. IEEE Transactions on Learning Technologies, 2017,11(2): 193-202.
[17]KUMAR G, BANCHS R E, D'HARO L F. Automatic fill-the-blank question generator for student self-assessment[C]// 2015 IEEE Frontiers in Education Conference. Texas: IEEE, 2015: 1-3.
[18]PATRA R, SAHA S K. Progress in computing, analytics and networking[M]. Singapore: Springer, 2018: 511-518.
[19]PATRA R, SAHA S K. A hybrid approach for automatic generation of named entity distractors for multiple choice questions[J]. Education and Information Technologies, 2019,24(2): 973-993.
[20]YANEVA V. Automatic distractor suggestion for multiple-choice tests using concept embeddings and information retrieval[C]// Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications. New Orleans: ACL, 2018: 389-398.
[21]SHAH R, SHAH D, KURUP L. Automatic question generation for intelligent tutoring systems[C]// 2017 2nd International Conference on Communication Systems, Computing and IT Applications. Mumbai: IEEE, 2017: 127-132.
[22]SUN X, LIU J, LYU Y, et al. Answer-focused and position-aware neural question generation[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: EMNLP, 2018: 3930-3939.
[23]NADEAU D, SEKINE S. A survey of named entity recognition and classification[J]. Lingvisticae Investigationes, 2007,30(1): 3-26.
[24]LIANG C, YANG X, WHAM D, et al. Distractor generation with generative adversarial nets for automatically creating fill-in-the-blank questions[C]// Proceedings of the Knowledge Capture Conference. Texas:K-CAP, 2017: 1-4.
[25]ZHOU X, LUO S, WU Y. Co-attention hierarchical network: Generating coherent long distractors for reading comprehension[C]// Proceedings of the AAAI Conference on Artificial Intelligence. New York:AAAI, 2020: 9725-9732.
[26]ENRICO LOPEZ L, CRUZ D K, BLAISE CRUZ J C, et al. Transformer-based end-to-end question generation[J]. arXiv, 2020(5):1-9.
[27]OQUAB M, BOTTOU L, LAPTEV I, et al. Learning and transferring mid-level image representations using convolutional neural networks[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Ohio:CVPR, 2014: 1717-1724.
[28]JIA Y, SHELHAMER E, DONAHUE J, et al. Caffe: Convolutional architecture for fast feature embedding[C]// Proceedings of the 22nd ACM international conference on Multimedia. Florida: Association for Computing Machinery, 2014: 675-678.
[29]HUH M, AGRAWAL P, EFROS A A. What makes imagenet good for transfer learning[J]. arXiv, 2016(6):1-10.
[30]YOSINSKI J, CLUNE J, BENGIO Y, et al. How transferable are features in deep neural networks[J]. Advances in Neural Information Processing Systems, 2014 (27): 3320-3328.
[31]RUSSAKOVSKY O, DENG J, SU H, et al. Imagenet large scale visual recognition challenge[J]. International journal of computer vision, 2015,115(3): 211-252.
[32]DENG J, DONG W, SOCHER R, et al. Imagenet: A large-scale hierarchical image database[C]// 2009 IEEE conference on computer vision and pattern recognition. Florida: IEEE, 2009: 248-255.
[33]KENTON J D, TOUTANOVA L K. BERT: pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of NAACL-HLT.Minnesota:NAACL-HLT, 2019: 4171-4186.
[34]YANG Z, DAI Z, YANG Y, et al. Xlnet: Generalized autoregressive pretraining for language understanding[J]. Advances in neural information processing systems, 2019 (32):1-11.
[35]DONG L, YANG N, WANG W, et al. Unified language model pre-training for natural language understanding and generation[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems.Vancouver: NeurIPS, 2019: 13063-13075.
[36]LIU Y, OTT M, GOYAL N, et al. Roberta: A robustly optimized bert pretraining approach[J]. arXiv, 2019(7):1-13.
[37]RAJPURKAR P, ZHANG J, LOPYREV K, et al. SQuAD: 100, 000+ questions for machine comprehension of text[C]// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Texas: EMNLP, 2016: 2383-2392.
[38]LAI G, XIE Q, LIU H, et al. RACE: large-scale reading comprehension dataset from examinations[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.Copenhagen: EMNLP, 2017: 785-794.
[39]SHARMA S, ASRI L E, SCHULZ H, et al. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation[J]. arXiv, 2017 (6):1-10.
[40]ENRICO LOPEZ L, CRUZ D K, BLAISE CRUZ J C, et al. Transformer-based end-to-end question generation[J]. arXiv, 2020 (5):1-9.