Developing and Validating the Psychometric Properties of the GenAI-Assisted EFL Learning Inventory in the Context of Chinese Universities

doi:10.12002/j.bisu.551

Abstract

Abstract:

The facilitative role of generative artificial intelligence (GenAI) in learning English as a foreign language (EFL) has been widely acknowledged, and students’ attitudes toward GenAI have a significant influence on the adoption of GenAI technologies. While current scales can assess students’ attitudes toward artificial intelligence (AI), they are not tailored to GenAI. Therefore, the reliability and validity of the existing scales require reconsideration. To address the research gaps, this study was to develop and validate a scale to assess Chinese university students’ attitudes toward GenAI-assisted EFL learning. Our study comprises two separate samples. As for Sample 1 (n=379), we used an exploratory factor analysis to develop a scale consisting of 15 items that we categorized according to three dimensions (affect, behavior, and cognition), explaining a total variance of 65.719%. As for Sample 2 (n=418), a confirmatory factor analysis confirmed that this three-factor structure was appropriate. In addition, we evaluated the scale’s criterion validity, internal consistency, and measurement invariance across genders, and found a strong performance across these measures. Thus, the scale developed in this study exhibited robust psychometric properties, and can effectively assess Chinese learners’ attitudes toward GenAI-assisted language learning in the context of Chinese universities.

Keywords: GenAI; GenAI-assisted EFL learning attitude scale; Chinese university students; factor analysis; criterion validity; psychometric properties

摘要：

生成式AI对英语学习的促进作用已得到广泛证实，其中学习者个体对生成式AI的态度是影响其使用情况的关键因素。尽管目前已有一些量表被用于评估个体对AI的态度，但并不适用于生成式AI，且可靠性值得商榷。鉴于此，本研究尝试开发生成式AI辅助中国大学英语学习者态度量表，并对其信度和效度进行检验。在样本1中，通过探索性因子分析，本研究构建了一个包含15个题项的量表，分为情感、行为和认知3个维度，累计方差解释率为65.719%。在样本2中，经验证性因子分析显示，三因子结构在数据中拟合性良好。本研究还对量表的效标效度、内部一致性以及跨性别测量一致性进行了评估，验证了其可靠性和有效性。本研究开发的量表具有较好的心理测量学特性，可以有效地评估我国大学生对使用生成式AI辅助英语学习的态度。

关键词: 生成式AI, 生成式AI辅助英语学习态度量表, 中国大学生, 因子分析, 效标效度, 心理测量学特性

CLC Number:

H319

Wang Yongliang, Wu Hanwei, Derakhshan Ali, Wang Yunsong. Developing and Validating the Psychometric Properties of the GenAI-Assisted EFL Learning Inventory in the Context of Chinese Universities[J]. Journal of Beijing International Studies University, 2024, 46(6): 30-45.

王永亮, 吴韩伟, Derakhshan Ali, 王云松. 生成式AI辅助中国大学英语学习者态度量表的开发与检验[J]. 北京第二外国语学院学报, 2024, 46(6): 30-45.

Figures/Tables 7

References 29

[1]	BERGDAHL J, LATIKKA R, CELUCH M, et al. Self-determination and attitudes toward artificial intelligence: Cross-national and longitudinal perspectives[J/OL]. Telematics and Informatics, 2023,82:102013 [2024-03-28]. https://doi.org/10.1016/J.TELE.2023.102013.
[2]	CARPENTER S. Ten steps in scale development and reporting: A guide for researchers[J]. Communication Methods and Measures, 2018(1):25-44.
[3]	CHEUNG G W & RENSVOLD R B. Evaluating goodness-of-fit indexes for testing measurement invariance[J]. Structural Equation Modeling, 2002(2):233-255.
[4]	DAVIS F D. Perceived usefulness, perceived ease of use, and user acceptance of information technology[J]. MIS Quarterly, 1989(3):319-340.
[5]	DE VET H C W, TERWEE C B, MOKKINK L B, et al. Measurement in Medicine: A Practical Guide[M]. Cambridge: Cambridge University Press, 2011.
[6]	DRAGOSTINOV Y, HARÐARDÓTTIR D, MCKENNA P E, et al. Preliminary psychometric scale development using the mixed methods Delphi technique[J/OL]. Methods in Psychology, 2022,7:100103 [2024-03-28]. https://doi.org/10.1016/j.metip.2022.100103.
[7]	FINCH H. Measurement invariance[C]//MICHALOS A C. Encyclopedia of Quality of Life and Well-Being Research. Dordrecht:Springer, 2014:3909-3912.
[8]	HARRINGTON D. Assessing confirmatory factor analysis: Model fit and model revision[M]//HARRINGTON D. Confirmatory Factor Analysis. New York: Oxford University Press, 2008:50-77.
[9]	HORSTMANN K T, KNAUT M & ZIEGLER M. Criterion validity[M]//ZEIGLER-HILL V & SHACKELFORD T K. Encyclopedia of Personality and Individual Differences. Cham:Springer, 2020:933-935.
[10]	HU L T & BENTLER P M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives[J]. Structural Equation Modeling: A Multidisciplinary Journal, 1999(1):1-55.
[11]	KLOTZ A C, SWIDER B W & KWON S H. Back-translation practices in organizational research: Avoiding loss in translation[J]. The Journal of Applied Psychology, 2023(5):699-727.
[12]	LI Chengchen, FENG Enhao, ZHAO Xian, et al. Foreign language learning boredom: Refining its measurement and determining its role in language learning[J]. Studies in Second Language Acquisition, 2024(1):1-28.
[13]	LIU Guangxiang & MA Chaojun. Measuring EFL learners’ use of ChatGPT in informal digital learning of English based on the technology acceptance model[J]. Innovation in Language Learning and Teaching, 2024(2):125-138.
[14]	MORRISON J T. Evaluating factor analysis decisions for scale design in communication research[J]. Communication Methods and Measures, 2009(4):195-215.
[15]	ROSENBERG M J, HOVLAND C I, MCGUIRE W J, et al. Attitude Organization and Change: An Analysis of Consistency Among Attitude Components[M]. Oxford: Yale University Press, 1960.
[16]	SCHEPMAN A & RODWAY P. The General Attitudes towards Artificial Intelligence Scale (GAAIS): Confirmatory validation and associations with personality, corporate distrust, and general trust[J]. International Journal of Human-Computer Interaction, 2023(13):2724-2741.
[17]	SUH W & AHN S. Development and validation of a scale measuring student attitudes toward artificial intelligence[J/OL]. SAGE Open, 2022(2)[2024-03-08]. https://doi.org/10.1177/21582440221100463.
[18]	VACHA-HAASE T & THOMPSON B. Score reliability: A retrospective look back at 12 years of reliability generalization studies[J]. Measurement and Evaluation in Counseling and Development, 2011(3):159-168.
[19]	VARGAS-SÁNCHEZ A, PLAZA-MEJÍA M Á & PORRAS-BUENO N. Attitude[C]//JAFARI J & XIAO Honggen. Encyclopedia of Tourism. Cham:Springer, 2016:58-62.
[20]	VILADRICH C, ANGULO-BRUNET A & DOVAL E. A journey around alpha and omega to estimate internal consistency reliability[J]. Anales de Psicologia, 2017(3):755-782.
[21]	WANG Yongliang & XUE Lina. Using AI-driven chatbots to foster Chinese EFL students’ academic engagement: An intervention study[J/OL]. Computers in Human Behavior, 2024, 159:108353[2024-03-28]. https://doi.org/10.1016/J.CHB.2024.108353.
[22]	WORTHINGTON R L & WHITTAKER T A. Scale development research: A content analysis and recommendations for best practices[J]. The Counseling Psychologist, 2006(6):806-838.
[23]	WU Hanwei, LIU Wentao & ZENG Yonghong. Validating the AI-assisted second language (L2) learning attitude scale for Chinese college students and its correlation with L2 proficiency[J]. Acta Psychologica, 2024,248:104376 [2024-03-28]. https://doi.org/10.1016/j.actpsy.2024.104376.
[24]	WU Hanwei, WANG Yunsong & WANG Yongliang. “To use or not to use?” A mixed-methods study on the determinants of EFL college learners’ behavioral intention to use AI in the distributed learning context[J/OL]. The International Review of Research in Open and Distributed Learning, 2024(3):158-178.
[25]	蔡基刚, 林芸. 学术论文写作的挑战与变革:借助ChatGPT直接生成一篇学位论文的实验[J]. 北京第二外国语学院学报, 2024(4):29-42.
[26]	李佐文. ChatGPT赋能外语教学:场景与策略[J]. 北京第二外国语学院学报, 2024(1):109-118.
[27]	梅冰. 师范生对人工智能赋能外语教学接受度探析[J]. 当代外语研究, 2024(2):89-99.
[28]	宋飞, 郭佳慧, 曲畅. ChatGPT在汉语作为外语教学中的应用体系及实践[J]. 北京第二外国语学院学报, 2023(6):110-128.
[29]	杨连瑞. ChatGPT大语言模型背景下的二语习得[J]. 现代外语, 2024(4):578-584.

题项内容	情感	行为	认知	共同度
1. 我觉得使用生成式AI辅助学习英语的过程非常有趣	0.701			0.674
2. 我喜欢借助生成式AI来解决我在英语学习中遇到的问题	0.797			0.783
3. 生成式AI辅助英语学习让我感到自信，觉得学习英语更容易了	0.621			0.647
4. 我愿意尝试使用生成式AI辅助英语学习		0.711		0.614
5. 我想尝试使用生成式AI进行个性化的英语学习		0.724		0.683
6. 如果学校开设有关生成式AI辅助英语学习的课程，我会参加		0.558		0.472
7. 如果有一个与生成式AI辅助英语学习相关的社团，我想加入		0.749		0.617
8. 我愿意持续学习有关如何使用生成式AI辅助英语学习的知识		0.727		0.672
9. 我想探索一些生成式AI使用方法，让英语学习更加便捷		0.607		0.617
10. 我认为未来的英语学习将与生成式AI密切结合			0.604	0.625
11. 我认为每个学生都应该掌握生成式AI辅助英语学习的方法			0.657	0.717
12. 我认为生成式AI对英语学习者是不可或缺的			0.722	0.699
13. 我认为值得学习使用生成式AI来提高英语水平			0.774	0.714
14. 我认为我可以有效地利用生成式AI来辅助我的英语学习			0.771	0.709
15. 我认为生成式AI在英语学习中带来的益处大于弊端			0.722	0.614

题项内容	情感	行为	认知	共同度
1. 我觉得使用生成式AI辅助学习英语的过程非常有趣	0.701			0.674
2. 我喜欢借助生成式AI来解决我在英语学习中遇到的问题	0.797			0.783
3. 生成式AI辅助英语学习让我感到自信，觉得学习英语更容易了	0.621			0.647
4. 我愿意尝试使用生成式AI辅助英语学习		0.711		0.614
5. 我想尝试使用生成式AI进行个性化的英语学习		0.724		0.683
6. 如果学校开设有关生成式AI辅助英语学习的课程，我会参加		0.558		0.472
7. 如果有一个与生成式AI辅助英语学习相关的社团，我想加入		0.749		0.617
8. 我愿意持续学习有关如何使用生成式AI辅助英语学习的知识		0.727		0.672
9. 我想探索一些生成式AI使用方法，让英语学习更加便捷		0.607		0.617
10. 我认为未来的英语学习将与生成式AI密切结合			0.604	0.625
11. 我认为每个学生都应该掌握生成式AI辅助英语学习的方法			0.657	0.717
12. 我认为生成式AI对英语学习者是不可或缺的			0.722	0.699
13. 我认为值得学习使用生成式AI来提高英语水平			0.774	0.714
14. 我认为我可以有效地利用生成式AI来辅助我的英语学习			0.771	0.709
15. 我认为生成式AI在英语学习中带来的益处大于弊端			0.722	0.614

模型	χ²/df	CFI	TLI	RMSEA	SRMR	修正题项
建议指标	≤ 3	≥ 0.9	≥ 0.9	≤ 0.08	≤ 0.08	—
初始模型	3.907	0.890	0.862	0.083	0.062	—
修正模型	2.757	0.934	0.917	0.065	0.053	BC1、BC2

模型	χ²/df	CFI	TLI	RMSEA	SRMR	修正题项
建议指标	≤ 3	≥ 0.9	≥ 0.9	≤ 0.08	≤ 0.08	—
初始模型	3.907	0.890	0.862	0.083	0.062	—
修正模型	2.757	0.934	0.917	0.065	0.053	BC1、BC2

	1	2	3	4	5	6	7	8
1. 对生成式AI的态度	—
2. 情感	0.871^**	—
3. 行为	0.878^**	0.600^**	—
4. 认知	0.912^**	0.727^**	0.718^**	—
5. 心理需求	0.450^**	0.349^**	0.430^**	0.418^**	—
6. 自主性	0.301^**	0.229^**	0.266^**	0.308^**	0.767^**	—
7. 胜任感	0.336^**	0.269^**	0.328^**	0.293^**	0.818^**	0.500^**	—
8. 联结感	0.427^**	0.326^**	0.422^**	0.385^**	0.804^**	0.350^**	0.510^**	—