Research on Validity and Validation of Language Proficiency Scales: A Review

doi:10.12002/j.bisu.299

Abstract

Abstract:

Language proficiency scales have played a significant role in national language strategy and language education. Recently, China’s Standards of English Language Ability (CSE) has drawn much attention from academia and the whole society. This paper reviews the current studies on validity and validation of language proficiency scales. It is revealed that most research has focused on one function of the scales—assessment tools. In terms of validity theory of language scales, two frameworks proposed by Zhu (2016) and Fang & Yang (2017) are discussed. Both frameworks hold that validity can be classified into several kinds, but they understand validity in different ways and apply different concepts to refer to the same validity. Comparison of these frameworks shows that Zhu (2016) and Fang & Yang (2017) each have their advantages and disadvantages. As for the experimental studies, interpretations of developing the scales and aligning them to other scales and tests have attracted the most effort. Discussion on the theoretical and experimental research of language scales leads us to propose potential future directions. On the one hand, it seems necessary to integrate the two current frameworks into a new model of validation of language proficiency scales that considers scale characteristics under the guidance of validity theory acknowledged in the educational and psychological measurements and language testing. On the other hand, it is arguable that descriptors of learners’ real language behaviors should come from collecting and selecting real data from a large group of language users rather than the subjective judgment of experts, teachers, and students, mainly because the subjective evaluation might not be fully reliable. Furthermore, CSE and other well-known language scales all boast that they should be used as standards on which language tests can be developed. Therefore it is an open question to what extent validity theories of language testing can be applied to validating language scales.

Keywords: language proficiency scales; language testing; language education; validation; China’s Standards of English Language Ability (CSE)

摘要：

语言能力量表在国家语言战略和语言教育中发挥着重大作用,2018年发布的《中国英语能力等级量表》（CSE）已经成为社会和学界关注的热点。本文在对量表效度研究进行梳理后发现,现有研究大多仅围绕量表的测评用途展开。在理论探索方面,朱正才（2016）和方绪军、杨惠中（2017）提出了各自的效度验证框架。两个框架均持分类效度观,但所用概念名称不尽相同,种类略有差别,各有优势和不足。在量表效度验证方面,已有研究主要集中在量表的解读和对接方面。在这些研究中,对学习者真实语言行为的描述多来自专家、教师和学生的主观判断,这些数据的可靠性有待验证。因此,笔者认为有必要在教育与心理测量和语言测试领域效度理论指导下,针对语言能力量表的特点,进一步整合量表效度验证框架。对已有量表的效度验证和新量表研发应从对语言使用者的调查转到基于大规模语言使用者语料库的研究上来。必须指出,CSE等国内外著名语言能力量表均定位为“标准”,而“标准”的语言能力量表是语言测试开发的基础,所以语言测试的效度理论在多大程度上适用于能力量表值得进一步讨论。

关键词: 语言能力等级量表, 语言测试, 语言教育, 效度验证, 中国英语能力等级量表

CLC Number:

H319

Li Qinghua, Kong Shuo. Research on Validity and Validation of Language Proficiency Scales: A Review[J]. Journal of Beijing International Studies University, 2020, 42(5): 32-45.

李清华, 孔烁. 语言能力等级量表效度研究评述[J]. 北京第二外国语学院学报, 2020, 42(5): 32-45.

Figures/Tables 1

References 48

[1]	Alderson J C. Common European Framework of Reference for Languages:Learning,Teaching,Assessment:Case Studies[C]. Strasbourg:Council of Europe, 2002.
[2]	Alderson J C. The CEFR and the need for more research[J]. The Modern Language Journal, 2007,91(4):659~ 663.
[3]	American Educational Research Association, American Psychological Association, National Council on Measurement in Education & Joint Committee on Standards for Educational and Psychological Testing. Standards for Educational and Psychological Testing[Z]. Washington, DC:American Educational Research Association, 2014.
[4]	Byrnes H. Perspectives[J]. Modern Language Journal, 2007,91(4):679~ 685.
[5]	Council of Europe. Common European Framework of Reference for Languages:Learning,Teaching,Assessment[Z]. Cambridge:Press Syndicate of the University of Cambridge, 2001.
[6]	Council of Europe. Relating Language Examinations to the Common European Framework of Reference for Languages:Learning,Teaching,Assessment:A Manual[M]. Strasbourg:Language Policy Division,Council of Europe, 2009.
[7]	Davies A. Assessing Academic English:Testing English Proficiency,1950—1989—The IELTS Solution[M]. Cambridge: Cambridge University Press, 2008.
[8]	Deygers B, Carlsen C, Saville N, et al. The use of the CEFR in higher education:A brief introduction to this special issue[J]. Language Assessment Quarterly, 2018,15(1):1~ 2. DOI URL
[9]	Figueras N. The impact of the CEFR[J]. ELT Journal, 2012,66(4):477~ 485. DOI URL
[10]	Fulcher G. Standards and framework[C]// Tsagari D & Banerjee J. Handbook of Second Language Assessment. Boston/Berlin:Walter de Gruyter, 2016: 29~ 44.
[11]	Green A. Linking tests of English for academic purposes to the CEFR:The score user’s perspective[J]. Language Assessment Quarterly, 2018,15(1):59~ 74.
[12]	Harsch C. How suitable is the CEFR for setting university entrance standards?[J]. Language Assessment Quarterly, 2018,15(1):102~ 108.
[13]	Kane M. Validation[C]//Brennan R. Educational Measurement. Westport,CT: Greenwood Publishing, 2006: 17~ 64.
[14]	Martyniuk W. Aligning Tests with the CEFR:Reflections on Using the Council of Europe’s Draft Manual[C]. Cambridge: Cambridge University Press, 2010.
[15]	Masashi N. The development of the CEFR-J:Where we are,where we are going[C]// Tomimori N,Furihata M,Haida K et al. New Perspectives for Foreign Language Teaching in Higher Education:Exploring the Possibilities of Application of CEFR. Tokyo:Tokyo University of Foreign Studies, 2012: 10~ 116.
[16]	Messick S. Validity[C]// Linn R L. Educational Measurement. 3rd ed. New York:American Council on Education and Macmillan, 1989: 13~ 104.
[17]	North B. The Development of a Common Framework Scale of Language Proficiency[M]. New York:Peter Lang, 2000.
[18]	North B. The CEFR in Practice[M]. Cambridge: Cambridge University Press, 2014.
[19]	Papageorgiou S. Linking international examinations to the CEFR:The Trinity College London experience[C]// Martyniuk W. Aligning Tests with the CEFR:Reflections on Using the Council of Europe’s Draft Manual. Cambridge,UK:Cambridge University Press, 2010: 145~ 158.
[20]	Saville N & Hawkey R. The English Profile Programme:The first three years[J]. English Profile Journal, 2010,1(1):1~ 14.
[21]	Tannenbaum R & Baron P. Mapping TOEIC Test Scores to the STANAG 6001 Language Proficiency Levels[R]. Princeton,NJ:Educational Testing Service, 2010.
[22]	Weir C J. Limitations of the Common European Framework for developing comparable examinations and tests[J]. Language Testing, 2005,22(3):281~ 300.
[23]	Wisniewski K. Empirical learner language and the levels of the Common European Framework of Reference[J]. Language Learning, 2017,67(S1):232~ 253.
[24]	Wisniewski K. The empirical validity of the Common European Framework of Reference scales:An exemplary study for the vocabulary and fluency scales in a language testing context[J]. Applied Linguistics, 2018,39(6):933~ 959.
[25]	蔡宏文. 产出型语言考试与语言标准对接的效度问题——概推性与一致性[J]. 现代外语, 2019(5):709~ 721.
[26]	方绪军, 杨惠中. 语言能力等级量表的效度及效度验证[J]. 外国语, 2017(4):2~ 14.
[27]	方绪军, 杨惠中, 朱正才. 制定全国统一的语言能力等级量表的原则与方法[J]. 现代外语, 2008(4):380~ 387.
[28]	国务院. 国务院关于深化考试招生制度改革的实施意见[Z]. 北京, 2014.
[29]	韩宝成. 国外语言能力量表述评[J]. 外语教学与研究, 2006(6):443~ 450.
[30]	韩宝成, 常海潮. 中外外语能力标准对比研究[J]. 中国外语, 2011(4):39~ 46.
[31]	何莲珍. 语言考试与语言标准对接的效度验证框架[J]. 现代外语, 2019(5):660~ 671.
[32]	何莲珍, 陈大建. 中国英语能力等级量表结构探微——听力描述语的横向参数框架与纵向典型特征[J]. 外语界, 2017(4):12~ 19.
[33]	揭薇. 英语口语考试与中国英语能力等级量表对接研究——以CET-SET 4为例[J]. 外语界, 2019(1):71~ 81.
[34]	金艳, 揭薇. 中国英语能力等级量表“口语量表”制定原则和方法[J]. 外语界, 2017(2):10~ 19.
[35]	李清华. 语言测试之效度理论发展五十年[J]. 现代外语, 2006(1):87~ 95.
[36]	刘建达. 我国英语能力等级量表研制的基本思路[J]. 中国考试, 2015(1):7~ 11.
[37]	刘建达. 中国英语能力等级量表[J]. 中国外语, 2019(3): 1, 11~ 12.
[38]	刘建达, 韩宝成. 面向运用的中国英语能力等级量表建设的理论基础[J]. 现代外语, 2018(1):78~ 90.
[39]	闵尚超. 接受型语言考试与语言标准对接的效度问题——一致性[J]. 现代外语, 2019(5):696~ 708.
[40]	潘鸣威. 中国英语写作能力等级量表的典型写作活动构建:系统功能语言学的文本类型视角[J]. 外语界, 2017(2):37~ 43.
[41]	杨惠中, 桂诗春. 制订亚洲统一的英语语言能力等级量表[J]. 中国外语, 2007(2):34~ 37.
[42]	杨惠中, 朱正才, 方绪军. 尽快制定统一的中国语言能力等级量表[C]//杨惠中,桂诗春. 语言测试社会学. 上海: 上海外语教育出版社, 2015: 132~ 142.
[43]	曾用强. 中国英语能力等级量表的“阅读量表”制定原则和方法[J]. 外语界, 2017(5):2~ 11.
[44]	张洁, 王伟强. 接受型语言考试与语言标准对接的效度问题——来自标准设定过程的证据[J]. 现代外语, 2019(5):684~ 695.
[45]	张凯. 语言测试概论:几个问题[M]. 北京: 北京语言大学出版社, 2016.
[46]	赵雯, 金檀, 王勃然. 大学英语语言能力标准的研制——理论、实践及启示[J]. 现代外语, 2015(1):102~ 122.
[47]	中华人民共和国教育部, 国家语言文字工作委员会. 中国英语能力等级量表(国家语言文字规范GF0018-2018)[S]. 北京: 高等教育出版社, 2018.
[48]	朱正才. 中国英语能力等级量表效度研究框架[J]. 中国考试, 2016(8):3~ 13.

效度证据	《标准》	朱框架	方杨框架
内容	测试内容的代表性	公平效度	内容效度
反应过程	测试的构念与测试过程中受试实际心理过程之间的吻合程度	无	无
内部结构	测试特定维度的理据;试卷实际所测的维度与欲测维度的一致性	构念效度; 内容效度	构念效度
与其他变量的关系	收敛性（与相同或相似构念的其他测试的相关）和区别性（与其他不同构念测试的相关）;与标准之间的关系	决策效度	效标关联效度
后效	测试实施后对测试有关方面产生的影响	教学反拨效度;社会影响效度	使用效度

效度证据	《标准》	朱框架	方杨框架
内容	测试内容的代表性	公平效度	内容效度
反应过程	测试的构念与测试过程中受试实际心理过程之间的吻合程度	无	无
内部结构	测试特定维度的理据;试卷实际所测的维度与欲测维度的一致性	构念效度; 内容效度	构念效度
与其他变量的关系	收敛性（与相同或相似构念的其他测试的相关）和区别性（与其他不同构念测试的相关）;与标准之间的关系	决策效度	效标关联效度
后效	测试实施后对测试有关方面产生的影响	教学反拨效度;社会影响效度	使用效度