Workshop Summaries
Psychometrics
1. Workshop: Psychometrics in Assessment
Facilitator: Professor Cees van der Vleuten
Psychometrics is the statistical theory supporting educational testing. There are three classes of psychometric theory:
- Classical test theory
- Generalizability theory
- Item-response theory
In this workshop the first two approaches will be discussed. By analyzing a test, statistics parameters that are used within classical test theory will be explained. After this, the basic notions of generalizability theory will be discussed. We will work through the steps that are required in undertaking a generalizabilty study, focusing on conceptual understanding rather than statistical understanding.
2. Workshop: Use of Generalizability for Design and Analysis of the OSCEs
and Other Performance-based Testing Methods
Facilitator: Dr David Swanson
Objective structured clinical examinations (OSCEs) and other performance-based testing methods (eg, oral exams, ratings of clinical performance on wards) are commonly used for the assessment of clinical skills and clinical decision making. Because these methods involve multiple sources of measurement error (eg, rater stringency, task difficulty, content specificity), classical test (true-score) theory does not provide the tools needed for investigation of their psychometric characteristics. In this workshop, participants will learn to view OSCEs and other performance-based assessment methods from the perspective of generalizability theory, which does supply the conceptual framework and statistical tools needed to estimate the reproducibility (reliability, precision) of scores and evaluate alternate approaches to test design and use of testing resources.
The workshop will provide a non-technical introduction to generalizability theory. It will focus, in particular, on the use of generalizability theory and other statistical methods for the analysis and improvement of multi-site OSCE administrations. To get the most out of the workshop, attendees should have a basic understanding of analysis of variance with repeated measures; familiarity with classical test theory is helpful but not required.
3. Workshop: Item Response Theory
Facilitator: Prof Steven Downing
The purpose of this Item Response Theory (IRT) workshop is to introduce participants to IRT in a practical, non-statistical, and mostly theory-free manner. The workshop will focus on IRT uses in medical education and its particular usefulness for higher-stakes tests in the health professions. No particular background in testing theory or statistics is required as the workshop is intended to be an introduction for generalist users of assessments who are not specialists in educational measurement.
IRT models will be contrasted with the more commonly used (and understood) Classical Measurement Theory (CMT) models. Strengths and limitations of both IRT and CMT models will be discussed, with pragmatic implications noted. The most frequently utilized IRT models (e.g., 1-, 2-, and 3-parameter IRT, Rasch) will be presented, together with the general statistical assumptions underlying these models and the consequences of violations of the assumptions.
Some typical applications of IRT measurement models, such as uses in computer-adaptive testing, will be noted and discussed. Participants will have the opportunity for interactive group work on IRT topics during this workshop and will receive a reading list to further their deeper understanding of Item Response Theory, should they wish more information on the topic.
Standard Setting
4. Workshop: OSCE standard setting - the borderline method
Facilitator: Dr John Patterson
There are numerous standard setting techniques available to determine cut-scores for medical assessments. These are broadly divisible into those which are test-based (typified by Angoff and Ebel methods) and those which are performance-based. Performance-based methods use direct observation and global judgements of candidates’ performances to define the borderline performance and determine the cut score. These methods are the only ones where the standard is set actually during an assessment. In principle these methods are ideal for use in OSCEs and obviate the need for lengthy, test-based standard setting sessions prior to the examination.
This workshop will introduce three performance-based methods applicable to OSCEs - the borderline group method, the borderline regression method and the contrasting groups’ method.
The methods can in principle be applied to any assessment marked by an examiner, but this workshop will consider only OSCE examinations. The key to performance-based standard setting lies in a dual-scoring process. At an OSCE station the examiner observes and scores a candidate’s performance against an objective list of items. This will provide the candidate’s score for the station. For performance-based standard setting the examiner then makes a second and global judgement of the candidate’s overall standard. When the global scores of all the examiners for all the candidates at a station are available, it is possible to determine, using conventional spreadsheet software, the passing standard for each OSCE station and for the examination as a whole.
By using Excel workbooks participants will learn the basic practicalities of OSCE marks' processing and how to generate cut scores using the three methods. Copies of the workbooks and other tools will be supplied for participants' future use. The relative advantages and disadvantages of the three methods will be disucssed. No prior knowledge of standard setting is required. Some knowledge of Excel is desirable, but not essential. Participants may wish to bring laptops to the workshop, but this is not a requirement.
5. Workshop: Angoff and Ebel Methods
Facilitator: Professor Peter McCrorie
Within the current climate of accountability and transparency of standards of competence in medicine and other professional courses, the setting of passing standards in examinations is becoming a crucial issue, both for internal and external quality assurance purposes.
This workshop will provide participants with the practical experience of applying the most widely-used standard setting processes, the Angoff and Ebel methods, involving scrutiny of examination material. The participants will use their professional experience to set pass marks, learn how to handle standard setting data and have the opportunity to discuss their judgements about standards.
Assessment Tools
6. Workshop: OSCE Workshop and Clinic
Facilitator: Dr Kathy Boursicot
30 minutes on the latest developments on OSCEs.
* Increasing authenticity/ validity
* Compensation and standard setting
* Rating scales vs. checklists for scoring
90 minutes: 4 OSCE participant case studies will be discussed.
- Participants are invited to bring their OSCE issues to this workshop.
- Each person / team will have 20 minutes allocated to their case for discussion.
Description of cases to be submitted by 18th October 2012. (see attached form)
Max. 500 words, with specific questions you want addressed
Kathy Boursicot has 18 years' experience of designing and implementing OSCEs at under-graduate and post-graduate levels in medicine, dentistry and veterinary medicine. She has been involved in the introduction and conduct of OSCEs in the UK, Denmark, Jordan, Malaysia, Myanmar, Qatar, Singapore, South Africa, Sudan and Sweden.
She led the Theme Group on Assessment of Clinical Performance at the Ottawa International Conference on Assessment in 2010, which reviewed the literature for best practice in OSCEs and WBAs.
She also has 13 years' experience of standard setting for written tests & OSCEs and has an extensive database of over 4000 references on OSCEs and standard setting, which will be available for consultation
7. Workshop: Workplace-Based Assessment
Facilitator: Dr Julian Archer
The goal of this session is to familiarize the participants with some of the workplace-based assessment methods currently being used. It will highlight the importance of formative assessment in learning, review some of the research on the methods, present a model for faculty development, and describe some of the current research on feedback to trainees. Active participation will be encouraged throughout and small group activities will focus on developing a faculty consensus on assessment standards as well as using the methods to provide effective feedback to trainees.
8. Workshop: OSCE Examiner Training
Facilitator: Angela Hall
The use of OSCEs in both formative and summative assessment at undergraduate and postgraduate level has become widespread, but training for examiners is underdeveloped. Examiner inconsistencies in marking and in the conduct of the exam itself are important contributors to error factors affecting the reliability of OSCEs. These inconsistencies also affect candidates’ perception of the fairness of this form of assessment.
This interactive workshop will cover an introduction to OSCEs for assessors, the common factors in examiner variability, examiner behaviour and some practice at marking videos of OSCE stations. Differences in judgments will be explored with a view to determining how best to improve examiner consistency. Both checklist and domain-based marking will be considered.
9. Workshop: Writing test items on “difficult” topics
Facilitator: Kathy Holtzman
Even the best item writers often find it difficult to construct MCQs focused on professionalism and ethics; systems-based practice and patient safety; and interpretation of the medical literature and evidence based medicine. Questions often turn out to ask for definitions rather than assessing whether examinees can apply their knowledge in these areas to decisions related to patient care. In this workshop, participants will see examples of well- constructed MCQs and learn how to more effectively structure writing scenarios and option lists to measure important competencies.
The workshop will be run in an interactive, seminar-style format that also includes small-group activities with participants self-selecting into one of three groups: professionalism/ethics, patient safety; or interpretation of the literature. As part of the small group work, participants will 1) work together to develop a list of tasks that are essential for examinees to know; 2) select two to three tasks from the list and work through scenarios, generating key features to be assessed; and 3) draft two to three items for presentation to the full group .
To get the most out of the workshop, attendees should be familiar with basic SBA item-writing
10. Workshop: Assessment of Professionalism
Facilitator: Professor Trudie Roberts and Dr Richard Fuller
Currently medical professionalism is very much in the public and medical eye. The GMC, employers and medical schools have to address issues around the teaching and assessment of professionalism at all levels.
The aim of this workshop is to facilitate a better understanding of teaching and assessing professionalism in the undergraduate medical curriculum, although the general principles are applicable to the postgraduate area. As part of the workshop, we will develop an understanding of what is meant by ‘patient centered professionalism’. We will develop a framework for introducing the concept of ‘medical professionalism’ and define desirable attributes and behaviours into the undergraduate medical course. Finally, we will construct a suitable framework for assessing these elements.
11. Workshop: Programmatic Assessment: Theory and Practice
Facilitator: Dr Elaine Dannefer
The purpose of this workshop is to introduce participants to both the theoretical framework and the practice implications of a programmatic approach to assessment. With the move to competency-based assessment, non-standardized methods of assessing performance are being developed as a way of addressing complex, domain-independent skills needed for optimal functioning. At the same, assessment for learning and the importance of formative assessments has joined a discourse that has been primarily focused on assessment of learning and its attendant psychometric concerns.
The programmatic approach offers a new paradigm to guide the deliberate arrangement of multiple assessment methods to accomplish the emerging understanding of the role of assessment in the learning process. In addition to providing an overview of the theoretical framework, the overall design of an assessment program will be considered with attention given to the various components that are needed for learning as well as for decisions about performance. Underlying principles and practice guidelines for designing assessment programs will be provided and participants asked critique a case example. The workshop will be interactive and focused on bridging theory and practice.
Reference Material:
1. Van der Vleuten CPM, Schuwirth LWT, Driessen E, Dijkstra J, Tigelaar D., Baartman LKJ, van Tartwijk J. 2012. A model for programmatic assessment fit for purpose. Med Teach 34:3, in press.
2. Dannefer EF, Henson LC. 2007. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med 82(5):493-502.
3. Dijkstra J, van der Vleuten CPM, Schuwirth LWT. 2010. A new framework for designing programmes of assessment. Adv Hlth Sci Educ 15(3):379-393.
Send to friend
Print this page