At A Glance:
- The ABO is taking a new approach to maintenance of certification with an alternative examination program called Quarterly Questions.
- The program takes a longitudinal view of assessment rather than using a 1-day snapshot. It combines microlearning experiences with ongoing evaluation of general ophthalmologic knowledge.
- For participants whose certificates expire between now and 2020, meeting the passing standard on three annual sets of Quarterly Questions will satisfy the MOC Part III knowledge assessment requirement.
The new examination alternative program from the American Board of Ophthalmology (ABO), called Quarterly Questions, is changing the maintenance of certification (MOC) experience for board-certified ophthalmologists. The debut of this online, self-paced program marks the end of the era of high-stakes, once-a-decade testing for ophthalmologists in practice and allows the ABO and its diplomates to come together at the intersection of learning and assessment. Feedback from the program’s first year indicates that the activity is meaningful, valuable, and even enjoyable.
How We Got Here
More than 100 years ago, ophthalmologists took a leap of faith by agreeing to trust one another to set professional standards for improving patient care and protecting the privilege of physician self-regulation. Today, the ABO honors this agreement through its mission to serve the public by certifying ophthalmologists through the verification of competencies. Under the guidance of 18 practicing ophthalmologists from around the country, the ABO approaches certification as a lifelong process that promotes excellence through continuous improvement. Programs such as MOC are designed by and for ophthalmologists and ophthalmic subspecialists to promote and recognize aspirational goals.
Somewhere along the way, however, MOC began to struggle under the weight of its good intentions. Participants found the program burdensome and not as clinically relevant as they expected. Limitations that stemmed from the four-part MOC program framework developed by the American Board of Medical Specialties (the umbrella organization for the nation’s 24 medical specialty boards) didn’t help improve the experience, and the ABO found itself trying to design a modern program while still approaching high-stakes testing the same way it had more than a century ago. It was time to evolve to meet the needs of today’s practicing ophthalmologists.
The Transformation of Testing
For a fresh start, the ABO sought the advice of practicing ophthalmologists in various stages of their careers, working in different practice environments around the country. Through a collaborative codesign process, the ABO explored new ideas and opportunities with ophthalmologists and ophthalmologists-in- training, professional societies and subspecialty organizations, and even patients. A new vision took shape for a collaborative program that championed both learning and assessment at the same time.
Patterned after the American Board of Anesthesiology’s successful Maintenance of Certification in Anesthesiology Minute program, the ABO’s Quarterly Questions knowledge assessment component for MOC satisfies Part III of the American Board of Medical Specialties’ fourpart MOC design. After 2018, the ABO will fully transition to Quarterly Questions and will no longer offer the Demonstration of Cognitive Knowledge Examination for Part III.
The Quarterly Questions program takes a longitudinal view of assessment rather than a 1-day snapshot. It combines microlearning experiences with the ongoing evaluation of everyday “walking-around knowledge” in ophthalmology. Inherent in the design of the Quarterly Questions program is the need to keep the content contemporary and relevant for the practicing ophthalmologist. Because medical knowledge grows quickly and practice patterns change, subject matter experts develop new questions annually to stay in step with the evolution of medical practice.
The Quarterly Questions program is an innovation in assessment, with the goal of helping to make MOC more relevant to individual practice (and, therefore, more meaningful and valuable to diplomates) while continuing to objectively assess physicians’ knowledge, judgment, and skills in ophthalmology (See “Anatomy of a Quarterly Question”).
Answering Quarterly Questions
Diplomates of the ABO can access up to 50 questions per year through Quarterly Questions, although annual participation is not required. The ABO releases new questions in three out of four quarters of every year (the fourth quarter may be used to catch up on previous questions). Two of the quarters are dedicated to knowledge-based questions that cover standard walkingaround knowledge in ophthalmology, and the third quarter challenges diplomates to apply new knowledge gleaned from peer-reviewed literature.
Diplomates can start and stop Quarterly Questions at their convenience, answering questions one at a time, in small batches, or all in one sitting. Questions are accessible from a computer, tablet, or mobile device. Each knowledge question must be answered within 1 minute without the use of outside resources. Article-based questions are timed at 5 minutes.
Retina specialists can choose all retina content for the knowledge section or test their expertise in other subspecialty areas. Among the 40 questions, participants can expect 10 questions that cover topics in core ophthalmology. For the 10 articlebased questions, participants may choose from a list of more than 30 options covering ophthalmic and medical topics, including three retinaspecific offerings in 2018.
Participants receive instant feedback after answering each question. The platform displays a key point (the main takeaway from the item) and a critique that explains the rationale for the correct answer and the reasons why the incorrect answers are not appropriate. References accompany each question as resources for those who wish to delve further into the topic. Along the way, diplomates can track their performance in relation to the aggregate score of the diplomate community. Research in the learning and assessment literature shows that this method of providing constructive, detailed feedback at the time of both correct and incorrect answers is effective for promoting learning and enhancing information retention.1,2,3
Scoring is based on meeting the annual passing standard a set number of times. The number is based on each diplomate’s certification date. For participants whose certificates expire between now and 2020, meeting the passing standard on three annual sets of Quarterly Questions will satisfy the MOC Part III knowledge assessment requirement. Diplomates whose certificates expire in 2021 or later should plan to meet the annual passing standard five times during the remainder of their 10-year cycle.
Promising Results, Positive Feedback
Preliminary analysis of 2017 pilot year performance data confirms that the program is meeting its intended goals. With an average score of 76% on all knowledge-based questions, participants demonstrated that, although they’ve mastered most of the material, they still have the opportunity to learn something new regarding the questions they missed. Of those who answered a question incorrectly and then reviewed the question feedback, 98.2% of survey respondents found the key point to be helpful, 97.3% found the critique to be helpful, and 94.3% found the references to be helpful. Eighty-five percent of overall participants said they learned something useful that would help them provide better care to patients in the future.
Another measure for evaluating the success of Quarterly Questions is the participants’ relevance rating of the questions. In the pilot, all content was based on core knowledge, with no subspecialty options. Perhaps because of this, participants rated the content somewhere between slightly and moderately relevant, a mean 2.7 on a 4-point scale. In 2018, the ABO introduced subspecialty-specific sets of questions to improve relevance.
The program’s article-based questions serve a different purpose from the knowledge-based questions. Primarily formative (educational) in nature rather than summative (assessment), the article-based questions were designed to help participants learn new insights and best practices from reading the articles. In the program survey, 94% of diplomates agreed or strongly agreed that article-based questions were useful in learning new, emerging information.
Once sufficient program and performance data are available, the ABO intends to publish research about how the Quarterly Questions program promotes learning in ophthalmology.
1. Bangert-Drowns RL, Kulik CL, Kulik, JA, Morgan, MT. The instructional effect of feedback in test-like events. Rev Educ Res. 1991;61(2):213-238.
2. Butler AC, Karpicke JD, Roediger HL. Correcting a metacognitive error: feedback increases retention of low-confidence correct responses. J Exp Psychol Learn Mem Cogn. 2008;34(4):918-928.
3. Wojcikowski K, Kirk L. Immediate detailed feedback to test-enhanced learning: An effective online educational tool. Med Teach. 2013;35(11):915-919.