RASCH methodology and adaptive AI are transforming healthcare education assessments

Dr Akram Ahmad, Founder and CEO of Academically Global, provides an in-depth explanation of the RASCH methodology and its underlying principles, offering insights into how it works and its applications

As a medical research scientist having worked in India, Malaysia and presently Australia, I’ve seen firsthand how important assessment exams are for healthcare professionals. Medicine, Pharmacy, Dentistry and all others under the healthcare umbrella are very high-stake professions and the integrity of the licensure and equivalence exam directly impacts the quality of healthcare that will be delivered to millions in that country. But do traditional examination methods truly evaluate a candidate’s competence?

Let’s take a brief look at the traditional assessments. Here usually the same weight is assigned to all questions. Some students tend to pick and answer the easier ones to meet the passing percentage. Even if the exam offers opportunities for those who have scored higher marks, two candidates with the same final score could possibly have vastly different competency levels. Another pressing issue is that the questions don’t adapt real time based on the skills of the students and this limits the efficiency of the assessment to gauge true proficiency.

So what’s the alternative:

Advanced psychometric testing, using the RASCH Methodology and Adaptive AI is a great alternative. This methodology is already being used in a basic form for instance in the AMC (Australian Medical Council) exam for IMGs (International Medical Graduates) and also the OPRA (Overseas Pharmacist Readiness Assessment). Both these licensure exams determine the equivalence of internationally trained doctors and pharmacists respectively to ensure that they have the knowledge and skills to practise in Australia’s advanced healthcare ecosystem.

What’s the RASCH methodology:

Developed by Danish mathematician Georg Rasch, the Rasch method is an advanced psychometric model for analysing categorised data.

In the case of the AMC exam, which is in the Computer Adaptive Test (CAT) format, each candidate receives customised questions based on their skill level. Each test is modelled from a calibrated set of questions in a common pool and a cut off score/pass percentage is defined for the examination and this is independent of the number of people attempting and their relative skill levels.

This is how it works

For all candidates the first question is selected from the common pool at random. If the answer is correct, the next question will be a more difficult question and if incorrect, the subsequent question will be a less difficult one. This process is continued up until the last question. As each question is answered, the computer calculates the score or ability estimate till that question. It is cumulative and the precision of the ability estimate increases as more questions are answered. In the traditional method, some questions are too easy while some are too difficult for most candidates but in this method since the questions are from a calibrated pool, they are selected based on individual knowledge and skill.

Interestingly, opposed to traditional assessment methods, the number of correct answers doesn’t determine the score and thus ability. Some might have progressively attempted more difficult questions while some might have attempted the easier ones based on their previous (correct or wrong) answers. The AMC score that would be represented on the AMC scale  then would be higher for a person who has attempted more difficult questions and got fewer correct than someone who has got more of the easy ones correct.

Integration with adaptive AI

These methods of assessments using the RASCH method are increasingly being integrated with Adaptive AI to ensure higher precision in the evaluation of candidate’s abilities and a deeper understanding of their strengths and weaknesses. The final score in such a method would truly reflect a candidate’s knowledge level and problem solving ability.

Challenges and opportunities in Indian healthcare education

There are a vast number of healthcare professionals who graduate from India every year. They are some of the brightest minds but the traditional assessment systems don’t do them enough justice. They also find the overseas licensing exams based on the Rash method challenging because they are not trained in that manner.

As someone who has guided a huge number of Indian healthcare professionals aspiring to establish careers across the world, I’ve come to understand that it’s not the lack of knowledge but it is the methodology that rewards rote learning as opposed to critical thinking that’s causing the hurdle.

If we were to adopt AI Adapted, RASCH based assessments it would lead to more reliable exams that accesses overall competence, a better trained workforce (because of fairer evaluation), improved patient safety, enhanced healthcare standards and ultimately increased global recognition.

The biggest challenge in implementing this RASCH Method however, would be the infrastructure constraints because adaptive testing would require examination bodies to possess robust digital ecosystems, and this is something that’s at a very nascent stage if at all. Another challenge would be that the transitioning from a legacy system to a psychometrically more advanced one requires extensive training to the bodies that conduct the assessments.

The solution to this challenge could be

• Introducing pilot programs in select institutions to understand the real time advantages of this method.
• Partnerships with edtechs to accelerate implementation as they might have the requisite specialisation in adaptive AI testing.

The world is moving towards fairer and smarter systems for examinations especially in healthcare. Having worked in this space long enough, I believe that the shift towards the RASCH method and adaptive AI testing especially in healthcare education in India is not just desirable but a necessity and we must embrace this change to keep up with the world.