About the Customer

Qure.ai is the world’s leading developers of Artificial Intelligence solutions for medical imaging and radiology applications. Pioneers in the field, they are amongst the most published AI research groups in the world having more than 30 publications and presentations in leading journals and conferences, including the first paper on AI in the prestigious journal – The Lancet.

The Pain Point

With tens of thousands of chest X-rays passing through the Qure.ai algorithms every day, it becomes critical for the Qure data science leadership to know the real-time performance of their algorithms across their user base. It is a well known fact that the performance of AI can vary dramatically based on patient ethnicity and equipment vendor characteristics – as an AI developer’s user-base scales, the likelihood of an error creeping through the system increases. The challenge becomes the orchestration of a mechanism where randomly picked Chest X-rays are double-read by a team of radiologists, labels established during these reads are compared against the AI outputs, and a dashboard is presented which contains real-time performance metrics (Area Under Curve, Sensitivity, Specificity etc.) with the ability to deep dive into the false positives / negatives.

CARPL Impact

CARPL’s Real-Time AI validation workflow allows AI developers to monitor the performance of their algorithms in real-time. Reshma Suresh, the Chief Operating Officer at Qure.ai, uses CARPL to get real-time ground truth inputs from radiologists and then compares the radiologist reads to the AI outputs, subsequently creating a real-time performance dashboard for QXR – Qure.ai’s Chest X-Ray AI algorithm.

CARPL makes the process very easy:

• Create a Dataset on CARPL
• Create a “Classification” Testing project on CARPL → choose the appropriate algorithm, i.e. QXR, and the Dataset created above
• Create an Annotation Template on CARPL → create whatever fields that need to get annotated
• Create an Annotation Project on CARPL → select the dataset and the annotation project created above
  • Link the annotation project to the Testing project
  • Select to “Auto-Assign” cases to radiologist(s)
• Now as data keeps getting added to the Dataset, either manually or through CARPL’s APIs, the data gets inferred on the AI and gets assigned to the radiologist(s), and as the radiologist(s) keep reading the scans, the real-time dashboard keeps getting populated!

Impact

While much has been spoken about monitoring of AI in clinical practice at the hospital level, it is even more important for AI developers themselves to monitor AI in real-time so that they may detect shifts in model performance, and intermediate as and when needed. This moves the process of AI monitoring and consequent improvement from a retrospective and post-facto process to a proactive approach. As we go and build on our vision to make CARPL the singular platform behind all clinically useful and successful AI, working with AI developers on helping them establish robust and seamless processes for monitoring of AI is key.