AI EVALUATION – MEASURING SUCCESS TO INFORM DECISION MAKING
Discussion Points ▪ Evaluation frameworks for AI in healthcare need to be tailored for Aotearoa New Zealand context ▪ Current AI implementation timeline (months) differs significantly from traditional health innovations (15-17 years) ▪ Key measurement categories identified: • Clinical measures (productivity, staff satisfaction) • Patient-related measures (access, engagement, satisfaction) • Administrative measures • System-level measures (integration with workflows) • AI-specific measures (explainability, reliability) ▪ Equity considerations must be incorporated into evaluation frameworks ▪ Explainability of AI remains a contested area with varying perspectives on its importance ▪ Cost vs value calculations need to be distinguished Key Actions ▪ Develop a pragmatic evaluation approach suitable for the rapid implementation timeline of AI ▪ Establish baseline measurements for comparison where possible ▪ Consider adopting elements from international frameworks (e.g., UK's Digital Technology Assessment Criteria) with NZ-specific modifications ▪ Incorporate ongoing monitoring into implementation plans with clear responsibility assignment ▪ Define success measures specific to each AI use case and problem being addressed Additional Notes ▪ Current regulatory framework in NZ lacks specific provisions for AI as medical devices ▪ Risk tolerance may vary between public and private organisations ▪ Benefits realisation should be clearly defined before implementation ▪ Evaluation should consider the entire model of care, not just the AI tool itself ▪ Stakeholder involvement (clinicians, patients, system leaders) is essential in defining success measures
Table sponsored by
- 16 -
Powered by FlippingBook