AI in Health and Care Award: Skin Analytics evaluation

Background

Skin cancer is the largest referring cancer speciality [1]. Since the early 1990s, skin cancer incidence rates have more than doubled in the UK [2], with 16,744 melanoma [3] and 155,985 non-melanoma skin cancers diagnosed on average per year [4]. At the same time, the urgent suspected cancer (USC) referrals for dermatology have increased by almost four times since 2010 [5] to over 680,000 in 2023 [6].

The growing demand for dermatology appointments is placing significant pressure on a workforce already facing a shortfall of 159 full time equivalent Consultant Dermatologists, representing 24% of the expected workforce [7].

The impact of the increase in demand and understaffed workforce is felt most tangibly through performance falling below nationally set targets (with only 80% of patients seen for a consultant appointment within two weeks of a GP referral in 2022/23) [8].

Artificial Intelligence as a Medical Device (AIaMD) can offer a solution to extend dermatology capacity to speed up current diagnostic pathways and improve patient outcomes.

Context

DERM is an AIaMD for detecting skin cancer. Developed by Skin Analytics, the algorithm analyses dermoscopic images of skin lesions, classifying them as benign or cancerous. DERM is already being used by an increasing number of NHS providers, and is integrated with existing skin cancer pathways, triaging patients before being assessed by dermatologists. The Department of Health and Social Care (DHSC) funded deployment and a real-world evaluation of DERM as part of the AI in Health and Care Award. The evaluation aimed to inform future implementation by assessing key domains such as safety, accuracy, effectiveness, value and sustainability. Working with the University of Surrey, Unity Insights conducted an evaluation of DERM in four NHS sites, across 9,649 patients between February 2022 and April 2023.

What we did

Quantitative analysis of DERM focused on performance, comparative effectiveness, and subgroup analysis investigating health inequalities. Qualitative study used a mixed method approach, including survey and interview for patients and staff, seeking to understand the perceived use and acceptability of DERM. DERM was assessed against machine learning best practice principles. Health economic modelling included a cost-utility analysis to understand the potential impact on patients’ quality of life, a cost-benefit analysis to compare changes in resource use against existing models, and a budget impact model to present financial impacts from a commissioning perspective.

Impact

In secondary care, DERM demonstrated high pathway sensitivity for malignant melanoma (97%), achieving its target rates of 95% (set in line with clinical accuracy seen in Cochrane reviews) [9], effectively triaging high risk lesions to the appropriate management outcome. Similar results were seen in the primary care model, but with wider confidence intervals due to the smaller sample size. Subgroup analysis showed that age, skin type, and deprivation did not change the expected distribution of cases assessed by DERM. It is worth noting that the sample size within subgroup analysis is limited and further analysis with larger sample sizes would improve confidence in the conclusions drawn.

Survey results showed patients perceived AI-enabled teledermatology positively, with 85% rating the service as good or very good. Most patients acknowledged the value of AI if it helped them to get an appointment sooner (67%). Staff reported the effect of the AI teledermatology service on capacity as “transformational”. Both staff and patients were reassured by the “second read” (a dermatologist review of DERM decisions) and would not be in favour of removing at this time. It is worth noting that there is no regulatory barrier in using DERM in autonomous pathways, and clinician sentiment may change as adoption increases and with positive support from recognised independent groups such as NICE. The national shortage of dermatologists means that large-scale use of DERM with the second read is not possible.

The machine learning review deemed DERM performed binary classification with strong results, and that algorithm development was encouraging with reusable and reproducible results.

Health economics compared teledermatology both with and without DERM against face-to-face referrals. All instances highlighted savings for the NHS. DERM in secondary care demonstrated a return of £1.7 for every £1 spent, albeit this was less than non-AI-teledermatology (£2.5 returned) due to higher overall costs. However, DERM released the most amount of dermatologist time, which is crucial given the current national workforce shortages. The quality of life effects of DERM on patients were inconclusive.

Findings from the evaluation are being used to inform further rollout of DERM in the NHS, as the technology is now being piloted in nine additional trusts as part of the NHS Cancer Programme [10].

References:

[1] https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/07/Cancer-Waiting-Times-Annual-Report-201920-Final.pdf

[2] https://www.cancerresearchuk.org/health-professional/cancer-statistics/statistics-by-cancer-type/melanoma-skin-cancer#heading-Zero

[3] https://cks.nice.org.uk/topics/melanoma/background-information/prevalence/

[4] https://www.cancerresearchuk.org/health-professional/cancer-statistics/statistics-by-cancer-type/non-melanoma-skin-cancer/incidence

[5] https://crukcancerintelligence.shinyapps.io/EarlyDiagnosis/

[6] Statistics » Cancer Waiting Times (england.nhs.uk)

[7] https://www.gettingitrightfirsttime.co.uk/wp-content/uploads/2022/07/DermatologyReport-Sept21o.pdf

[8] https://www.england.nhs.uk/statistics/statistical-work-areas/cancer-waiting-times/

[9] Visual inspection for diagnosing cutaneous melanoma in adults – PMC (nih.gov)

[10] 9 additional trusts to use AI to reduce skin cancer backlog and increase efficiency – Skin Analytics (skin-analytics.com)