Mixed-methods evaluation of a digital health information and remote monitoring app for heart failure

How we worked with the client

Unity Insights were commissioned to perform an evaluation of a digital health information and remote monitoring app, developed in-house by an NHS trust in the South of England for patients with heart failure. The app provides various information repositories about heart failure, including general information about heart failure and decision aids for seeking medical attention, as well as a feature allowing for remote blood pressure monitoring, appointment booking and reminders, and a direct patient-clinician messaging service.

The evaluation used a mixed-methods approach, combining qualitative, quantitative, and health economic analysis, to assess the multidimensional impacts of the solution on patient and staff experiences, clinical outcomes, and system value. The evaluation was guided by a theory of change, which depicted how the inputs and activities of the solution were expected to result in high-level impacts for different cohorts (patients, staff, system, and public). The evaluation also addressed nine evaluation questions, which covered the aspects of acceptability, effectiveness, implementation, value, and health inequalities of the app.

The qualitative analysis involved two data sources: a patient survey and a staff survey. The patient survey was distributed by the trust to patients on heart failure pathways who used the app. The survey consisted of 23 questions that asked about patient demographics, The solution usage patterns and access methods, favourability of different features, effectiveness and confidence of condition management, and app improvement suggestions. The staff survey was distributed by Unity Insights to clinical and administrative staff relevant to the heart failure pathways and management of the app. The survey consisted of 8 questions that asked about the acceptability of the solution among staff, as well as their perceptions of its effectiveness in practice. Deductive thematic analysis was used to examine trends in patient and staff feedback regarding their usage and views of the app.

The quantitative analysis involved three data sources: enquiry data, Hospital Episode Statistics (HES) queries, and pseudonymised patient data from the app. The enquiry data consisted of an anonymised log of messages sent to the heart failure team. The HES queries were used to obtain data on inpatient and outpatient activity and costs within the cardiothoracic surgery and cardiology specialties, for the period of April 2020 to June 2023. This data was separated into pre- and post-implementation timelines, and compared with a synthetic counterfactual constructed from the England average for cardiology services, to isolate the effects of the solution on clinical outcomes such as readmissions and Did-Not-Attends (DNAs) using a difference-in-difference approach. The patient data consisted of blood pressure, pulse, and weight readings provided by patients on the remote monitoring pathway through the app, attached to a pseudonymised patient-ID and a date stamp. This data was analysed to assess adherence patterns and trends in blood pressure and pulse among patients on this pathway.

The health economic analysis involved a cost-benefit analysis, which reported the key costs and benefits of the solution as monetary values, from a social perspective. The analysis followed the methodology in The Green Book (HM Treasury, 2022), and applied optimism bias, inflation adjustment, and discounting to the costs and benefits. The main unit cost databases that were used to source data included: Personal Social Services Research Unit’s (PSSRU) ‘Unit Costs of Health and Social Care 2021’, Unit cost database, and National Institute for Health and Care Excellence’s (NICE) resource impact template. Other sources used included HES, the NHS National Cost Collection database, General Practice organisational and population data, and academic literature. The analysis included five benefit streams: reduction in readmissions, reduction in DNAs, reduction in paper usage, reduction in postage costs, and reduction in carbon emissions; and four cost streams: licensing costs, staffing costs, development costs and implementation costs.

The results indicated that the app was well regarded by patients and by staff. It is effective in its role, adds value to users, imparts valuable information and is user-friendly. This was further supported by the health economic analysis, which indicated a positive return on investment in all scenarios modelled.

The evaluation provided recommendations to the leadership team for the app to support future implementations, mostly relating to preventative and pre-emptive actions that anticipate likely risks and challenges with further spread and adoption.

What were the impacts of the evaluation?

Utilised a difference-in-difference method to quantify key patient impacts

Difference-in-difference methodology involves examining the changes in a variable between two timepoints within a group receiving the intervention and a control group. From this data, a counterfactual may be constructed which represents the trends in the control group of the chosen variable, but from the same baseline as the intervention group. This allows for an estimation of what values of the chosen variable would have looked like in the intervention group had the intervention not been implemented.

As shown in the above graph, the counterfactual follows the same pattern over time as the control group, but is anchored to the start point of the intervention group, allowing for accurate assessment of the intervention effect.

In the case of this evaluation, difference-in-difference methodology was utilised to quantify the effects of the intervention on hospital admissions and DNAs.

Used routine healthcare datasets to demonstrate value

Data from Hospital Episode Statistics from the period of April 2021 to April 2023 was utilised to perform difference-in-difference analyses. Other public data sources were also leveraged, including the NHS National Cost Collection dataset and the Unit Costs of Health and Social Care.

Estimated the return on investment to the NHS of adopting the digital tool

A cost-benefit analysis was performed to assess the return on investment to the NHS of implementing the tool at the pilot site, as well as at an additional site close to the pilot site, and across the whole of England. The results of this modelling could be used to support decisions on the further commissioning and spread of the app to cardiology services throughout the NHS.

What value did this evaluation add?

The evaluation was able to demonstrate that the app was well regarded by patients and by staff. It is effective in its role, adds value to users, imparts valuable information and is user-friendly. This was further supported by the health economic analysis, which indicated a positive return on investment in all scenarios modelled.

The evaluation provided recommendations to the solution’s leadership team to support future implementations, mostly relating to preventative and pre-emptive actions that anticipate likely risks and challenges with further spread and adoption.