Understanding the impact of Magic Notes through validating evaluation findings

How we worked with Beam and Kent County Council

Kent County Council implemented Magic Notes (a generative AI tool for recording, transcribing, and summarising discussions and assessments in social care) over an eight-week period (the test and learn pilot). In total, 29 staff members used the tool to complete care needs assessments, occupational therapy assessments, or used the tool for general uses (such as team meetings).

Beam conducted an evaluation to explore whether Magic Notes delivered benefits to social care staff. Unity Insights were commissioned by Beam to validate their methodology and findings. The validation approach taken leveraged Unity Insights’ experience as an independent evaluator, alongside the NICE Evidence Standard Framework (ESF), which informs the evaluation of digital health technologies (DHTs) for use in the health and care system. The insights gathered from Beam were analysed against relevant standards to determine the level and quality of evidence in fulfilling each standard.

What were the impacts of the evaluation validation?

Methodology and findings

The analysis approach used to examine usage data was overall appropriate, however there was room to improve the robustness of analysis. Statistical testing was not conducted on the usage data, meaning no statistical conclusions could be drawn. Survey questions appropriately answered the evaluation questions, however demographic and occupational data was not collected, meaning there was no indication of whether the sample was representative of the wider staff population. Staff working group responses were also not analysed through thematic or sentiment analysis, limiting the robustness of the analytical method chosen.

Beam’s evaluation confirmed that Magic Notes reduced administrative time (6.8 hours saved per week) and improved work quality (staff rated their quality of conversations as 9.1/10 with Magic Notes, compared to 6.8/10 before Magic Notes), with minor limitations in the analytical method not affecting the overall interpretation of findings. Beam’s evaluation suggested that Magic Notes improved the quality of conversations and written documentation, though further analysis is needed to further understand staff and client perceptions.

Assessing evaluation findings in relation to the NICE ESF for DHTs

Evaluation findings suggested that Magic Notes was acceptable to its users and yielded staff time savings. Staff suggested no negative impacts on people that draw on care and support or services, and that Magic Notes was successfully integrated into their routines. This suggests the evaluation contributed towards fulfilling standard 15 (show real-world evidence that the claimed benefits can be realised in practice). The evidence base could be strengthened by conducting a time-and-motion study and replicating the evaluation in further implementation areas.

Beam measured weekly usage of Magic Notes. Continuation of usage data monitoring should be organised alongside Kent County Council to contribute towards standard 16 (the company and evaluator should agree a plan for measuring usage and changes in the DHT’s performance over time). As Magic Notes uses AI, Beam and Kent County Council should agree on post-deployment reporting of changes in performance, processes for measuring performance, and agreeing who, how, and when performance changes should be reported. This will further contribute towards standard 16.

What value did Unity Insights add?

Unity Insights were able to independently validate the evaluation findings collated by Beam to determine whether the conclusions were accurate and suitable based on the methodology used and the analytical method adopted. It is important for an independent validator to assess the validity of an evaluation conducted by the same party that created the innovation, because this helps reduce bias and ensures that the findings are objective, credible, and trustworthy. This will help decision-makers understand whether and to what extent Magic Notes would be beneficial to implement within other areas.

“We are delighted with the impacts seen during the Magic Notes Test & Learn and excited about the opportunity to scale these benefits across our service. By saving valuable time, Magic Notes enables our staff to focus on what matters most – offering tailored support and building stronger relationships with people who draw on social care and support. The response from staff has been fantastic, and it’s clear this approach can help us not only improve efficiency but also enhance the quality of care and support we provide for our Kent residents.”

Tricia Pereira, Head of Service Improvement & Interim Assistant Director Strategic Safeguarding, Practice, Policy & Quality Assurance; Adult Social Care and Health (ASCH) at Kent County Council 

Relevant documents

Magic Notes Validation Report

Magic Notes Infographic