How to set an evaluation up to succeed: Supporting RateMyShift in NHS Wales

Strong evaluations are rarely rescued at the analysis stage. They are set up to succeed much earlier by agreeing what the intervention is expected to change, asking focused questions, selecting feasible measures, and making responsibilities clear before data collection begins.

Unity Insights developed an evaluation scoping plan for the RateMyShift pilot at Swansea Bay University Health Board, in partnership with Health Innovation West of England and funded by the University of the West of England. RateMyShift is a real-time workforce intelligence platform that captures shift-level staff feedback, enabling managers to respond to issues, support staff, and gain insight into team conditions and workforce experience over time. The pilot brings together the RateMyShift workforce intelligence platform, coaching for managers, and workforce transformation support from Health Education and Improvement Wales (HEIW). It is intended to create a cycle in which staff share their shift experiences, managers respond through meaningful one-to-one conversations, and recurring workforce barriers become more visible.

Five steps for setting an evaluation up to succeed

1. Clarify how the intervention is expected to work

Before deciding what to measure, develop a shared understanding of the intervention and how it is expected to create change. This prevents the evaluation from focusing too narrowly on individual activities or outputs without considering how they contribute to the intended outcomes.

What we did: For this piece of work, we considered the RateMyShift platform as one part of a wider intervention. The evaluation scoping plan explored how staff feedback captured through the platform could prompt supportive conversations with managers, with coaching helping managers respond constructively. It also considered how aggregated insights could inform wider workforce transformation support from Health Education and Improvement Wales. This ensured that the evaluation reflected the relationship between the different components rather than assessing the technology in isolation.

2. Develop a theory of change

A logic model can provide a visual foundation for an evaluation by showing how an intervention’s activities are expected to lead to change. Developing the underlying programme theory collectively allows stakeholders to test the assumptions within this pathway and agree what success could look like before data collection begins.

What we did: RateMyShift had already developed an initial logic model, which provided a valuable starting point. Unity Insights facilitated a workshop with the project team to build on and refine this existing model. The workshop explored the anticipated impacts and outcomes in greater depth and identified the measures that could demonstrate progress. The refined logic model (below) then informed the evaluation questions and wider measurement framework.    

3. Translate ambitions into focused evaluation questions

Evaluation questions should establish exactly what the evaluation needs to find out. Keeping them focused makes it easier to identify the evidence required and prevents data collection from becoming unnecessarily broad or burdensome.

What we did: We translated the intended outcomes of the RateMyShift pilot into three evaluation questions. These focused on whether managers were using the platform and coaching support, whether staff felt heard and experienced improvements in their working conditions, and whether engagement and trust were sustained over time. The questions provided a clear structure for the evaluation and were used to determine which measures should be prioritised.

4. Prioritise the evidence that matters most

Not every possible measure needs to be collected. A proportionate evaluation should identify the minimum evidence required to answer its key questions, while retaining additional measures that can provide further insight where capacity and data availability allow. Consistent with UK GDPR principles, data collection should be limited to information that is necessary, relevant, and proportionate to the evaluation’s objectives, ensuring that all data collected serve a clear and legitimate purpose and are not excessive for the intended use.

What we did: We developed a measurement framework that distinguished between essential and supplementary measures. The essential measures included whether staff continued to complete shift ratings, whether flagged ratings received follow-up, and how quickly managers responded. Survey questions were also included to explore whether staff felt heard and whether managers felt confident conducting supportive one-to-one conversations. This gave the team a manageable evidence set while preserving opportunities for more detailed analysis.

5. Plan how the evaluation will work in practice

A strong evaluation plan should explain how the work will be delivered, not simply what will be measured. Practical arrangements should be agreed before data collection begins so that responsibilities are clear and avoidable gaps do not weaken the findings.

What we did: The evaluation scoping plan set out when data should be collected and clarified the responsibilities of the organisations involved. It also addressed information governance, data quality, and transparent reporting. Practical risks were considered in advance, including low survey response rates and differences in how staff groups or implementation areas might be recorded across datasets. For example, we recommended agreeing consistent categories across the platform, survey, coaching, and organisational data so that findings could be compared accurately.

Conclusion

Evaluation planning is not an administrative prelude to the “real” evaluation. It is the work that makes credible evaluation possible. By clarifying how an intervention is expected to work, prioritising feasible evidence and planning delivery in advance, teams can reduce avoidable gaps and produce findings that are more useful for learning and decision-making. While our evaluations can include more complex methods such as statistical testing, health economic analysis, and system demand modelling, the appropriate approach depends on the evaluation questions, available data, and intended users. For this project, the priority was to provide a robust methodology that the delivery team could implement themselves. This work shows how a concise, theory-based plan can make evaluation accessible to a delivery team without an evaluation background, while preserving the structure needed for more sophisticated analyses in future phases.