Rewind to 2020 - what would we have done differently?

Hindsight is a wonderful thing and this blog is about sharing the things that didn’t go quite so well throughout Catalyst and The National Lottery Community Fund COVID-19 Digital Response, so that others can learn from our mistakes. The following points come mostly from our evaluation partner, inFocus, who spoke to and surveyed charities and digital partners from all of the fund’s programmes. It’s also from our internal reviews and retrospectives. So before I get into it, I want to share the prime directive that we use for all of our retrospectives at CAST and Catalyst, as it is something I really stand by:

"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."

Here are the top six learnings that I’m going to dig into in more detail in this blog:

1. Manage expectations through clearer briefs and detailed introductions
2. Take time to establish good partnerships between charities and agencies
3. Combine Discovery with Definition (with a short break in between)
4. Understand and cater for different access needs
5. Be more flexible around timings
6. Talk about sustainability earlier

1. Manage expectations through clearer briefs and detailed introductions

From the inFocus Sector Challenge evaluation:

• “During the surveys the most common area of improvement touched upon by the charities was that there was a need for more clarity around the project brief and purpose of the programme. These charities widely shared the view that their expectations could have been managed better, in stating that products may not be finalised by the end of the programme and in the potential need for one group to own the product and/or apply for additional funds for it.”

• “Another area of improvement revealed by five charities during the surveys was that there was a need for a more ‘directive start’ to the programme. In order to overcome teething issues, charities felt that there was a need for more conversation at the beginning to gain a better understanding of other charities within the clusters and to define a shared understanding of the workshop format and co-production.”

We learnt after the Discovery Programme that we needed to be better at conveying who is going to be involved, what their roles are within the programme and when they will be in touch. We had a multitude of stakeholders involved in our programmes: the funder’s evaluation partner, CAST’s external evaluator, the CAST/Catalyst point people, the open working team (also known as the OWLs), the digital experts, and more. This was confusing enough for us - so we can only imagine how the grantees felt.

We learnt and responded in later programmes by explaining in the kick off session exactly who was involved, what their role in the programme was, and what the expectations were from the grantees to engage with these people.

• “​​It would be great if charities had more support from CAST/Catalyst in writing their briefs and on discovery work. I think we would have been able to hit the ground running if we’d had a more clearly defined brief.” - Digital Partner

We also learnt that we needed to make sure expectations were clear through producing watertight, crystal clear briefs and project plans. Some were great, others were harder to get your head around as a digital agency that was wanting to get involved.

‍‍

2. Take time to establish good partnerships between charities and agencies

The Sector Challenge Programme was the most innovative of the programmes, and so not surprisingly the one where there were the most challenges. The hypothesis was that by grouping different charities together to work on a sector-wide challenge, that they could together define a solution that benefits everyone. 

We partnered them with a digital expert whose responsibility it was to steer the thinking, facilitate the partnership and define a solution. It was a huge success in terms of engineering collaboration between diverse organisations, but it felt rushed, far too time intensive, and challenging for digital agencies and the charities involved. 

Here are a few things that inFocus suggested be approached differently if it was to run again:

• Put the digital agency firmly in the driving seat. They need to be positioned as the trusted facilitators and service designers that they are. They might not be subject-area experts, but they are the glue that will hold everything together and need to be introduced and positioned in the right way from the outset.
• Give plenty of time to establishing ways of working and getting to know each other. We grouped small charities with large ones, all with different experiences of the problem space. Time is needed to get to know each other’s strengths and agree how you will work, the tools you’ll use and how you will go about reaching consensus and making decisions.
• Take time to reach a shared understanding of the purpose and outcome of the work. Without that, there is ambiguity and confusion, which is a shaky foundation to start from.‍

‍3. Combine Discovery with Definition (with a short break in between)

Organisations were offered the opportunity to join the newly-created Definition Programme after they had completed the Discovery Learning Programme. The two programmes were quite distinct which resulted in grantees working with different groups of charities and different digital partners each time. 

The cohorts for the Definition Programme were more aligned in terms of what they were trying to achieve (for example, there was a cohort of seven charities purely focussed on developing an e-learning solution). However, the digital partners on the Definition Programme found it challenging having not previously been through the discovery process with the charities that they were grouped with. 

• “It was like starting again with some of the charities as the time between them being on Discovery and starting Definition was too long” – Digital agency.

In hindsight, the continuity would have been beneficial for both grantees and charities.

From the inFocus evaluation:

• “More energy should have been spent on handing over the charities to the digital partners. This was partly due to the fact that the charities would have found themselves with different partners to the ones they had been working with on the Discovery project. In some cases, the new partners may have taken a slightly different approach and may have had a slightly different way of working. That was hard for some of the charities.”

The recommendations from inFocus included:

• “Combine Discovery and Definition Programmes – with a short break built in.”
  

• “Better transition between Discovery and Definition programme - make it more seamless but reduce the gap between the start and end of the two programmes.”

4. Understand and cater for different access needs 

• “Some of the mapping tools were a little difficult for me having Dyslexia” Participant in the Definition programme.

• “The main challenges I faced when taking part in the programme were surrounding having difficulty following some of the activities that we had to do as a group due to my own specific learning styles and ways of understanding information. “ Participant in Sector Challenge Programme

We used Miro when some people had never seen it before. I remember my first time on Miro - I’m pretty sure I accidentally deleted the entire board. In a group environment, it’s not always the best option.

We also put links into the Zoom chat, which people with a sight impairment couldn’t easily access. 

Now we always ask before a workshop about access needs, we subtitle all of our videos, steer clear of Miro in workshops (or screen share), send the written transcript of the workshop in a document ahead of time and generally try to follow the golden rule to never assume when it comes to tech tools. Earlier this year, we convened an event on digital accessibility, to help us - and the wider network - understand best practice.  Of course, there’s every chance we’ll make mistakes in the future, and if we do then please tell us.

And another thing which we’re still learning to get right is language. Those of us who have had jobs in digital agencies or been part of digital teams are so used to jargon that we barely notice ourselves using it. Most of the sector works in digital services in one way or another, so by using plain language it empowers people and builds confidence and inclusivity. You can find more on this in the jargon-busting section of our site.

5. Be more flexible around timings

“There was a lot to take in within 10 weeks - it is a short period of time and doesn't allow for staff leave/public holidays/capacity of small teams within organisations and capacity of digital agency partners” Participating charity

“The pressure to complete the project before the end of the financial year definitely had a negative impact on the outcome of the project.” Participating charity

“Include a week break half way through the programme to allow people to catch up and take a pause! More opportunity to meet with different organisations in other cohorts.” Participating charity

“The timings of the session were really incompatible with being a working mum. I was made to feel like a nuisance - all our sessions started at 3pm (school pick up time). It hadn't been considered.” Participating charity

A four-week Discovery programme is too short. It should have been six weeks. The time it takes to recruit and schedule user research participants can take two weeks alone. And having Christmas/Summer holidays bang in the middle of a Development Programme is super-disruptive. We should have planned for that, by adding a bit of extra time to catch up before and after the holiday period. 

For the Definition and Development Programmes, the programmes were a fixed 10 weeks. There was an intensity there, but that kept the momentum going and the end goal in sight. For some it was challenging because they were relying on the availability of service users to test their solutions. For others, school holidays and furlough had an impact on the capacity of organisations. Looking back, I think deadlines are important in creating pace and building momentum, but adjustments should be anticipated and flexibility built in.

A bit of extra dedicated pre-planning time for the digital partners to get to know the charities would also have been helpful. And at the end, some dedicated ‘wrap up’ time for the charities to share their story and outputs more widely might also have been a good idea.

“There is a high amount of contact time with long meetings, which is nice but also very demanding on my time and can be knackering when over Zoom!”  Participating charity

‍

6. Talk about sustainability earlier

“The other piece is the unknown around the future of the product. It feels like we’ve had to ask those questions and push those things forward. We have experience in it but it would be nice if they (Catalyst/CAST) owned that as it’s their product more than our product. I don’t know what I think about that but we’ve had to ask about what happens next and the plan.” Digital agency working on Service Layers/Platforms.

We commissioned some incredible digital partners to deliver work for the sector products that could be used across the sector (Notifications, for example). In hindsight, we should have talked to the partners earlier about continuation and sustainability. But with further funding uncertain, this was deprioritised over delivery of what was in scope. This has led to a pause in progress between funding periods, although without immediate funding I’m not sure how that could have been avoided. 

Thankfully, CAST and Catalyst have received further funding to continue building on the outputs of the past year - including the Open Working toolkit, Service Recipes, Open Referrals, Notifications, Resource Hubs and the Data Collective. And we are prioritising sustainability models as part of the work going forward.

‍

Summary

It’s a real challenge to try and summarise a year’s worth of learnings from multiple programmes and evaluation reports. But flexibility around timings, setting and managing expectations, and building strong partnerships from the start definitely stood out across all of the feedback we received. 

Also, considering accessibility and sustainability purposefully and early may seem obvious but when you’re delivering at pace and at scale, these are the things that can slip down the priority list.

We all hate to get things wrong but I have to admit, being able to write this blog and feel confident about these six improvements feels like quite a victory. And hopefully a little bit useful to others looking to run digital funding programmes in the future.

‍

Visit the 'Digital Fund Handbook' for more information about the programmes delivered as part of the Catalyst and The National Lottery Community Fund.

‍

Image by Arnesh Yadram from Pexels