The document discusses gauging long-term impact in the social sector. It outlines how SOS CV measures the impact of its programs to ensure effectiveness and quality improvements. Impact is assessed along three components: impact on individuals, communities, and society. Non-financial impact is measured through interviews with stakeholders, while financial impact is measured through social return on investment. A pilot study found that for every €1 invested, SOS CV programs return €4-€6 in benefits, demonstrating their positive long-term impact.
2. 1Gauging Longterm Impact in the Social Sector.pptx
To ensure the best results for the children in our programmes,
we need to measure our impact
Impact
Transparency
Quality
improvements
At SOS CV, we strive to
ensure the best results in
lives of child participants.
If we measure the effects of our
work, we will create transparency
of our performance.
This is a precondition ...
... to improve or maintain the quality
of our programmes. This ensures we
maximumise the effectiveness of our
work.
Source: Icons designed by www.flaticon.com
3. 2Gauging Longterm Impact in the Social Sector.pptx
Our work and the corresponding inputs have three different effects
over time
Input ImpactOutcomeOutput
All resources spent for
programme interventions
Immediate effects of
our programmes
Short-/medium-term
effects of our work
Long-term effects on both
children and communities
For example, the
children who have good
living conditions in their
SOS families
For example, the
financial and human
resources invested in
family-based care
For example, children in
SOS families have a
good educational
foundation by the time
they leave our care
For example, when living
as independent adults,
they can take care of
themselves and their
families
4. 3Gauging Longterm Impact in the Social Sector.pptx
Our work and the corresponding inputs have three different effects
over time
Input ImpactOutcomeOutput
All resources spent for
programme interventions
Immediate effects of
our programmes
Short-/medium-term
effects of our work
Long-term effects on both
children and communities
Impact Assessment focuses on long-term impact
and thereby helps us achieve the best results for
our children in our programmes
For example, when living
as independent adults,
they can take care of
themselves and their
families
5. 4Gauging Longterm Impact in the Social Sector.pptx
Now,
let's shift our
focus to Impact
and how we can
measure it
6. 5Gauging Longterm Impact in the Social Sector.pptx
We assess impact for our core work in
Family Strengthening (FS) and Family-Based Care (FBC)
Family Strenghtening Family Based Care
7. 6Gauging Longterm Impact in the Social Sector.pptx
We assess three components of impact, considering both non-
financial and financial impact of our programmes
Impact on individuals, the community and the society
Long-term
effects
on the
community
Long-term
effects
on individuals
and
their families
Long-term
effects on the
society
(Social return on
investment)
8. 7Gauging Longterm Impact in the Social Sector.pptx
Non-financial impact is measured primarily by interviewing stakeholders
Score
2 31 4
doing
well
not
doing
well
Former (and
optionally current)
participants are
interviewed and
answers are scored
whether they are
doing well or not
Impact on the
individual
Impact on the
community
Different
stakeholder groups
are interviewed,
e.g., local
authorities, other
NGOs/CBOs and
community leaders
9. 8Gauging Longterm Impact in the Social Sector.pptx
Protection & social
inclusion
Impact on the individual is measured along dimensions holistically covering key aspects
of well-being
Care
Physical
health
Food security
Accommodation
Education & skills
Livelihood
(economic security)
Social & emotional
well-being
Source: Icons designed by www.flaticon.com
10. 9Gauging Longterm Impact in the Social Sector.pptx
Complementary dimensions are considered to assess the impact of our work on the
communities that we support
Source: Icons designed by www.flaticon.com
Community awareness
Alternative care
Progress towards
sustainability
Next-
generation
benefits
Giving &
volunteering
Impact on community
via individual
Impact on community
via programmes
Community-based support systems
11. 10Gauging Longterm Impact in the Social Sector.pptx
For financial impact, we measure the
social return on investment
Social return on investment
(SROI)
The social return on investment (SROI)
measures how much benefit an
investment creates for society
How many dollars worth of benefit
for society are created
by my 1-dollar investment?
12. 11Gauging Longterm Impact in the Social Sector.pptx
Financial impact is assessed by comparing costs and benefits of a programme
:Benefits
The SROI quantifies the benefits
of a programme for the broader
community ...
Costs
... and compares it to the costs of
the programme
Non-quantifiable
elements are
excluded
Conservative
assumptions
are applied
13. 12Gauging Longterm Impact in the Social Sector.pptx
The approach has been successfully applied
We tested the approach in two well-established SOS CV programme
locations: Hawassa, Ethiopia, and Mbabane, Swaziland.
The approach worked well and is a good framework for future assessment.
We even learnt how to further improve the methodology.
While we were impressed by the impact we are having in these locations,
areas for development were identified to further improve our work.
14. 13Gauging Longterm Impact in the Social Sector.pptx
Targeting the most vulnerable children, who have limited prospects
for future success, the results confirm our impact
The pilot study provided insights of SOS having some very positive impact on the situation of the children
Between 70 and 95% of former participants reportedly
perform well in at least six of eight dimensions
Especially the scores in education and food were
exceptionally high
Local stakeholders highly value the work of SOS
CV staff in terms of
• being a key player in child protection
• strengthening networks
• building capacity and sharing knowledge
On an individual level On a community level
15. 14Gauging Longterm Impact in the Social Sector.pptx
In particular, nearly all participants of FBC perform well in at least 6-
out-of-8 dimensions — an indication of the ‘value added’ by our work
0 50 100
70%
All dimensions
75%
At least 7/8
dimensions
90%
At least 6/8
dimensions
95%
Individuals doing well in key dimensions (in %)
30%
30%
FBC (n = 20) FS (n = 10)
0 50 100
46%
58%
92%
84%
38%
42%
FS (n = 13)FBC (n = 19)
SwazilandEthiopia
16. 15Gauging Longterm Impact in the Social Sector.pptx
Every €1 invested returns at least €4 to €6, depending on location
SOS CV programmes Ethiopia
SROI of 4:1 SROI of 6:1
Swaziland
Potential for additional upside:
Conservative assumptions applied
Actual impact even higher:
Non-quantifiable elements excluded
17. 16Gauging Longterm Impact in the Social Sector.pptx
Please do not hesitate to contact us if you want to learn more about
our social impact assessment approach
Douglas
Reed
Germain
Houedenou
Douglas.Reed@sos-kd.org
Germain.Houedenou@sos-kd.org
Peter
Fechner
Fechner.Peter@bcg.com
Alexander
Baic
Baic.Alexander@bcg.com
Notas del editor
A key question that we have to ask ourselves is “How can we make sure that the children participating in our programmes receive the best support possible?” and “How can we make sure that this support has a long-term impact?”.
The basic answer is that “We need to measure the effects of our work in the first place”. This is to make our impact visible, so that we can maintain and improve the quality of our work, and so achieve sustainable results into the future.
When measuring impact, we first need to be clear on what it means. There are various ways of looking at impact in social development work. One of them is the results chain. Let's have a quick look at it:
Animate chain, start with input
It all starts with the input that we invest into our programmes, to carry out certain activities – for example, in offering family-based care through SOS families.
Animate chain, follow output
These inputs and activities lead to some immediate effects, which are the outputs – you can think of the number of children who are actually living in SOS families.
Animate chain, follow outcome
These outputs lead to certain results in the short-to-medium term, which are the outcomes - for instance the quality of education that children receive while growing-up in SOS families, and can rely on when they leave the care of SOS.
Animate chain, follow impact
The impact, is then the long-term effect of our work. Ideally, a child who has grown-up in an SOS family goes onto have a higher standard of living than he or she would otherwise have had, and is relatively healthy and happy. In the best case, he or she even passes these benefits onto his or her own children, multiplying our investment and ensuring sustainable impact.
[animate transition with previous slide: fade out of the elements on the left, fade in new elements]
Impact Assessment aims to measure exactly these long-term effects.
It can help to find out if we do indeed make the best use of our resources, to bring about the best results in the long-term. Put another way, it tells us if we deliver on our goals, ensuring that we achieve the most that we can for the children in our programmes.
Now, let's shift our focus to impact and how we can measure it.
I assume you would agree that this is good idea and something worth investing in.
Now, let's take a look at how it can be done.
In impact assessment, we will focus on the two core areas of our work: Family strengthening and family-based care.
Animate, show both boxes and pictures
In these areas of work, it is relatively easy to identify former-child participants and supported communities. We can then assess the impact in their situation.
Let's go back to the results chain.
Animate chain (process bar).
You will recall we said that we will focus on impact.
Animate chain, follow impact.
How we see impact is made up of three key elements: Firstly, our work has an effect on the individuals who participate in our programme and their families
Animate chain, individual box.
Think again of the child in the SOS family who, eventually, grows up into an independent adult, with a good job and has a stable family.
-
Secondly, we look beyond the individual, to the long-term effects we have on the communities we work in:
Animate chain, community box.
We work with many stakeholders in the community to build strong support systems for the children of our target group and their families.
-
Thirdly, these positive results in the lives of individuals and their communities translate into certain financial benefits for society:
Animate chain, SROI box.
-
The children who participate in our programmes will – hopefully – in future earn their own money, take care of their own families, pay taxes and contribute to their community.
Also, if communities are better able to support vulnerable children and their families, then the government shall need to spend less on alternative care and social support, and so save money.
These benefits make up what is known as the ‘social return on investment’.
We need to invest some time researching, to find out what our impact actually is. The methodology is fairly simple: We talk to the people involved.
Okay, it's a bit more sophisticated than that. Regarding the impact on the individual, we interview a sample of former participants.
[animate gray call-out on the left]
A prepared questionnaire helps us to always ask the same questions. The answers of the former participants are then scored. The logic is simple: Either they are doing well – or not. I will give you an idea of what we ask them about on the next page.
Wherever we need to dig a bit deeper, to paint a clearer picture of the situation, then we can also bring together former-participants together for group discussions.
We also talk to communities.
[animate gray call-out on the right]
Well, of course not the communities as whole, but key stakeholders involved, including local authorities and community leaders, service providing partners and local staff. Again, we talk to them through individual interviews, or together in groups.
Let's get an impression of what we assess, so that we can to piece together a picture of the impact on the individual.
We identified eight dimensions that cover different areas of well-being of a former participant.
Some of these dimensions are quite ‘soft’, such as care, social and emotional well-being, and protection from discrimination, abuse and exploitation. For care, we focus on whether children are being properly cared for and protected by their parent or other care-giver.
The ‘harder’ dimensions tend to be more easy to measure, such as livelihood, education and skills, food security, accommodation and physical health. For livelihood, we look at the economic situation of the individual, finding out whether there is enough money to cover household and family expenses. Regarding education and skills, we are interested in educational attendance and performance, and the potential outlook on the job market.
As mentioned before, this means of assessment is tailored to our organisation and the way we are working in our programmes.
The dimensions underlying the community assessment are a bit more complex. Here, we want to assess what has changed in a community, in terms of responding to the situation of our target group, ‘before’ an ‘after’ SOS has engaged with that community.
Direct effects that we are looking for are general awareness within the community of our target group and the issues affecting them. Ideally, we would like to see a support system for our target group, which is embedded within the community itself. This may include such things as home-based care groups for orphans and vulnerable children, and community-based child protection mechanisms. We also aim for sustainability of such responses to the situation of our target group, such that relevant support shall continue even when SOS withdraws from day-to-day involvement in that community.
Beyond the direct effects, we assume that our work also has some indirect effects via (former) participants. These might include former-participants being more willing to give of own their time and money to address issues in their community, as well as expected ‘next generation benefits’ if they are better able to give their on children a good upbringing.
Certainly, this may sound a bit abstract, but once you talk to key stakeholders ‘on the ground’, you quickly get a good picture of SOS's impact in the community.
For financial impact, we measure the social return on investment.
The basic concept is fairly simple, since we want to measure the benefit that one Dollar creates once it is invested into one of our programmes.
[animate blue box]
As such, the question that we ask is „If I invest one dollar in the programme, how many dollars worth of benefit will be generated for society?“
To fully answer this question, we need to put a monetary value to the benefits that a programme has for the individual and for the broader community.
Think, for instance of the income that a former participant earns today and how more he or she can expect to earn until retirement.
We can also stretch that, to include the income of his or her children. Because the former-participant benefitted from the programme, he or she may be able to invest more into their own children’s development, so that their children have better prospects in life and themselves earn a higher income.
We can also add the income of the staff who are employed by the programme. Or include donations by former participants.
If we add all of this together, we can get to quite a sum of money.
On the other hand, we need to sum up all of the costs of the programme.
When we compare the benefits and the costs it tells us the social return on investment.
To make this calculation as realistic as possible, we base it on some conservative assumptions. [animate left gray call out]
Given the fact that we can't put a monetary value to every single benefit experienced by individuals and the community [animate right gray call out], we can expect the actual social return on investment to be even higher than the reported one.
We have tried the approach and found that it works ‘on the ground’, giving a picture of the impact of the programmes involved.
To give you an idea of what the results of an Impact Assessment can look like, let's take a few minutes to reflect on some of the key findings from the pilot assessments at Hawassa in Ethiopia and at Mbabane in Swaziland.
Before you go further, you might want to read through the information provided.
Not only the Impact Assessment process, but more importantly the programmes themselves can be seen to deliver on some key expectations.
[animate: fade in images]
The pilot study gave us valuable insights, both on an individual and on a community level.
[animate: fade in text below images]
Depending on the location, and whether we are looking at family strengthening or family-based care, between 70 and 95% of former participants are performing well across at least six-out-of-eight dimensions.
This is certainly a result that we can feel good about!
Remember, we are working with the most vulnerable children, who can’t generally be expected to do as well as other children, unless they receive significant support.
Given these results, it’s perhaps not surprising that local communities also value the work we are doing.
Of course, not all of the results were perfect, as there is no perfect programme! The programmes involved gained some valuable insights into areas of their work that can be improved and they are now been able to address these issues in their planning for the future.
Before we take a look at the process, let’s dig a bit deeper into some of the main results.
Of course, if you are interested, you can also download the detailed report on the pilot results by clicking on the icon at the lower right hand corner of the screen.
Let's take a closer look at the results on the individual level. How are the former participants doing?
As already mentioned, nearly all participants performed well in at least six out of the eight dimensions. What is more, at least 30% are doing well in ALL eight dimensions. For FBC in Ethiopia, three quarters of all participants are doing well in all dimensions. If you are interested in more concrete information, click the orange buttons to view examples of the results.
You may have noticed that the results in FBC are generally higher than for FS. This is to be expected, as the intensity of support in FBC is that much higher.
The results for the social return on investment are also looking pretty good!
These indicate that for every one Euro invested into the Children’s Village programmes in Ethiopia and Swaziland, a benefit of at least 4 Euros came back to society, with as much as 6 Euros for Swaziland.
While this already looks good, we need to keep in mind that the actual impact is higher: [Animate, show text on the right]
…As we did not include in the calculation those benefits for which it is difficult to give a monetary value. If we were able to add these in, the figure for social return on investment would be higher.
Also, it could be even more [Animate, show text on the left]
….As we used conservative assumptions in making the calculation, so that it would be as realistic as possible and not inflate the results.
Now, let's see what we can use these results for.
It’s great to know that impact has caught your attention.
We believe that impact should be everyone’s business in SOS, given the immense value it can add to our work. If you feel the same, there are several ways you can get more directly involved.
Let’s take the next steps together, in making bigger impact a reality in our programmes.