Evaluation issues in the NSW not-for-profit sector: Workshop Report

The Workshop

NSW AES Evaluation Society, Evaluation workshop, 16 May 2007, Mission Australia, Bathurst Street, Sydney

An evaluation workshop was held on May 16 at Mission Australia for not-for-profit organisations to present and discuss issues, especially those concerned with conducting multi-site complex human program evaluations. The following package is a record of these presentations and the discussion that followed after. The issues raised were further discussed at The Centre's June Networking Breakfast held on Tuesday, 19 June 2007.

• Evaluation Workshop notes
• Presentation: Youth Off The Streets (YOTS)
• Presentation: Evaluation at The Benevolent Society
• Presentation: UnitingCare Burnside
• Measuring progress in YITAV communities – how will we know that we're making a difference? (YWCA NSW, Research and Innovations in collaboration with Southern Cross University, Centre for Children and Young People)

Evaluation Workshop notes

1. Presentation overview: Evaluation issues in the non-profit sector
2. Case studies: Each Speaker addresses an aspect of how they deal with conducting multi-sited complex program evaluations. For approximately ten minutes each they will discuss where they are in terms of evaluation and the challenges they face.
3. Discussion and feedback

Evaluation issues have arisen as an area of importance to not-for-profit organisations, especially those that conduct multi-sited complex human service programs. This workshop responded to a need in the sector to network and identifies evaluation issues.

1. Youth Off The Streets (YOTS)

• YOTS is exploring ways to build an evaluation culture into their organisation. Capacity building is one of the biggest challenges.
• A key pressure YOTS faces is the need to provide more "professional" evaluations to funders and potential funders that demonstrate they are meeting the needs of their target population, in context of limited funds for more sophisticated evaluations

Challenges include:

• building in time and resources to design and implement evaluations and train program staff
• convincing funders that some of their contributions are valuable if directed towards evaluation rather than only service delivery.
• building trust with young people so they feel comfortable giving feedback about YOTS services.

Group discussion

• A small community organisation reinforced that there is a perennial challenge of identifying and measuring the nature of what are successful outcomes. There is a challenge between meeting the evaluation requirements of legislative and regulatory funding bodies and identifying for YOTS whether they are meeting the needs of the young people in their programs. This issue was described as YOTS providing 'feel good and warm fuzzy stories' for corporate funders and corporate volunteers but also needing to know how YOTS is positively impacting the lives of young people.
• Another issue is measuring different outcomes for different programs /clients of their services.
• Who decides what success is for a client? For YOTS clients and staff getting to school, getting out of bed, learning skills for seeking personal/emotional help can be success measures- as well as what they want to get out of being part of a YOTS program i.e. client directed success factors rather than industry/funder/service-provider directed success factors.
• Another challenge discussed was evaluating young people who are often transient and/or may only briefly access YOTS (or a similar youth service). One person suggested an evaluation strategy of creating a client pathway with checkpoints. Reaching a checkpoint is a client success factor even if a client drops out of the service at some point.
• For YOTS, a key issue is keeping long term contact with clients who are transient.
• A question was raised as 'How do you join these competing evaluation outputs to get an overall picture of progress when dealing with various problems in an organisation?'
• The rapid growth of an organisation that is primarily focused on service delivery has made it challenging to build a culture of evaluation.
• The scale of the task in setting appropriate and universal evaluation indicators is large and complex; this requires professional support to establish and implement.
• A major challenge is gathering any longitudinal data on YOTS' clients given many are a transient population.

One organisation suggested looking at self-defined goals for young people as a way of addressing the issue of appropriate indicators across an organisation that delivers a complex range of programs. Another suggestion was made about looking at goal procedures for clients. These would differ individually but as an evaluation process across the organisation would be a robust way of setting relevant indicators.

2. Benevolent Society (Ben Soc): Andrew Anderson

• A key challenge is rolling out a variety of services across NSW.
• Building evaluation capacity in their many programs is what Ben Soc are addressing.
• One issue has been whether Ben Soc should leave evaluation to external evaluators?

Comments

• A question was asked: how does an organisation adapt to model a similar evaluation framework like Ben Soc's?
• Ben Soc has had to keep their framework adaptable to a range of their human services they deliver such as early childhood and aged care.
• One lesson Ben Soc has learnt is that it cannot be too directive on a framework and how it will be operationalised.
• Ben Soc has adopted a centre-by-center basis for developing evaluation frameworks.

• How useful has it been to have their services documented?
• Very useful having a facilitated discussion by an internal or external person to get program staff to discuss goals and assess if they are achieving them.
• This process has assisted Ben Soc work out what the essence of their projects.
• The process of discussing what they do in detail helps change the way of thinking and realising that what they are doing makes a difference.
• There are benefits in having a similar process of program reflection among staff for building an evaluation culture within the organisation.

• Were these evaluation processes taken up enthusiastically or was there resistance?
• For the most part, people are interested but questioned its purpose.
• Depending on the program area, there are varying levels of experience with the purpose and process of evaluation.
• Engaging staff in this reflection process on programs assists creating an evaluation culture.

• How has Ben Soc approached service users' involvement in developing measures and tools for evaluation?
• The process of getting people to discuss helps achieve a participatory outcome.
• This participation may have been easier because of their group they chose to work with.

• What about getting feedback for what doesn't work?
  • Ben Soc is aware of this and is trying to work with those they have funding for.
  • Clients are helping to design data collection mechanisms and processes.
  • Ben Soc has feedback from evaluation experts to try to build a robust set of measures and indicators.

• One organisation in the audience stated the challenges of the shoestring development and the relative luxury of Ben Soc's process. Can something similar be done without needing many resources?

• External vs. Internal Evaluation: what do you prefer or does it depend on the service?
• Depends on the service being evaluated.
• For an internal evaluation, the benefits are building capacity within the organisation and seeking engaging in the process.
• A combination of internal and external because data from an external evaluation is used to discuss internally.

3. YWCA: Jacqueline Shimeld: Research and Innovations Manager

• YWCA NSW operates according to the four principles articulated in the Y It Takes A Village (YITAV) strategy [see Y's evaluation strategy sheet].
• The Y operates a social enterprise: two hotels and a conference centre ('the Hotel that gives back to the community') which supplements the external funding used to develop and deliver human services programs.
• The Y set out to review and refocus its core business to be more effective and meet emerging needs; and avoid being 'everything to everyone'.
• The goal of YITAV is long-term improvements in people's lives and building community through 'increased life skills' and 'community connectedness'.
• The evaluation challenge is how to measure the community wide impact of the collective outcomes of the raft of YWCA NSW programs and services operating under YITAV.
• The Y is at the stage of trialing the evaluation tool developed for the purpose.
• The Y has made evaluation a headline issue and program managers designing new programs are remarking that the indicators of the YITAV evaluation tool assist the articulation of evaluation elements at the program development stage.
• It is more challenging to apply the YITAV evaluation tool to programs that predate the strategy although they are all consistent with the YITAV strategy.

Feedback

Does anyone use any data collection software?

• Some larger organisations said they have built databases themselves.
• Other organisations said they use different software for different projects.

Will the programs be collecting the data for you?

• This YITAV focused data collection will replace the current process of program data collection and bring consistency to the organisation's evaluation processes. The data collection responsibility (and individual program evaluation) will remain with program managers. The collation and analysis of all program data will be the job of the R&I manager.

What do you envision will happen?

• The Y will be able to show better what it does and how effective it has been across communities.
• Individual programs at YWCA NSW will still be collecting the same information but their data will feed into an overarching report of the collective outputs and outcomes of all programs.
• The evaluation tool and strategy has to be dynamic so that it able to respond to changing times and demands. There is a need to experiment with what are the key indicators.
• A challenge for the Y is struggling with the richness of information available – it will be a challenge to effectively collate the qualitative information that so vividly tells the story of the impacts of activity.
• A key intention of the evaluation process is to determine how we can bring process improvement as well as outcome improvements to our program and service provision.

4. Burnside: Karen Bevan: Manager – Social Policy & Research Program

• Strengths-based organisation
• Have about 70 programs that all have different activities and services.
• Provide inequality programs and advocacy to break the disadvantages to vulnerable children, young people, and families.
• Use a social justice framework.
• Working in an environment for change which is critical in evaluation
• Have had a research and evaluation focus for ten years
• First evaluation plan in 2000, changing all the time.
• Agency-wide strategic plan and measuring.
• Burnside asks their services if they have made a difference.
• STARS – tells us how much we did and how well they did it (did they meet the goals they identified).
• Annual service feedback process: Have Your Say.
• Trigger evaluation themselves.
• Should be a positive proactive process

Feedback

• How does evaluation contribute to the broad strategic framework and it's relationship with monitoring
• Key question: what sort of difference did we make
• Some dilemmas: does Burnside get the best value for its investment in particular programs and what is the value of the internet and external evaluations?
• How does evaluation actually drive improvement within an organisation?
• What about the competitiveness [not sure what this refers to: between organisations for funding or is it more about designing evaluation frameworks and implementation plans?]
• Not-for-profits are all different so there needs to be more space for better collaboration in the field of evaluation issues (design, implementation and review).
• It was acknowledged that there are some business realities but they don't need to necessarily discuss them during evaluation collaborations.
• Some organisations said that it is frustrating that there are so many people feeling the same problems and trying to solve the same troubles. 
• It was suggested to investigate developing an electronic network where people can maybe tap in and ask each other (E-community). Other attendees suggested more face-to-face opportunities.
• Capacity building can actually come from within the sector for smaller organisations.
• Some evaluation research demonstrates the benefits of collaboration; others stated this is contradictory to the competitive nature of funding in the not-for-profit sector.
• Complimentary expertise, if we keep operating in our own places it's really hard to address evaluation.
• A suggestion was made for people from one program in evaluation helping others in another organisation.

Return to the national volunteering news