The NSW Government is promoting a consistent approach to program evaluation with the aim of improving programs and providing more rigorous evidence of program outcomes.
NSW Government Program Evaluation Guidelines
The NSW Government is committed to increasing transparency of expenditure on programs and providing a better understanding of their outcomes. One of the reforms to achieve this is the NSW Government Program Evaluation Guidelines, which recognises program evaluation as a key tool to support evidence based policy and decision making in government.
The guidelines are a guide for consistency in evaluation across the sector and sets out:
- different types of evaluation and how they might be used
- a strategic approach to evaluation that prioritises evaluation, and scales evaluations based on the characteristics of different types of programs
- key principles of good evaluation practice
- how findings generated by evaluation can be used for learning and better decision making.
The guidelines also explains how evaluation will be integrated into budget and performance management processes.
The Centre for Program Evaluation and capability building
The Centre for Program Evaluation has key functions under the NSW Government Program Evaluation Guidelines. The Centre has been established in NSW Treasury to conduct methodologically rigorous evaluations of large and significant NSW Government programs (including process, outcome and economic components), to lead evaluation practice across NSW, and to support the NSW Government Evaluation Community of Practice.
Building evaluation capability across the sector is an important function of the Centre. A central feature of this will be the implementation of the Evaluation Community of Practice. The Evaluation Community of Practice will encourage:
- promotion of the role of evaluation in delivering public value for the people of NSW
- an understanding of the strengths and limitations of evaluation, and its role in evidence based decision making
- closer alignment between evaluation and program/policy design
- the sharing of knowledge, expertise and experiences across agency boundaries
- improving skills and the sharing of resources.
The community will be open to all those with an interest in evaluation, but will primarily provide an opportunity for evaluation specialists and program staff, including non-government partners, to engage on discussion, meet periodically to reflect on their practice, share their experience with others, and explore new approaches to evaluation. The CPE will play a lead coordinating role to organise meetings, seminars and disseminate information, in consultation with Department of Premier and Cabinet (DPC).
Principles and standards for evaluation
The NSW Government Program Evaluation Guidelines – sets out good practice principles for program evaluation. The Toolkit looks at how they can be applied in practice.
Good practice principles
- Evaluations should be built into program design
- Evaluations should be methodologically rigorous, with appropriate scale and design
- Evaluations should be conducted with the right mix of expertise and independence
- Evaluations should be timely to support and influence decision making
- Evaluation processes should be transparent and open to scrutiny
Standards for conducting program evaluations have been developed by different organisations around the world covering quality and ethical evaluation. In Australia the Australasian Evaluation Society has produced Guidelines for the Ethical Conduct of Evaluation
Definitions for program and evaluation
It is important to have a shared understanding of what we mean by 'program' and 'evaluation'. Evaluation is a new field that continues to evolve, so that many concepts and terms are used in different ways. This Toolkit reflects the approach set out in the NSW Guidelines.
Rigour, utility, feasibility and ethics in program evaluation
In line with the good practice principles, this Toolkit looks at how the design of a program evaluation balances rigour, utility, feasibility and ethical safeguards:
- 'Rigour' in evaluation refers to the quality of the evidence, and the validity and certainty around the findings. For outcome evaluations in particular, rigour includes assessing the extent that observed outcomes were due to the program.
- 'Utility' refers to the scope for evaluation users to actually use the findings, particularly when information is needed at a certain times to inform decisions.
- 'Feasibility' refers to the practicalities of collecting evidence in relation to the maturity of the program, and to the availability of time, skills and relevant data.
- 'Ethics' refers to reducing the risk of harm from the evaluation and also doing the evaluation in a culturally appropriate way.
Related activities to collect evidence
Evaluation is part of a spectrum of other activities used to collect evidence and assess programs. Activities which are not program evaluation can support it and produce valuable information in their own right. They include:
- Program reviews – typically quicker, more operational assessments of "how we are going" often to inform continuous improvement.
- Monitoring – a management process to periodically report against planned program targets or KPIs, usually focussed on program outputs.
- Research – closely related to evaluation, but can ask different types of questions that may not be related to the merit or worth of a program.
Other concepts and definitions
Documents about evaluation may include common terms about program evaluation, which are sometimes used in different ways, such as evaluation framework and evaluation plan. The following definitions are how the terms are used in this Toolkit.
- Evaluation guidelines or evaluation plan. For a program evaluation, this is a template and guide for evaluation plans that can be used for related evaluations. For a large program, it may set out the overall evaluation purpose, main evaluation projects, related activities and reporting over the period of the program. The NSW Government Program Evaluation Guidelines refers to program evaluation plans that may be developed during the program design phase, covering the purpose, key questions, primary audience, resources, who will conduct the evaluation, baseline data and methodology, budget and timeline, and plans to disseminate and/or publish findings. The details and scope will differ from evaluation to evaluation.
- Evaluation strategy. For an organisation, this refers to the structures and processes to achieve overall intended outcomes from evaluation. For a large program the term may also be used in the same sense as evaluation plan above, where it refers to the overall evaluation purposes, the main evaluation projects and reports over the period of the program, and related activities such as developing data and building capacity for evaluation.
- Evaluation design. For a program evaluation project, the design outlines the research methods that will provide the evidence to achieve the objectives of the evaluation. The evaluation design should match the scale of the program and the significance of the evaluation; and be as rigorous as possible while meeting needs for project utility, feasibility and ethics.
- Evaluation project plan or workplan. This is the workplan to implement a program evaluation project, putting the evaluation design into practice. Like any project plan, a workplan sets out the tasks, responsibilities, reporting requirements, team, governance, resources, risk management and timing for the evaluation.