« Previous | Contents |
Listen
The Supporting People (Housing Support) Outcomes Framework, which gathers outcomes for housing support services, previously funded through Supporting People grant, through capturing the distance travelled by an individual service user, was piloted in seven local authorities in 2007. The evaluation of the pilot sought to assess whether the Outcomes model could be introduced consistently at local authority level to capture useful information, which in turn could be used as a means of public and ministerial reporting. The research has been undertaken by Craigforth.
Main Findings
- Since the pilot of the Supporting People ( SP) Outcomes Framework was launched in 2007, there have been important changes in the national context. Ring fencing of Supporting People funding has been removed and the Scottish Government has committed to reducing the burden of reporting on local authorities. The SP Outcomes Framework does however have considerable resonance with the National Performance Framework for public services since it offers a model of measuring outcomes.
- Local authorities and service providers participating in the pilot expressed strong support for the Outcome Framework, particularly its focus of measuring outcomes through distance travelled. This support was evident both at the outset and following their experiences in the pilot.
- Overall the evaluation found a lack of accuracy and consistency both in the way in which the Outcomes Framework was interpreted and also in the processes adopted by different providers in using the Framework. Greater accuracy and consistency was achieved by individual providers, in particular where either a single individual undertook all the assessments and reviews, the provider had introduced a system of checking, or the local authority provided active support and co-ordination.
- There were difficulties with the counterfactual - the term used to refer to the assessment of the position of the service user if the service was not being provided: this sought to measure the preventative benefits of housing support services. While pilot participants recognised the value of measuring the preventative benefits, the counterfactual proved difficult to implement, in part because it caused distress to some service users who did not want to consider a situation without support, and in part because providers felt that they lacked the necessary skills to identify what the pathway might have been in the event of no service being in place.
- Service users in the main valued the process of assessment and review which formed part of the Framework. This applied in particular to younger people (16-24 year olds) and those whose circumstances changed relatively quickly, where 'progress' was more evident. Others valued the opportunity to discuss their needs and their services with their provider.
- The overall conclusion was that the Framework was still useful, even with the changed circumstances surrounding funding for housing support services. While ownership of the framework best rested at the local level, the Scottish Government had an important role to play in promoting the tool, including developing an improved IT platform and better ways of recording the preventative benefits of housing support, along with better guidance and training on the tool's use.
Background
Since the evaluation of the Framework was commissioned, the Scottish Government has shifted the emphasis further towards focusing public services on outcomes and has also committed to simplifying and reducing the level of scrutiny of local government. As part of a package of measures, ring fencing was removed from certain budgets in April 2008, including the Supporting People budget. This is likely to lead to housing support becoming more integrated within a wider spectrum of care and support services. These factors may lead to an increased rather than reduced need to evidence the outcomes achieved through housing support.
The Supporting People (Housing Support) Outcomes Framework
The Outcomes Framework offers a model of measurement which gathers 'soft' outcomes through capturing the distance travelled by an individual user of services. It is intended to be used as an integral, and ongoing, part of a provider's assessment and review of needs, rather than an external 'evaluation'. The tool was designed to help providers discuss with clients their progress and help providers and those commissioning services evidence and discuss the value of the support offered.
The model Outcomes Framework was adapted from the Housing Support Outcomes Matrix developed by City of Edinburgh Council. It is structured around four categories (accommodation, health, safety and security, social and economic wellbeing). The Framework then identifies 'elements of support' within each of these categories. The needs of service users are assessed on a five point scale in relation to those elements relevant to the individual's circumstances: a 'high' score implies that intervention is needed immediately while a 'low' score implies that the service user has low level service needs and or may be able to exit the service in the near future. The Outcomes Framework includes outline descriptions of the types of circumstances which would be included at each of the five levels of support.
The core task of the provider is to assess, with the participation of the service user, what level of support is required in relation to each element and category of support. Over time it is expected that the position of the service user will change: this will then be reflected in a changed assessment with the difference between assessments providing the 'distance travelled'.
In order to measure the preventative benefits of housing support services, providers are also asked to assess the position in relation to the various elements of support if the service was not in place (the 'counterfactual'): this can be with or without the involvement of the service user.
Research Aims and Objectives
The main aim of the research was ' To assess whether the Outcomes model being piloted could be introduced consistently at local authority level to capture useful information, which in turn could be used as a means of public and ministerial reporting at national level'.
In addition the contractors were tasked with making recommendations about the future implementation of the Framework.
Pilot Participants
Seven local authorities volunteered to participate in the pilot. It is important to note that other local authorities may not be as interested in, or supportive of, the Outcomes Framework.
Providers within these 7 local authorities were then recruited so that the range of different types of housing support services and of different client groups were represented. In total 33 providers were involved to some extent, with 26 completing the pilot.
The pilot took place from June to December 2007, with providers being asked to assess some or all of their service users at the outset of that period, and then reviewing these same service users in October (or earlier if they were leaving the service).
The Evaluation
An evaluation framework was developed which identified the following 4 broad objectives for the Framework, that it
1. Measures outcomes for individual clients
2. Measures impacts of the programme across local authorities
3. Does not place excessive data collection burdens
4. Provides useful information for national public and ministerial reporting.
Within each of these a series of success factors were identified, such as in relation to achieving accuracy and consistency in the data collected; the relevance of the Framework for individual service users, providers and local authorities; and the relationship the Framework could have with other assessment processes. The report is structured around these objectives and success factors.
A combination of quantitative and qualitative research methods were used to evaluate the Framework.
(a) Surveys of providers participating in the pilot at the outset of the pilot and then again at the end of the pilot. Overall, 30 providers completed the initial questionnaire and 17 completed the final questionnaire. Telephone interviews were also undertaken with 29 providers around the mid point of the pilot.
(b) Fourteen of the providers were visited during the pilot, two from each of the 7 local authorities. During these visits, assessment interviews were observed, staff from the service providers and service users were interviewed. Overall, 33 service users and 35 staff were interviewed during the site visits.
(c) Visits were made to the 7 pilot local authorities to interview Supporting People staff.
(d) Feedback was also provided on an ad hoc basis by participants in the pilot where they had encountered particular issues or difficulties: 9 responses were received.
The intention had also been that the final dataset of cases assessed during the pilot would be reviewed as part of the evaluation to assess both the quality and coverage of the data (including cross-referencing to the evidence from the site visits) and also the extent to which it was suitable and relevant for national and ministerial reporting. Our ability to be able to use this data for the latter objective was however severely constrained by issues relating to the quality of the data.
Measuring Outcomes for Individual Clients
Providers themselves were concerned about the accuracy with which they, or their staff, were completing the Framework. In part this related to particular sections of the Framework where differing interpretations were possible, and to various technical difficulties with the software package. There were however also more complex issues, which impacted on accuracy. For example, a service user may be reluctant to reveal the full nature of their problems at an initial meeting with a provider. The 'accuracy' of their situation may therefore not be recorded until they are more settled and have confidence in the provider.
Overall within the services observed during the site visits, there was a high degree of consistency in completing the Framework by different support workers within the same service. There was however significant variation between services in the processes used to complete the Framework: this variation applied irrespective of whether services were catering for the same client group or were providing similar types of housing support service.
The variation related to the number of staff involved in the assessment process; whether the assessments were completed on a paper copy of input directly onto a PC; and the location of the assessment interviews, which took place in a mixture of home and office settings.
Confusion amongst providers over some aspects of the Framework, such as whether or not they were supposed to complete particular sections, also resulted in inconsistencies in approach.
The 'counterfactual' proved problematic, causing distress to some service users who did not want to consider a situation without support, and to some providers who sometimes felt that they lacked the skills to know what the individual's needs would have been without support.
Not all service providers involved service users in the assessment, although the proportion increased from the initial baseline assessment in July to the review assessment at the end of the pilot.
Service users were mostly happy to be involved in the completion of the Framework. For some, particularly younger people (aged 16-24 years), it provided useful feedback about the progress or lack of progress which they had achieved. Others were simply happy to be helpful to the service provider, or considered that it provided a good opportunity to spend time with their support worker.
There were differing views about the extent to which the Framework was suitable in its current form for all client groups. Some of the concerns related to the current structure and terminology within the Framework, with a view from providers that it needed to be tailored more to the experiences and circumstances of particular groups.
The pilot involved a review 3 months after the initial assessment. Overall, most providers and local authorities considered that the frequency of the review should be longer for those services users whose needs changed more slowly, such as most older people and those with learning difficulties.
Measuring Impacts across each Local Authority
A pre-requisite for being able to measure impacts of any programme of support across the local authority is that there is consistency and accuracy within and across providers in their area when they are using the Framework: this was not however evident during the pilot. Higher levels of accuracy and consistency appeared to be achieved where local authorities had been more active in providing support to providers, suggesting that the Framework had the potential to measure outcomes across the local authority. Local authorities themselves felt that this had been the case in the pilot, but this could not be verified in the evaluation because of the quality issues with the final dataset.
The emphasis on outcomes was welcomed by the participating local authorities, although there was also some uncertainty about the implications of the changes at national level (the commitment from the Scottish Government to reduce the reporting burden on local authorities and the removal of ring fencing) for data collection and reporting of housing support services.
Local authorities saw that the Framework could deliver information on the impact of their investment, provide input into service reviews and inform strategic planning. A number also saw potential for integration with other assessment processes (such as the Single Shared Assessment) and other outcomes indicators (such as the Community Care outcomes).
Data Collection Issues
The evaluation sought to establish whether the data collection burden involved in the Framework was reasonable.
On average providers reported that by the end of the pilot it took an average of 20 minutes to complete the Framework, although the time varied for individual providers from five minutes to more than an hour. Longer interviews tended to be associated with particular clients, who required more time to engage with the process. For most providers the time required reduced during the course of the pilot as they became more familiar with the Framework.
In the main, local authorities considered that the Framework provided additional information and would not replace existing information requests made to providers. Provider views as to whether the additional resourcing requirements of the Framework would be acceptable were affected by whether they thought it could be integrated into other assessments or management tools.
The resource implications of the Framework were affected by difficulties with the software, increasing the time involved for participants. These difficulties also affected the quality of the data collected.
Information for National Public & Ministerial Reporting
No conclusion could be drawn as to whether the Framework would be capable of delivering relevant and useful information for national public and ministerial reporting. This was mainly because of the level of inaccuracies and inconsistencies within the data collected. The evidence did suggest that more intensive management, improved guidance and greater support would substantially reduce this difficulty.
However in the context of the Concordat, particularly the intention to reduce the reporting burden on local authorities and the removal of ring fencing from the Supporting People programme, national reporting at the level of detail of the Framework may no longer be appropriate.
There was recognition from some of the local authorities that the Framework offers the potential for use more widely than for housing support services only: this might include certain care services. This would also fit with the reality that service users are often accessing a range of services simultaneously. Other developments are taking place in developing tools which can capture the views and experiences of service users to inform service evaluation and development, such as the User Defined Service Evaluation Tool ( UDSET): the experiences of piloting the Supporting People Framework have the potential to inform these other tools.
Conclusions
The Supporting People Outcomes Framework remains relevant, in spite of the removal of ring fencing, as a tool which has the potential to measure outcomes at individual service user level. Its focus on low level, preventative type services is of particular relevance, given the current policy interest in preventing or delaying the need for more intensive services.
There was strong support for the Outcomes Framework from local authorities and providers participating in the pilot, both at the outset and after the experiences of the pilot itself. Both local authorities and providers had many suggestions for improvements which could be made to the Framework.
The lead for introducing the Framework in any particular provider could be taken by the local authority or by the individual provider. Greater consistency will be achievable where the local authority promotes and supports the use of the Framework: however it is unlikely that all local authorities will choose to introduce it. The evidence from the evaluation is that individual providers may see benefits in using the Framework, even where not expected to do so by the local authority, as a means of demonstrating the contribution of their service. Several providers indicated that they would want to continue using the Framework after the pilot was completed.
Improvements to the Framework are required so as to reduce ambiguities and areas of confusion: the evidence from the evaluation was that more explicit guidance and more hands on support and management would substantially increase the level of consistency and accuracy. There is also a need to consider alternative approaches to measuring the preventative aspects of services (the 'counterfactual').
This document, along with "Evaluation of Supporting People (Housing Support) Outcomes Framework" the full research report of the project and further information about social and policy research commissioned and published on behalf of the Scottish Government, can viewed on the Social Research website at: www.scotland.gov.uk/socialresearch. If you have any further queries about social research, please contact us at socialresearch@scotland.gsi.gov.uk or telephone 0131 244 7573.
« Previous | Contents |