Job Network Evaluation Stage three effectiveness report
Executive summary
Job Network was implemented in May 1998 to deliver employment services to job seekers and employers. It is a national network of about 200 private, community and government organisations. Job Network replaced publicly funded job brokerage delivered by the Commonwealth Employment Service (CES), a range of labour market programmes which delivered short-term training, wage subsidies and work experience, and a case management system.
The Job Network model represents a radical change in the way employment services are delivered, with the role of government changing from a direct provider to a purchaser and regulator of services. The model was underpinned by a focus on outcomes. While Australia is not alone among OECD (Organisation for Economic Co-operation and Development) countries in introducing market-type mechanisms in employment servicing, Job Network is the most comprehensive example of this approach.
The objectives of Job Network are to:
-
deliver a better quality of assistance to unemployed people, leading to better and more sustainable employment outcomes;
-
target assistance to job seekers who need it and who can best benefit from it;
-
address the structural weaknesses and inefficiencies inherent in previous arrangements for labour market assistance, and put into effect the lessons learnt from international and Australian experience; and
-
achieve better value for money (especially in a tight budgetary environment).
Evaluating Job Network
At the time the changes were announced there was considerable public interest in the process for reviewing such a radical shift in delivery arrangements. A comprehensive Evaluation Strategy was developed following the Budget announcement in 1996 and released publicly in April 1998 (DEETYA 1998a). The evaluation’s purpose was to assess how well Job Network was working and to provide information for later policy adjustment. The strategy involved three reporting stages, recognising that the reforms would take several years to develop and mature.
The first two stages of the evaluation reported in May 2000 and May 2001. These reports were largely based on performance under Job Network’s first contract. Stage 1 focused on the implementation of Job Network and its early operation, while Stage 2 examined equity of access to assistance and outcomes, early indicators of the impact of assistance (on the likelihood of leaving income support) and regional performance.
This final report provides information on the lessons learnt from evaluating Job Network over the period of its operation since May 1998. In assessing effectiveness, the evaluation has examined the sustainability of employment outcomes, the impact of the major services in improving employment prospects, how well Job Network is endorsed by its clients, its responsiveness to special needs and its macro-economic impacts. The evaluation has also considered the relative contribution of factors that can explain the level of effectiveness achieved and how Job Network has progressed against its design principles.
The Evaluation Strategy also included an independent review of the policy framework underpinning Job Network, which is being conducted by the Productivity Commission. A draft report of the Commission’s review was released in March 2002 (Productivity Commission 2002). A final report was produced in September 2002.
Numbers assisted
Each year Job Network helps many job seekers and employers. In 2000-01, more than 860 000 job vacancies were lodged on Australia’s National Vacancy Data Base, which can be accessed through JobSearch. Also in that year, 319 600 job seekers were placed into jobs of at least 15 hours over five consecutive days through Job Matching, 74 800 participated in Job Search Training, 278 600 participated in Intensive Assistance, 6500 were helped to start their own business through the New Enterprise Incentive Scheme (NEIS) and 20 800 were placed into harvesting jobs.
Outcomes and their sustainability
Outcomes achieved from Job Network services three months following assistance (post-assistance outcomes) have generally reflected the level of disadvantage of participants and local labour market conditions (DEWRSB 2001d). In 2000-01, positive outcomes (those either employed or studying) were:
-
71% (66% employed) for Job Matching;
-
52% (43% employed) for Job Search Training; and
-
45% (39% employed) for Intensive Assistance.
NEIS achieved positive outcomes of 85% (83% employed). Outcomes for Intensive Assistance were more likely to be in part-time jobs, while outcomes for Job Search Training were evenly shared between full-time and part-time jobs.
Achieving sustainable outcomes is one of Job Network’s main objectives. The evaluation found that employment outcomes in the short-term are a good indication of employment outcomes in the longer-term. Across the three main services, about four out of five of those in jobs three months after assistance were employed in the longer-term:
-
for Job Matching, 83% of job seekers who were employed three months after placement were employed 12 months later; and
-
for Job Search Training and Intensive Assistance, 82% and 76% of those employed in the short-term were also employed after eight months.
For many job seekers the quality of outcome (in terms of both income and skill levels) improves over time, supporting the view that for some job seekers a low quality initial job can provide a ‘stepping stone’ to a better quality job. It is significant, moreover, that the sustainability of outcomes appears to hold generally for most job seekers, including those who are more disadvantaged.
There are, however, a group who return to income support over time. Among job seekers referred to Intensive Assistance who leave income support, for example, about 30% had subsequently returned to the income support register by 15 months after referral. Of course, such people may regain some employment after this point, but this figure highlights the intractability of the problems confronting certain job seekers in securing a sustained transition from income support to financial independence. This problem of returning to income support is also evident among the long-term unemployed in other countries and was very apparent under labour market programmes that preceded Job Network. Notwithstanding the progress made in recent years, there is considerable scope to improve targeting and design of optimal interventions for job seekers with different characteristics in different labour markets, so they achieve lasting results. This is an important area for ongoing research within Australia, as in other OECD countries.
Impact of assistance on employment
The evaluation presents estimates of the ‘net impact’ of Job Search Training and Intensive Assistance on the employment prospects of participants. Net impact measures the effectiveness of a programme, as it controls for outcomes that would have occurred in the absence of assistance. It does this by comparing the employment rate of job seekers involved in a labour market intervention (a programme group) with those of a group that (as far as possible) has not experienced this intervention (a control group).
Net impact findings need to be understood in the context of what can realistically be expected from labour market interventions. International evidence is that relatively few interventions for disadvantaged job seekers achieve substantial impact. The findings also need to be interpreted very carefully because of the inherent difficulty of forming a control group in the presence of universal entitlement to assistance.
In particular, the new estimates presented in this report are likely to be conservative because it is not possible to develop a pure control group: members of this group may be-and some actually are-referred to assistance, which may in turn affect their outcomes so that the net impact measure of the programme is reduced. Consequently, the new net impact measures do not reflect a pure comparison between an intervention and no intervention; rather, they compare an intervention to a combination of no intervention and other forms of assistance. In addition, the estimates do not measure the improved employment prospects of participants in assistance who have not yet secured a job. Finally, it needs to be remembered that positive net impacts are a static measure of effectiveness. Even if modest, these benefits compound over time for successive groups of participants, producing more significant effects on the level of disadvantaged job seekers when considered over a longer time frame.
Unlike previous net impact studies (reported in DEETYA 1997 and DEWRSB 2001c), the preliminary estimates in this report use a new methodology that attempts to measure the cumulative effect of three elements: compliance, programme and attachment effects. Compliance effects result from referral to a programme and derive from the fact that some job seekers increase job search activity or report existing activities to avoid the requirements of participation in a programme. The programme effect represents the ‘value added’ by the programme, as reflected in changes in the level and effectiveness of job search activity and employability that comes from participation. The reduction in job search activity that results from actual participation in the programme constitutes the attachment effect.
It is now possible to obtain a qualified measure of these effects by estimating net impact from referral to a programme and from commencement in assistance. For the reasons detailed above, these estimates are preliminary, are likely to be understated and need to be interpreted with due caution, especially as they still need to be replicated. Previous employment net impact studies (DEETYA 1997), which measured net impact after participation, did not provide estimates of compliance and attachment effects. The same methodology applied to current employment assistance (Job Search Training and Intensive Assistance) indicates significant net impacts, particularly for disadvantaged job seekers, when compared to the programmes that they replaced. The revised methodology now being adopted, however, offers additional insights that favour its application into the future.
The net impact of Job Search Training and Intensive Assistance, based on the new methodology, was found to be modest, particularly for Intensive Assistance. For many job seekers, the benefits of participation in Intensive Assistance were substantially reduced by the attachment effect which is likely to reflect the extended programme duration and activity levels. This suggests that the current 12 to 15 month period for Intensive Assistance is too long, and that reducing the duration of assistance and intensifying the level of activity during participation, including job search, could improve overall impacts. For both programmes, a considerable part of the net impact from referral came from the effects of compliance. For Job Search Training, however, the evaluation also found that participation in assistance seemed to increase the motivation to look for work and to improve job search techniques.
Disaggregation of the results indicated a degree of variation among client groups. Significantly, the more disadvantaged among those referred to Intensive Assistance appeared to benefit most from the programme (compared to other job seekers). The compliance effect from referral to Intensive Assistance was highest for the mature-aged and those with low levels of education. Job seekers aged 25 and over, and those from a non-English-speaking background, were found to benefit most from Job Search Training. Both Job Search Training and Intensive Assistance, however, were associated with significant levels of deadweight cost (this is the proportion of job seekers who would have got jobs even in the absence of assistance), suggesting the need for even tighter targeting.
Cost-effectiveness
The introduction of Job Network has resulted in a substantial reduction in the cost of achieving employment outcomes. Job Network costs per employment outcome have been the lowest achieved in the past decade: at about $5000-$6000 since mid-1998, compared to between $10 000 and $16 000 in the mid-1990s and $8000-$9000 in the early 1990s.
In 2000-01, costs per employment outcome were $560 for Job Matching, $1390 for Job Search Training and $5440 for Intensive Assistance. These costs for Job Search Training and Intensive Assistance were dramatically below those of the comparable programmes that they replaced-respectively, $2600 for Job Clubs and $12 100 for the group of programmes replaced by Intensive Assistance. These substantial efficiency gains have translated into improved cost-effectiveness.
Raising effectiveness
The net impact findings suggest that there is scope to increase Job Network’s effectiveness. By examining a range of factors that contribute to the performance of Job Network as an employment service-targeting of assistance, Centrelink’s performance of its ‘gateway’ functions, provider behaviour, the fee structure and the quality of services to job seekers and employers-the evaluation canvassed several areas for possible improvement to Job Network.
Targeting
Targeting of assistance has improved significantly with the introduction of an objective screening tool. Effective targeting, however, relies on the accuracy of the Job Seeker Classification Instrument (JSCI) to determine a job seeker’s relative likelihood of remaining unemployed and eligibility for Intensive Assistance. While there is scope to improve this process through better data gathering and adjustments to the instrument’s weights (something which is now in train), the net impact analysis suggests effectiveness could also be increased by raising the threshold at which job seekers become eligible for Intensive Assistance. Many job seekers currently referred to Intensive Assistance, especially those with relatively low JSCI scores and unemployed for shorter periods, may well have secured employment in the absence of assistance. The potential for job seekers to benefit from assistance is also an important issue for targeting. The finding that about 60% of participants leave Intensive Assistance and do not secure a job (in the short term), and that about half of all those commencing Intensive Assistance since July 2000 have already participated in Intensive Assistance, points to the need to revise the assistance regime and review the extent to which people can or should recycle through a programme that has not achieved a positive outcome.
Centrelink’s role
Access to Job Network for most job seekers occurs via Centrelink. Centrelink has the responsibility to explain the Job Network system to job seekers and to facilitate contact between job seekers and the Job Network member (including choice of provider). While Centrelink provides job seekers with information on the services available from Job Network, the evaluation found that many job seekers were unclear about Job Network’s role and the distinction between Job Network and Centrelink. Many job seekers do not appear to fully understand all their rights and responsibilities under Job Network, or how to maximise their chances of finding work. Relatively few providers thought that the job seekers referred to them had a good knowledge of the services they could obtain from Job Network, how Job Network worked or the role expected of them in relation to active job search.
Services to job seekers
Providers can make a difference to the employment prospects of the job seekers they assist. The performance of providers has been found to differ according to both their approach to business and in the level and type of services delivered (DEWRSB 2001f). In respect of services, the evaluation found that high-performing Intensive Assistance providers seem to be more pro-active and more responsive to the needs of both job seekers and employers, including providing a screening function for employers. The more successful providers were more likely to offer training in both job search and job-specific skills and to have more frequent contact with their job seekers. For Job Search Training, high-performing providers were perceived by their clients to be more responsive to job seeker needs. These providers were seen as offering more services and access to facilities, being pro-active in referring job seekers to employers and in keeping them motivated, and also being more likely to submit participation reports to Centrelink when participation fell short of what was required.
Analysis of possible factors that explain better outcomes found that, in general, job seeker characteristics and previous work experience were the most important determinants (rather than the type of assistance). For Intensive Assistance, a positive relationship was found (when other factors were controlled for) between post-assistance employment and participation in voluntary work for both males and females; and in education level and specific job training for males. Previous work experience and helping job seekers to stay motivated to look for work were found to have a positive influence on post-assistance employment for Job Search Training participants.
The evaluation found that 74% of job seekers surveyed in 2001 reported that they were either satisfied or very satisfied with the services. Among the reasons for job seekers’ satisfaction with Job Network were that the services helped them stay connected with the workforce and that they were ‘treated like an individual’. For those dissatisfied, the main reason across all service types was a reported lack of assistance. The quality of services under Job Network reportedly has improved compared with previous arrangements. Job seekers who had experience under both the CES and Job Network reported that Job Network had improved access by having shorter queues and waiting times. The greater incentive to perform provided by the competitive nature of Job Network translated into a higher level of service. From a comparative perspective, the main criticisms voiced by job seekers and providers were that less financial assistance was available for longer and more expensive training of people with more significant employment barriers, for assisting with transport costs, or for purchasing equipment such as tools and clothing.
Job seeker motivations and attitudes influence the level of services. More pro-active clients were found to demand and obtain more services from their providers. In addition, Job Network has successfully equipped many job seekers with skills to search for their own jobs, including improving their self-promotion and preparation of résumés, which improves their employability.
Many job seekers, however-especially those who were unlikely to obtain an outcome-received limited assistance from their provider. The evaluation found that provider behaviour in this regard was consistent with aspects of the incentives in the Intensive Assistance fee structure. Given the risks associated with investing heavily in job seeker support for uncertain outcomes, providers have tended to focus their efforts on those job seekers whose prospects of securing an outcome are strongest, and who do not require a substantial investment to overcome their barriers to employment. This finding favours a revised incentive structure where provider risks are reduced and the returns from investing in outcomes for the very disadvantaged are improved. It should be noted, however, that the incentives in the fee structure are not the only incentives providers respond to. The star ratings (an outcome-based measure of relative performance) and contract renewal also exert a strong influence on the quality of the services that providers deliver.
As mentioned above, there would also appear to be substantial scope for increasing the intensity of Intensive Assistance but reducing its length. Reducing the length, combined with a requirement that job seekers maintain job search activities while in assistance, has the potential to increase effectiveness. It would also reduce the significant attachment effect noted earlier, which is associated with Intensive Assistance in its current form.
Employer servicing
The evaluation found that employers have increased their use of employment agencies to recruit staff, in line with an increase in the use of recruitment methods generally. The proportion of employers using Job Network, however, was unclear. While reported usage of Job Network appears to have declined recently, many employers were not aware that they were dealing with a Job Network member. Those employers who did use Job Network rated the service highly. In 2001, 90% of employers were satisfied or very satisfied with their agency-a substantial increase from 84% in 1999. Employers indicated that the benefits of Job Network included screening and short-listing applicants; interviewing job seekers; and providing employers with access to a large pool of applicants.
Research with Job Network members found that high-performing providers devoted a significant amount of energy to developing strong relationships with selected employers. Providers reported spending a considerable proportion of their time cultivating relationships with employers as a way of maximising placement outcomes, with some seeking exclusive relationships with employers.
The evaluation found that there is scope for improving Job Network’s performance through attracting employers who previously have not used Job Network. For the third Job Network contract, Job Matching services are to be expanded to a wider range of job placement agencies. This should increase Job Network’s market share by attracting more employers who have not previously used it. More extensive and effective marketing by Job Network members to employers (particularly medium and smaller employers) is also warranted. Qualitative evidence indicates that many employers have misconceptions about the services available from Job Network, including that it only caters for unemployed people on income support, or that it mainly deals with those who are perceived as unemployable or who do not want to work.
Progress against design principles
The evaluation also examined Job Network’s progress against the design principles underpinning the 1996 reforms to labour market assistance. These principles, which were based on the lessons of overseas and Australian evaluations, relate to the integration of income support with participation in labour market assistance; competition in service delivery; targeting of services (discussed above); flexibility in the provision of services; a focus on (job) outcomes; and job seeker and employer choice.
Structural integration between income support and gateway access to employment services was achieved with the establishment of Centrelink. The Australians Working Together (AWT) package, which will be progressively implemented from July 2002, is designed to strengthen integration by improving the linkages between different types of assistance. The evaluation has considered integration in terms of stakeholder perceptions of how well Centrelink facilitates access to Job Network and the relationship between Job Network members and Centrelink. The evaluation found that there are still job seekers and employers who remain unclear about all the services that Job Network has to offer. Over 80% of providers reported that they were satisfied with Centrelink’s service. Providers identified areas for improvement as a need for better communication between Centrelink and Job Network and for Centrelink staff to have a better understanding of Job Network.
While open tender rounds disrupt the market and temporarily reduce placement activity, they provide an opportunity to raise market performance overall. Competition in service delivery was found to be widely supported among Job Network stakeholders because it increases choice and focuses activities on performance. The adverse effects of competition, mentioned by some stakeholders, were reserving job vacancies by some providers, inappropriate practices and minimal assistance to some disadvantaged job seekers. Removing limits on provider case loads would strengthen competition, but any move in this direction would have to be implemented carefully to avoid market failure.
Providers have the flexibility to design innovative ways of addressing their client’s employment barriers. There are indications that higher-performing providers, through their recruitment practices, are more likely to be innovative. The evidence at this time, however, is not yet conclusive. Moreover, the evaluation found that while job seekers can choose which provider to go to, the services offered by providers were often regarded by job seekers as similar. This is of concern if the services were ineffective.
A focus on outcomes is reflected in the considerable cost efficiency gains achieved by Job Network compared with previous labour market assistance arrangements. Under the second contract, this focus was strengthened with the awarding of business based on provider performance and the release of increased performance information, including star ratings. The evaluation has explored some of the different ways that providers have responded to Job Network incentives which are designed to focus providers on securing outcomes. Evidence suggested that, as noted above, most stakeholders seemed to support the focus on outcomes. The evidence of job sustainability and progression added further support. Short-term job outcomes, however, may not be suitable for all job seekers, especially those who require a substantial investment to overcome their barriers to employment. The outcomes focus, involving star ratings of relative performance, was also seen by some as contributing to some inappropriate practices in Job Network.
The extent to which job seekers exercise an informed choice when selecting a Job Network provider is unclear. Over 50% of Job Search Training and Intensive Assistance participants surveyed in 2001 reported that they chose their provider. Of those who did choose, most did so on the basis of the provider’s location. The release of the star ratings and streamlining of the referral process, which has recently been piloted, are expected to facilitate more informed choice. The operation of choice, however, is constrained by current limits on providers’ contracted capacity and the lack of incentive this creates for providers to market their services actively. The evaluation suggests consideration be given to a gradual relaxation on these limits.
Ongoing policy adjustments and further research
The performance of Job Network clearly demonstrates that substantial private and community sector involvement in the delivery of employment services is viable. Much has been achieved in the relatively short life of the new arrangements, which have proved to be significantly more cost-effective than those applied previously.
Job Network, however, necessarily remains a ‘work in progress’ which needs to evolve further as lessons are learnt and more information on what supports high performance becomes available. As part of this process of improvement, the evaluation has explored ways of further raising the effectiveness of Job Network. There is considerable scope for incremental policy adjustments to build on the successful performance to date, especially for the most disadvantaged job seekers.
The evaluation has also highlighted issues where further research is warranted. These include the following areas.
-
The lack of a pure control group and the need for replication of the findings confound to some extent the new net impact measures developed for this evaluation. Further investigation is also needed to measure the level of any bias in the control group and the extent to which alternative approaches can control this bias. The current net impact study only measures job outcomes 12 months after referral to and commencement in a programme. While this approach potentially represents a considerable improvement over previous methods, the current study could be supplemented by estimates of the sustainability of net impact, improved job prospects in the absence of having a job and changes over time in labour force behaviour relative to labour market interventions. Insights from such research will be important in achieving further reductions in the incidence of churning.
-
The evaluation suggests that the current approach to targeting, which largely relies on the JSCI, could be refined. The JSCI is important in determining the level of disadvantage that job seekers face, but is not the optimal means of determining the most appropriate form of assistance to address this disadvantage. The report discusses an approach to targeting that is based on the likelihood of success from existing assistance options. This approach has the potential to increase the effectiveness of current types of assistance as well as the extent to which any new forms of assistance are needed.
-
The segmentation approach to job seekers’ attitudes could make possible tighter targeting of different types of assistance. Informed by job seeker attitudes, providers could better allocate additional assistance to the job seeker on the basis of the available attitudinal information. The job seeker may then be perceived to have a greater capacity to benefit. More research, however, is needed to investigate the quantum and stability of segmentation, its relationship to outcomes and whether it has application within a system of targeted interventions. Research is also needed into the effect of job seekers’ attitudes on the net impact of assistance.
-
The evaluation has identified some servicing issues where further research is warranted:
-
specialist providers were introduced into Job Network for the second contract. While they seem to have improved the quality of service, their impact on effectiveness has yet to be established;
-
consideration could be given to greater tailoring of assistance in Job Search Training. Currently most Job Search Training courses are generic. Using different training packages for job seekers whose circumstances are different may offer scope to further increase the effectiveness of Job Search Training; and
-
the performance of Job Network for job seekers from a non-English-speaking background was examined in the evaluation. Within this diverse group, however, more understanding is required of how well Job Network meets the needs of job seekers with different cultural backgrounds.