Job Network evaluation - Stage two progress report
Executive summary
Background
In the 1996-97 Budget, the Federal Government announced a new framework for the delivery of labour market assistance. The new system was introduced on 1 May 1998 and represented a move to a more flexible system of delivering employment assistance. The Government’s four key objectives in reforming employment assistance were to:
- Deliver a better quality of assistance to unemployed people, leading to better and more sustainable outcomes.
- Target assistance based on need and capacity to benefit.
- Address the structural weakness and inefficiencies inherent in previous arrangements for labour market assistance, and to put into effect the lessons learnt from international and Australian experience of labour market assistance.
- Achieve better value for money.
More than 300 government, community and private organisations were contracted to deliver employment services. Centrelink was established as the gateway to Job Network with responsibility for registering and classifying job seekers for Job Network services. The first contract period for Job Network ended in February 2000, at which time the Government introduced changes to improve services under the second employment services contract (ESC 2) for the period 28 February 2000 to 2 March 2003.
Scope of stage two of the evaluation
The Evaluation Strategy for Job Network, as endorsed by the Government in December 1996, includes three evaluation reports (DEETYA 1998). The first report (which was released in May 2000) covered the first 17 months of Job Network and was aimed primarily at assessing the implementation of Job Network together with early market experiences. The second (the current report) is a report on progress and covers the operation of Job Network over the whole of the first contract period (May 1998 to end-February 2000) and the early months of the second contract period (generally to September 2000). The third report, on the effectiveness of Job Network in securing sustainable employment for job seekers, is due to be finalised at the end of 2001.
The current report focuses on four key areas of interest with respect to the operation of Job Network:
- The development of Job Network.
- Access to assistance for different job seeker groups.
- The performance of Job Network in achieving outcomes for clients.
- The regional performance of Job Network.
The report should be read in conjunction with Job Network evaluation stage one which included a description of the services available under Job Network and the rationale for the changes to employment services.
Main findings
Stage two of the evaluation found that the overall performance of Job Network is encouraging. Preliminary evidence suggests that Job Network is having a positive impact on the post-assistance employment prospects of those assisted. Job Network is delivering better value for money than the previous labour market assistance arrangements. Specifically:
- Job Network has made further progress towards a competitive market by expanding the geographic coverage and competitive basis of its services. The number of sites increased by 45% overall, doubling outside capital cities. One hundred and ten sites deliver specialist services, around one-third of them servicing indigenous job seekers. Competition was introduced for the first time for Intensive Assistance tenders, protected by a minimum set price, and explicit bids for specialist services were available to provide greater choice in the provision of Intensive Assistance tailored to particular job seeker groups.
- Client flows into Job Network services have been significant, with more than one million job seekers referred to Intensive Assistance. Over 670 000 Job Matching placements have been effected over the period of the market examined in the report (May 1998 to September 2000). Participation rates for most job seekers are close to expected, in terms of their share of assistance compared to their share of the Job Network-eligible population, and their participation relative to the rate of participation in Job Network for all job seekers.
- Post-assistance outcome levels for Job Network services compare well with those of previous labour market programs. Three months after a Job Matching placement, 73% of job seekers were either employed (70%) or undertaking education and training, while three months after leaving Job Search Training or Intensive Assistance, the outcome rates for job seekers were 47% (38% employed) and 42% (35% employed) respectively. Outcomes for NEIS were 84%, with 81% employed.
- Preliminary estimates of the contribution that Job Network makes to post-assistance income-support status (its effectiveness) suggest a substantial net impact of about 10 percentage points for Intensive Assistance and a more moderate impact of three percentage points for Job Search Training. These net impact levels are similar to the average net impact levels of those programs operating in the mid 1990s that were replaced by Job Search Training and Intensive Assistance.
- Job Network is achieving outcomes and net impact at a substantially lower cost than previous labour market assistance arrangements, thus providing better value for money. Data on efficiency (unit cost and cost-per-outcome) indicate that the cost of assistance under Job Network is well below that of the previous regime and, across all programs, is less than the unit cost of assistance provided during the 1990s:
- Job Search Training is delivered at about half the cost-per-job of Job Clubs.
- The cost-per-job for Intensive Assistance is more than $5000 lower than the average of the programs which it replaced.
The evaluation, however, has identified some areas where Job Network’s operation could be improved. Not all job seekers are benefiting equitably from Job Network assistance and there is considerable variation in employment outcome rates between job seeker groups. Groups with consistently low outcome rates compared to others include older job seekers (aged 55-64), those on unemployment allowances for more than two years, job seekers with less than year 10 education, indigenous job seekers and those with a disability.
Detailed findings
The development of Job Network
Since implementation in May 1998, Job Network has made considerable progress towards a competitive market. The second tender increased Job Network’s geographic site coverage by 54%. Around 200 organisations are now contracted to provide employment assistance services from 2010 sites. The number of sites outside capital cities has almost doubled (from about 600 to 1100) and more than 250 localities (half of which are in regional and rural areas) have an employment service where none previously existed. Specialist services are offered in 110 sites, around one-third of them servicing indigenous job seekers.
A key objective of the second tender was to lift the performance of the market by retaining good performers from the first contract and supplementing those with new market entrants that had the potential to perform well. About 87% of the organisations contracted in the first tender round were contracted for the second. Increased competition and basing contracts on demonstrated performance have led to a significant shift in the composition of organisations delivering services. There was a considerable increase in the market share of community-based and charitable organisations and private sector organisations.
Competition was also strengthened through the introduction of price competition for Intensive Assistance bids, with quality protected by a set minimum total price and stronger accountability measures to improve the monitoring of service. Competitive tendering was also introduced for specialist services to provide greater choice in the provision of Intensive Assistance tailored to particular job seeker groups.
The transition period to the second contract appears to have had a negative impact on Job Network operations. The tender process was resource intensive, temporarily disrupted placement and referral activity and imposed an increased workload on Centrelink. While open tenders have advantages, these adverse implications raise questions about the appropriateness in the future of the use of a full tender as the best means of purchasing employment services from Job Network providers.
Market share
The evaluation report presents several measures to indicate the extent of market penetration of Job Network in accessing available vacancies (necessary to maximise the potential for outcomes for job seekers). Indicators are encouraging, although more informed judgements about Job Network’s share of the market are not possible until time series data are available on a range of measures including:
- The share of vacancies lodged with Australian Job Search (AJS) compared to total job vacancies lodged as measured by the Australian Bureau of Statistics (AJS averaged 46% per month of total vacancies for 1999-00).
- AJS Internet vacancies as a proportion of total Internet vacancies (AJS represents 36% of the advertisements covered by the ANZ internet survey).
- The proportion of employers using Job Network (38% of employers recruiting in the year to June 1999).
- Job Network placement activity compared to total placement activity by employment placement businesses, as measured by the ABS (at 30 June 1999, Job Network comprised only 16% of employment placement businesses, but placed 64% of permanent employment placements and 10% of all placements in the year to end June 1999).
Trends in client flows
Client flows into the main Job Network services have been significant. Between May 1998 and September 2000:
- There have been a total of 676 600 Job Matching placements, of which 20% went to job seekers who were only eligible for Job Matching services. The ‘Job Matching only’ service was introduced in August 1998 and the proportion of ‘Job Matching only’ job seekers (31% in September 2000) has increased over time.
- More than 425 800 job seekers were referred to Job Search Training, an average of 23 100 per month for the six months to September 2000. The overall take-up rate (the number of clients commencing as a proportion of those referred) was 32%. This rate has declined over time, and stood at 22% in September 2000. Preliminary research shows that the main reason for job seekers not taking up Job Search Training was that they no longer met the requirements for continued income support-this includes job seekers who obtained employment between being referred and commencing in assistance.
- More than one million job seekers have been referred to Intensive Assistance. Of these, 66% commenced in assistance. The take-up rate for Intensive Assistance has also declined over time. In Intensive Assistance, however, the take-up rate appears to be influenced by the availability of other labour market assistance options such as the Community Support Program, Community Development Employment Projects for indigenous job seekers, the Department of Family and Community Services’ funded disability employment services and other Mutual Obligation activities (such as Work for the Dole).
- More than 14 600 job seekers commenced the New Enterprise Incentive Scheme (NEIS). Commencement numbers averaged 6600 during 1999-00, compared to the annual average of 6800 over the period 1996-97 to 1998-99.
The volume of clients becoming eligible for assistance and the take-up rate (commencement to referral ratios) will influence the extent to which Job Network can meet placement and commencement numbers set for the second contract period. Recent data indicate that activity levels in Job Network may be lower than estimated. The extent to which this reflects stronger than anticipated labour market conditions in 1999-00 underlies the need for providers to be sufficiently flexible to be able to respond to changing market conditions.
Access to assistance
Participation rates for most job seekers are close to expected. Some job seeker groups, however, have participation rates either below their representation in the Job Network-eligible population or below their predicted participation rates. These groups include indigenous job seekers (whose participation rates in Intensive Assistance are lower than those for any other group of disadvantaged job seekers), youth, sole parents and job seekers in receipt of income support other than Newstart Allowance. Participation in Job Network is voluntary for job seekers not on income support (which includes many youth) and for those in receipt of non-activity-tested allowances (such as sole parents).
Indigenous job seekers
A survey of indigenous job seekers showed that Centrelink’s role, as the gateway to Job Network is not well understood by many-only 8% of indigenous job seekers perceived that registering with Centrelink was useful in looking for work. Research findings demonstrate the importance of services that are culturally appropriate. Access to indigenous staff or staff who were sensitive and familiar with indigenous communities was important, particularly for job seekers living in remote locations and in communities. Within Centrelink, responsiveness to cultural and personal circumstances and the competence of staff in classifying job seekers and explaining information (eg, using indigenous-specific information sessions) are critical service factors.
There is scope for improvement in indigenous job seeker servicing, especially post-referral follow-up (including those who do not commence) and post-placement support. Building relationships between Job Network and indigenous communities (including local Community Development Employment Projects) would help facilitate greater awareness and understanding of Centrelink’s role and Job Network services.
Performance in achieving outcomes
The performance of Job Network to date has been assessed in this report by examining outcome measures, together with preliminary estimates of effectiveness and efficiency. A more comprehensive analysis will be available for the third stage of the evaluation.
Post-assistance outcomes
Preliminary post-assistance outcome levels for Job Network services compare well with those of previous labour market programs. Three months after a Job Matching placement (between March and June 2000), 73% of job seekers had achieved a ‘positive’ outcome, ie, were either employed (70%) or in education and training. For Job Search Training and Intensive Assistance, post-assistance positive outcome rates for those who left assistance between May 1998 and June 2000 were 47% (38% employed) and 42% (35% employed) respectively. The positive outcome rate for NEIS in this period was 84%, with 81% employed.
A comparison of shares of job seekers leaving assistance with shares of outcomes shows that these shares are broadly similar for most job seeker groups. There is considerable variation, however, in the outcome levels for different job seeker groups and some groups have consistently lower outcomes than other job seekers across all services. These include older job seekers (aged 55-64), those on unemployment allowances for more than two years, job seekers with less than year 10 education, indigenous job seekers and those with a disability.
Effectiveness
Effectiveness is measured by assessing what extra outcomes were achieved from participating in assistance, compared to the experience of a comparison group of similar job seekers who were not involved in assistance. The difference in outcomes achieved provides a measure of the net impact of assistance. Preliminary estimates of net impact for Job Search Training and Intensive Assistance have indicated that Job Network is having a positive impact on the post-assistance employment prospects of those assisted. These are comparable to levels experienced under employment programs operating prior to Job Network (under the banner of Working Nation).
- Off-benefit outcomes (the proportion of job seekers moving off unemployment allowances) for Intensive Assistance participants averaged over 31%, compared with a little over 21% for the comparison group-a net impact of around 10 percentage points.
- Job Search Training participants achieved an off-benefit outcome rate of just under 27% compared to under 24% for the comparison group-a net impact of about three percentage points.
- The average income-support net impact of the labour market programs replaced by Intensive Assistance was about 10 percentage points in 1996 and the income-support net impact of Job Clubs (that offered a similar type of assistance to Job Search Training) was estimated to be four percentage points in 1996.
The report examined a number of factors that may contribute to net impact-an analysis of the types of services offered by high-performing and low-performing providers and an analysis of activities and attitudes among Intensive Assistance participants, relative to their length of time in assistance.
Successful Job Search Training providers were more likely to provide assistance in preparing for interviews and to send job seekers to interviews to speak to employers than other providers. High-performing providers of Intensive Assistance were more likely to use training in job search skills than training in job specific skills. Again, the focus on interviews was stronger and associated with greater outcomes. High-performing providers also tended to have more success in improving job seekers’ self-confidence.
Preliminary research into the variation in outcome levels for Intensive Assistance participants examined the impact of efforts to find work, attitudes to looking for work and to Intensive Assistance, and the frequency of contact between job seekers and their employment officer. This analysis indicated that optimism, motivation, job selectivity, attitudes to work and the extent of pro-active job search all contribute to differential outcome levels between job seeker groups. The distribution of outcomes by time in assistance and the change over time in the pattern of job search activities by Intensive Assistance participants suggests there is the potential for some disadvantaged job seekers to increase their job search activities and that some Job Network members could focus additional efforts in the second half of Intensive Assistance.
Efficiency
The performance of Job Network may also be assessed in terms of its efficiency with regard to the cost of assistance. The cost-per-participant (unit cost) was $200, $425 and $2260 for Job Matching, Job Search Training and Intensive Assistance respectively. Costs-per-employment outcome are $290 for Job Matching, $1130 for Job Search Training and $6200 for Intensive Assistance. Comparing the estimates for Job Search Training and Intensive Assistance to costs applying under the programs replaced by Job Network shows that:
- Job Search Training is delivered at about half the cost-per-job of Job Clubs (which offered a similar type of assistance).
- The cost-per-job for Intensive Assistance is, in real terms, more than $5000 lower than the average of the major labour market programs operating in 1995-96 that it replaced.
Regional performance of Job Network
Performance variation between regions is closely related to the characteristics of job seekers in the local area and the strength of the local labour market. The report found regional variation in the application of the JSCI; coverage of services; participation in Job Network; and in measures of outcomes and effectiveness.
Variation in participation in the three main Job Network services reflects different labour market conditions and differences in the characteristics of job seekers at these locations. Take-up rates are lower in rural and remote areas. These results may reflect the greater difficulty the job seekers in these locations have in accessing Job Network. There are, however, a number of exceptions to the general trend of lower take-up in rural and remote areas that suggest that wider considerations may be at work than issues of access alone.
A preliminary net impact study of Intensive Assistance, to assess regional effectiveness of Job Network, showed some regional variation in effectiveness. For Intensive Assistance, strong labour markets such as Sydney, Brisbane and Perth appeared to do somewhat better than weaker labour markets such as South Australia and Tasmania. These findings contrast with a separate study into the effectiveness of Work for the Dole, which found net impact to be greater in weaker labour markets (DEWR 2000h).
Issues for the future
The overall performance of Job Network since its introduction in May 1998 is encouraging. This evaluation has also raised a number of issues worthy of consideration in the context of its future operation. These include:
- The appropriateness of the use of a full tender (global tendering) in the allocation of future business to Job Network providers. Owing to the uncertainty of prospective business volume and sizeable transition costs in the move to ESC 2, the use of global tendering has had a significant impact on placement and referral performance in the early months of ESC 2. DEWR is undertaking a review of the tendering process to identify the optimal approach to apply in the future.
- The extent to which business levels can be met from the available job seeker pool especially in a climate of good economic and labour market conditions. Trends in inflows and take-up of Job Search Training and Intensive Assistance indicate that there may already be some difficulty in meeting business levels in some areas. Economic conditions and take-up rates are not the only factors influencing whether business levels are maintained. Provider performance, eligibility changes and provider capacity are also relevant.
- The possibilities for increasing the effectiveness of Intensive Assistance. The pattern of outcomes and activity levels for those who remain in assistance longer-who tend to be the more disadvantaged job seekers-suggest that there may be scope to adopt strategies to increase net impact by increasing outcome levels for these disadvantaged job seekers. Further research is required into the distribution of outcomes and its implications for Intensive Assistance.
- The effectiveness of Intensive Assistance in stronger labour markets. Preliminary evidence suggests that effectiveness is somewhat greater in stronger labour markets. This prompts questions about the relative effectiveness of Intensive Assistance under more depressed economic conditions (eg, if there were an economic downturn). This, combined with the more general finding that local labour market conditions have a significant influence on a number of Job Network performance measures (including take-up rates), may have implications for the operation of Job Network where job seekers do not have access to strong labour markets.
February 2001