1. Summary

Ofgem has adapted its analysis of operating costs to take some account of criticisms that we raised in our previous paper.

However, we are still concerned that the analysis does not arrive at an appropriate efficiency judgement. In particular, it:

• fails to find a way of justifying the extent to which differences between the estimated equation and an individual company’s position should be deemed to be a sign of efficiency as opposed to merely the error present in any statistical process;
• retains bias from the mis-specification of the basic equations and the definition of total costs, the omission of explanatory variables and the choice of a linear functional form;
• makes comparatively small adjustments when considering the impact of capital costs and mergers and does not consider their cumulative impact;
• assumes that the electricity distribution industry should be able to move rapidly to its estimated quartile point without justifying the speed of adjustment or considering the implications for average industry productivity growth rate.

There is no certain way of deciding the extent to which it is reasonable to assign the equation errors to efficiency rather than to genuine error. The judgement that Ofgem has made is equivalent to adopting an optimistic assumption about the proportion of true error and then, even if that was the case, insisting that companies can move to a point that one cannot be confident is not at a level that few companies could be expected to achieve.

Errors in the base equation are likely to be all the greater because it fails to take account of capital costs, which affect the level of operating cost efficiency, or merger savings. The equation that purports to make an adjustment for capital costs does not do so but adjusts for capital spending. This is unjustified in economic theory and, since a relatively low stock of capital assets may be associated with a greater need for capital expenditure, might even increase the original mis-specification. We demonstrate the impact of correct specifications

The adjustments for mergers and capital costs are given little weight in the final outcome and are made separately, whereas the cumulative impact (which we show) is required to make the correction. The small adjustment for mergers is particularly unfortunate. Our results suggest different, and less dramatic, adjustments. The alternative method of data envelopment analysis also suggests that the size of Ofgem's efficiency judgements is exaggerated.

The aim of benchmarking operating costs is to forecast the costs of an averagely efficient company. The averagely efficient company should earn the cost of capital; more efficient firms should earn more; and less efficient firms less. It is easier to project average costs on the basis of the average cost than on a frontier or quartile cost because there is uncertainty as to where the frontier is and how quickly it is reasonable to assume that companies will reach it. Ofgem's assumption that it will be reached in 2004-05 requires an industry average efficiency growth of 5.25% a year to have been taking place over the past two years which, even accepting Ofgem's claim that the normal rate is 2%, is unreasonable.

In short Ofgem should:

• Accept the uncertainty of the process it has adopted and make an assessment of each company's efficiency relative to the average that is less than any equation error;
• Correct mis-specifications, particularly that of "total costs";
• Allow fully for total costs and mergers rather than making minor adjustments;
• Project average costs, rather than assuming that there has already been a period of abnormally high efficiency growth to reach a notional frontier.

2. Ofgem's initial proposals

Ofgem has published its latest view on normalised operating costs and total fault costs for all DNOs, having made adjustments for a number of factors it considers make the costs more comparable across the industry.

Top-down benchmarking has then been performed using these normalised costs. Regression results have been on a composite scale variable (CSV) cost driver. This attributes 50% weight to network length, and 25% equally to customer numbers and units distributed. Figure 1 shows the basic regression produced by Ofgem in the initial proposals.

1. Baseline regression normalised opex plus faults 2002/03

Although Ofgem has retained an only slightly altered form of the equation, which we criticised in our previous paper, it has modified its approach in three respects in that it has:

• set the frontier target at an upper quartile, defined as the mid-point of the third and fourth company, rather than at the lowest observation;
• estimated a further equation adding average capital spending in DPCRs 3 & 4 to the opex figure in an attempt to proxy total costs;
• estimated a third equation on data for the 9 company groups holding DNO licences to attempt to adjust for savings obtained in mergers.

Starting DPCR 4 operating cost allowances have been set at the higher of the upper quartile point of the base analysis and the average of the similar points in the base, "total cost" and 9 corporate group regressions.

However, we are still concerned that the analysis does not arrive at an appropriate efficiency judgement. In particular, it:

• fails to find a way of justifying the extent to which differences between the estimated equation and an individual company’s position should be deemed to be a sign of efficiency as opposed to merely the error present in any statistical process;
• retains bias from the mis-specification of the basic equations and the definition of total costs, the omission of explanatory variables and the choice of a linear functional form;
• makes comparatively small adjustments when considering the impact of capital costs and mergers and does not consider their cumulative impact;
• assumes that the electricity distribution industry should be able to move rapidly to its estimated quartile point without justifying the speed of adjustment or considering the implications for average industry productivity growth rate.

The next four sections of this paper consider each of these points in turn.

2. Uncertainty and frontiers

There are many sources of uncertainty in top-down benchmarking comparisons.

The base data are themselves uncertain, vary from year to year, stem from different accounting practices and are, in most cases, the result of allocations and internal transfers between entities within larger companies.

Ofgem has made adjustments to the data to attempt to correct biases that may be produced by systematic differences in, say, accounting policy but the adjustments will themselves involve uncertainty. In some cases the adjustment is in respect of a cost driver where Ofgem is, rather than estimating a general equation using all cost drivers, claiming prior knowledge of the impact of some cost drivers and confining the regression to the remaining drivers. In reality, the adjustment is uncertain and, had a general regression been conducted, the uncertainty would have been indicated by the standard error on that driver’s coefficient.

The decision to make no adjustments in respect of mergers is itself one to which uncertainty attaches.

The choice of the form of benchmarking from the many possibilities available makes the comparison uncertain. The result would have been different had different choices been made in:

• The dependent variable, in particular the treatment of capital costs
• The mathematical form of the equation – linear, log linear, quadratic etc.
• The treatment of cost drivers: there are probably hundreds (certainly tens) that are possible in theory, may have been advanced by DNOs and do affect costs to some extent; the more important include the weights on the three scale drivers and the omission of local wages, population density, voltage and customer mix, losses, quality of supply and merger dummies.

Finally, even if there had been no choices made in treatment of data and in the functional form, the results would be uncertain. In most regressions the uncertainty is revealed in the equation statistics, which are functions of the errors of the estimates, the difference between the equation estimate and the observed value. However, in this approach, that difference is taken as a measure of the impact of another explanatory variable, relative efficiency, which is not directly observable and so is not included in the regression.

There is no certain way of deciding the extent to which it is reasonable to assign the equation errors to efficiency rather than to genuine error. Ofgem has chosen the upper quartile of the error distribution, between the third and fourth companies as an indicator of the extent of inefficiency with the presumption that observations beyond that point (the first and second companies) may be the result of error. The distance between the mean and the upper quartile of the normal distribution (the size of the movement from average to frontier) is two thirds of the standard error of the equation (which, at 14%, is quite large). There is a two thirds probability that companies’ data points will be within one standard error of the estimate and, of course, a 50% probability that they will lie between the upper and lower quartiles.

The upper quartile is one standard error from the frontier company, Southern, and only 30% of a standard error from the second company, Yorkshire. The quartile frontier is 40% of the distance from the mean to the outlying observation and 70% of the distance to the second. One indication of the extent to which these individual observations are uncertain is the degree to which a change in the equation specification can move them. In total cost equations, the outlying observation (Southern) moves by more than two standard errors and is the "less efficient" of the two companies defining the lower, inefficient quartile at the other end of the probability distribution.

The upper quartile judgement that Ofgem has made is equivalent to adopting the rather optimistic assumption that half the equation error was true error and the other half indicated efficiency and then insisting that companies can move to a point that one can only be 75% confident is less than one and two thirds standard errors away from the mean of the efficiency distribution, i.e. is a level that no more than 5% of companies could be expected to achieve.

This illustrated in Figure 2 which shows an efficiency distribution centred on the lower quartile (75%) point of the random error distribution on the assumption that the total error splits half and half between efficiency and error.

1. A company at the lower quartile of the equation's true error distribution will have extreme difficulty meeting the efficiency target even if half the difference from the estimated equation line is due to efficiency differences

If three quarters of the error is true error and the remainder efficiency the position is even worse. There is not much more than 60% confidence that the imposed point is less than two efficiency standard errors away, i.e. is a level that no more than 2.5% of companies could be expected to achieve.

2. Mis-specification

Ofgem’s regressions present relationships between a measure of output and standard controllable costs but the form of relationship investigated is limited in that it is a simple linear form and omits other explanatory factors. At best this makes judgements derived from the relationships uncertain and, at worst, the equations may be mis-specified and so positively misleading.

This section considers a number of aspects of the specification:

• The assumption that costs are unaffected by mergers;
• The omission of an impact of differences in capital assets;
• Other features, such as the equation form and cost drivers such as losses.

Some of the cumulative impacts are assessed in section 5. This shows that the outcome is significantly affected if one either considers total costs or uses data from the nine corporate groups – both aspects Ofgem has started to consider in its initial proposals, but separately and with limited weights. Moreover, there are many other omitted variables – e.g. wage rates, customer mix, sparsity, voltage levels – and other possible functional forms.

1. Mergers

There are a number of functional areas where synergies could be achieved by merged DNOs in contrast to a single DNO that was a member of a wider group, for example:

• Removal of one head office
• Single regulation team covering 2 licence entities
• Single asset management team in one location
• Single control room

Improvements in procurement as a result of increased purchasing power for 2 licence entities such as materials that are specific for distribution

1. Previous understanding about the treatment of cost savings

At the last price control, a broad-brush adjustment was made to pass back to consumers the benefit of mergers that had already taken place. This was based on the assumption that "a sustained reduction of half the fixed costs, such as corporate costs, would arise as a result of distribution companies merging". The policy made at the time was to allow companies to "retain the benefit of merger savings during the five years following the merger" and thereafter reduce revenues by £12.5m p.a. (1997/98 prices).

The merger policy statement in May 2002 changed this position. From June 2002 mergers would be subject to a total revenue reduction of £32m (2001/02 prices) over 5 years to those customers affected by the transaction. This was based on the view that a merger reduces the number of comparators available to Ofgem, and by implication may affect the rate of change of the "efficient" frontier.

When E.ON were negotiating with Aquila for the purchase of Midlands Electricity, discussions with Ofgem clearly indicated that merger savings would not be passed back to customers until the end of the fifth anniversary of the transaction. On the basis of this understanding, E.ON agreed to pay the £32m merger tax. The merger was sanctioned by E.ON based therefore on the belief that they would retain any subsequent merger savings from 16th January 2004 to 15th January 2009.

1. The wrong total costs proxy

Basic economic theory suggests that any measure of partial productivity, such as operating costs per unit of output, will depend on the ratio of those costs to the other costs of production. Ofgem’s own consultants, Cambridge Economic Policy Associates (CEPA), pointed out the dangers of failing to consider this point in their paper for Ofgem of October 2003.

1. Difference from "total costs"

Ofgem had previously said that it would consider "regression analysis using total costs (opex and capex)" as a "sense check", and it has now done so. However, this measure of expenditure does not accord with any normal economic definition of total costs. Total costs are operating costs plus capital costs, in the sense of the cost of the services of the capital stock. Capex, on the other hand, is expenditure on additions to the capital stock.

Ofgem’s initial proposals describe the approach as not having strong theoretical foundation. The Frontier Economics report on "Balancing Incentives" (March 2003) argued that this methodology did not take account of past investment. In particular, unless all companies are at similar points in their investment cycle, the cash cost approach will ignore existing asset, and hence some companies with newer networks will appear more efficient compared with companies that have predominantly older networks. The report therefore recommended that since investment is long lived, a capital stock approach should be adopted for deriving total costs.

There is no reason to believe that capex will be a reasonable proxy for the capital stock. If the stock of capital assets is relatively low, operating expenditure is likely to be relatively higher (because it is substituting for the services of new assets) and there is likely to be a greater need for capital expenditure to increase the capital stock. Thus, Ofgem’s check might even increase the original mis-specification.

There is little to recommend a total cost measure of the sum of opex and capex other than that it can be measured fairly easily. The difficulty of measuring meaningful capital costs is said by Ofgem (and was previously said by CEPA) to make total cost benchmarking difficult to implement. The following paragraphs therefore consider such measurement.

2. Definitions

1. The 1990 asset value was based on the flotation price and companies were sold at different discounts to their CCA accounting asset values
2. Regulatory depreciation has differed from economic depreciation

1. Results

Table 1, which calculates the efficiency score relative to Ofgem’s definition of the upper quartile, shows that, although the total cost approach has a smaller impact for Central Networks than for many other companies, there are many marked differences that are not captured by Ofgem’s method.

1. Different adjustments for total costs

There are many possible other variations in specification.

Ofgem uses linear equations and adjusts down to the quartile by subtracting a fixed sum from each DNO’s computed costs. This seems odd when some DNOs are several times the size of others. Use of a log form would mean that a constant percentage is subtracted. Other functional forms might produce yet more different results.

Ofgem has not presented evidence to support the change to a 50% weight on line length. Even use of an agnostic 33% weighting on each would improve Central Networks companies’ efficiency scores by 2-3%.

Losses could be tested as a driver of cost by taking the inverse of the proportion of losses, and treating it as an additional exogenous variable. Alternatively, a prior adjustment could be made for differences in losses at the data normalisation stage.

The upper quartile has been taken to me the mid-point between observations for the third and fourth companies (second and third in nine company regression). This depends on the random position of those particular companies. A less variable position would be given by the point two thirds of a standard error from the regression line.

4. Cumulative correction

This section considers the cumulative impact of correcting mergers and capital costs.

1. Ordinary Least Squares

1. Results of merger adjustments

Ofgem has examined the business plan questionnaires submitted by DNOs and has initially calculated a range of fixed costs of between £14m and £22m. The regression in figure 1 shows that fixed costs are imputed to be £20m, equivalent to one-half of the fixed costs of a combined DNO group of two licence entities. However whilst Seeboard had merged with EPN and LPN in June 2002, there would not have been sufficient time for the group to have extracted any merger benefits, Seeboard should therefore be treated separately from the remainder of the EDF group.

A £10m adjustment is therefore made to each of the following DNOs:

• EPN, LPN, Manweb, NEDL, Southern, WPD S. Wales, WPD S. West, YEDL Scottish Power and Scottish Hydro

Figure 3 below shows the OLS regression and the corrected regression (frontier) for opex plus total faults, which is based on the view that all of the residual is attributed to inefficiency. By implication, the corrected regression ignores random variation including measurement error in the disturbance term.

1. Opex + total faults using 50% weight for network length with a £10m merger adjustment

1. Results of 9 groups of companies

An alternative way of taking account of merger synergies for benchmarking purposes is to model the analysis on 9 groups of companies, as Ofgem has done. This will reflect the scale of each group when assessing efficiency and assumes that each will have one set of fixed costs. It also provides a counterweight to the impression that there are 14 genuinely independent data observations. Many of these are produced by separating the costs of merged companies and are not independent, given they will generally have one management team. Pretending that they are artificially boosts the equation statistics and exaggerates the explanatory power.

Figure 4 shows the OLS regression, corrected regression and upper quartile regression under a 9 company opex plus total faults group regression.

1. Regression of opex plus total faults based on 9 groups of companies

This confirms the analysis for the 14 company regression with a £10m merger adjustment.

1. Total costs

As shown in section 4.2.3, the results change markedly when total costs are used. Southern and Yorkshire, seemingly the most efficient companies under opex analysis, are no longer in that position. Indeed, Southern appears strikingly inefficient. Seeboard’s position is greatly improved.

1. Total cost regression

When merger adjustments are applied in addition the position changes slightly further.

   

2. Total cost regression with merger adjustments



The analysis is confirmed by the nine group regression.

 

3. 9 company total cost regression

The merger adjusted or 9 company total cost efficiency scores are shown in the table below. As might be expected the range is smaller than when the analysis is done incorrectly for one factor alone and ignoring the effect of mergers.

1. Total cost efficiency scores allowing for mergers

In the second consultation paper on price controls (December 2003), Ofgem stated that regression analysis would be supported by data envelopment analysis (DEA). This policy was confirmed in the policy paper (March 2004). ). As DEA is a non-parametric technique, it has the potential benefit of being less sensitive to the low number of observations.

We have therefore re-produced the top-down benchmarking analysis for 9 groups and 14 DNOs with merger adjustments using DEA under the assumption of varying returns to scale. Whereas, in the regression analysis, the problem of multicollinearity requires the use of a composite scale variable, this is not the case for DEA and the three separate outputs (units, customers and line length), which have to be achieved separately, can each be incorporated in the analysis. This results in more companies being on the frontier. Table 3 and Table 4 below show DEA results using both three and one output for all 14 DNOs opex + faults (without merger adjustment) and for the 9 groups. The efficiency scores are compared with Ofgem’s quartile efficiency.

We then conduct DEA on total costs and making adjustments for mergers (or conducting the analysis on 9 groups) but do so only using one CSV output, although there are good grounds for using three with consequent increases in efficiency scores.

It should be remembered that DEA assumes that stochastic errors are absent, and hence the scores shown will underestimate efficiency.

1. Operating costs DEA using 14 DNOs without merger adjustments

Table 3 shows the DEA results for 9 corporate groups using opex and total faults as the definition of costs. These results are compared against the regression analysis. Although this is not a direct comparison with DEA, which is a frontier benchmarking tool, the results are compared against the regression quartile. The more complex functional form implicit in DEA produces generally higher scores.

Efficiency (%)	Opex plus total faults
DNO	DEA 3 outputs	DEA one output	Regression quartile
Seeboard	91	91	70
UUE	91	91	82
Midlands	92	92	86
East Midlands	100	100	103
WPD	77	75	78
N&Y	100	92	98
SPMW	77	78	84
SSE	100	100	108
EdF	100	100	86

1. 9 groups opex DEA

1. Operating cost frontier adjusted for merger costs (shown against unadjusted actual costs)



  

2. 9 company operating cost DEA frontier

Figure 10 and Figure 11 show total cost DEAs allowing for merger effects. The shape of the frontier is more plausible than that for operating costs alone and the efficiency scores are comparatively bunched. They suggest that, given the uncertainty of all the possible approaches to the analysis only minor adjustments for efficiency would be warranted.

1. Total cost DEA frontier adjusted for mergers (shown against unadjusted actual costs)



 

2. Total costs - 9 groups

Data envelopment analysis (DEA) is a non-parametric benchmarking tool, which uses linear programming to derive frontier efficiency. The constraints in the deterministic model assume a probability of 1 that as set out below.

Min efficiency (q)

subject to

yl - y >= 0

xl - xq <= 0

However, benchmarking is not an exact science, and is vulnerable to modelling errors including the drivers of cost that have been assumed. It is possible to also apply the same probability theory to DEA, and hence make assumptions regarding the degree of confidence that all variation from the frontier is explained by inefficiency alone.

Stochastic input and output constraints can be applied to DEA to assign a probability of less than one to address this issue. The approach taken is based on chance-constrained programming developed by Land, Lovell and Thore (1993). This allows the model to be transformed into a non-linear programming problem. Stochastic DEA assumes constant returns to scale, which is in contrast to the DEA approach applied above that assumes that some of the inefficiency is explained by scale rather than technical inefficiency as a result of adopting varying returns to scale.

To assess how the introduction of stochastic DEA affects efficiency, Table 5 compares DEA with a number of probability levels; 25%, 50% and 80%, with opex plus total faults adjusted for mergers as input. Given that Ofgem has only applied normalisation adjustments to 2002/03 data, the input constraint is treated as deterministic, the approach applied in conventional DEA. However stochastic output constraints are used based on 11 years worth of data from 1992/03 to 2002/03.

The distance between the DEA frontier and the chance-constrained programme represents the stochastic error term. Some DNOs (including the company lying on the DEA frontier) will have performance in excess of 100% efficiency when this approach is applied to the data. The results show that even if a 25% probability is used for stating that the gap from the stochastic frontier is explained by inefficiency, the stochastic error for Central Networks Midlands represents 39% of the DEA performance gap. The stochastic element of the performance gap rises to 69% if a 50% probability is applied to the data.

A comparison of the results of stochastic DEA assuming constant returns to scale and DEA under varying returns to scale suggests that for Central Networks Midlands, the stochastic error term under a 25% probability is analogous to the inefficient scale assumption in Table 5 below.

1. Comparing DEA and stochastic DEA using 14 DNOs with merger adjustment under a constant returns to scale assumption

4. Averages and frontiers

The preceding analysis argues that Ofgem has specified its benchmarking comparisons incorrectly. It also demonstrates that there is considerable uncertainty in the comparisons and that results are affected by the method of comparison used.

If no relative efficiency differences can be demonstrated with any degree of confidence, cost projections should be based on the potential for general efficiency gains relative to each company’s cost levels.

If relative efficiency differences can be demonstrated, cost projections might be based on either the average cost for all companies implied by the benchmarking relationship or on the costs incurred by the most efficient companies. It is probably the former that should be the most influential.

1. The average firm earns the cost of capital

The calculations involved in determining the cost of capital are based on an average over all companies. The averagely efficient company should earn the cost of capital; more efficient firms should earn more; and less efficient firms less.

The argument is sometimes put forward that this is not the case for regulated utilities because they are not subject to competitive forces and so inefficient firms are not driven out of the industry. Only the more efficient firms are comparable with the economy as a whole and should earn the cost of capital.

However, although there is little competition to retain customers, there is both capital market competition (in which less efficient firms are taken over) and comparative competition and so, in the absence of evidence that efficiency differences are larger in electricity distribution than in other industries, the argument is not persuasive.

The projected costs to which a normal return should be added are those of an averagely efficient company.

2. Implications of frontier or quartile based targets for average productivity growth

The costs of the present frontier might be helpful in projecting the costs of a normally efficient company because the difference between the two represents the scope for improvement. However this presupposes that:

• The frontier is known with reasonable accuracy
• It is reasonable to expect catch-up within the assumed period

In fact the position of the frontier will be particularly uncertain since it depends on the specification of the relationship and the data of the outlying company or companies. Moreover, in most industries there is a gap between frontier and other companies and best practice is not generally adopted within a short period.

Use of the average would be preferable because it removes the need to define a frontier or to specify how fast that frontier should be approached. Projecting average costs is easier. Given that post-2004-05 projections use a trend rate of efficiency gain, there is no reason why this should not be used from 2002-03.

The 4% movement in costs this would produce is far less than that of two thirds of a standard error to the upper quartile, which is 9%. 25% of companies are at the quartile. The remaining 75% have to move an average of something like 14% in two years. That is 0.5x14x0.75 or 5.25% a year as an average for the industry as a whole. There has been no evidence cited to suggest that this is reasonable.

Furthermore, the incentive properties of using the average are preferable, as the work by Frontier Economics showed, since it in effect mimics the role of competitive markets. There is no strong incentive for any individual DNO not to undertake cost reduction initiatives when they are feasible because future allowances are not affected by this action.