Home | Contents | Summary | CV

Geoff@Horton4.co.uk www.Horton4.co.uk

SUMMARY

PURPOSE OF BENCHMARKING

THE FULL MODEL

THE FEASIBILITY OF REGRESSIONS

NECESSARY A PRIORI ASSUMPTIONS

POSSIBILITY OF DISAGGREGATION

APPENDIX1 RECENT DEVELOPMENTS

APPENDIX2 STATISTICAL METHODS

United Utilities (UU) and Powergen (PG) are preparing for the upcoming price reviews of their regulated electricity businesses, United Utilities Electricity (UUE) and East Midlands Electricity. Horton 4 Consulting has been retained to help prepare a case on how Ofgem should take account of ownership differences between distribution companies in making comparative assessments within the price review process. Ofgem relies heavily upon assessments of the relative efficiency of companies to derive its estimates of the scope for future cost reductions. Since the last price review several distribution network operators (DNOs) have come into common ownership via mergers and take-overs so that at present there are 14 DNOs owned by 8 companies.

This paper addresses the problems of benchmarking relative efficiency given the reduction in the number of distribution owners. It discusses:

• The purpose of benchmarking relative efficiency;
• The full model necessary to assess relative efficiency;
• Its functional form, the specification of economies of scale and scope, and their relationship to mergers;
• The method and difficulties of a 14-company cross-sectional regression;
• An 8-company regression and the necessary a priori assumptions ; and
• The possibility of disaggregated efficiency comparisons of the sort done by OFWAT.

Appendices briefly discuss:

• Recent developments in efficiency benchmarking; and
• Statistical methods of assessing efficiency;

Throughout the discussion it is helpful to bear in mind that the central problem in assessing relative efficiency for distribution businesses is that there are more potential explanations for differences in unit costs than there are data observations to test whether they are valid explanations. It is therefore necessary to expand the database (e.g. by using international comparisons) and/or to reject or restrict explanations by a priori reasoning in order to make any assessment at all.

This report concludes that these problems are now even more acute and that:

• Ofgem’s 1999 method of frontier definition is not a reasonable approach;
• There cannot be said to be 14 distinct cost observations and there are problems even in generating 8;
• These 8 are bunched in three groups from which it is difficult to draw conclusions;
• It is likely that the width of the range of plausible a priori judgements involved in positioning any regression or frontier line means that the uncertainty surrounding the position of the line exceeds the deviation of most companies from it;
• Disaggregated analysis may be of value but it is complex and it would be difficult to prepare the data and methodology in time for this review.

When cost data are compared it is necessary to:

• (when comparing total companies - presently 8) use regression "dummies" or adjust the costs of merged companies to subtract a sum representing the fixed costs they could not yet be expected to have saved by merger. Ofgem’s assumptions would suggest about £13m for a non-contiguous merger (or a contiguous one that is not long-standing) and a further £6.5m for all recent mergers. These deductions should be made for each additional company (i.e. twice when three are merged, three times for four, etc.).
• (when comparing individual DNOs - 14) use regression "dummies" or adjust the costs of merged companies to add a sum representing the fixed costs they could be expected to have saved by merger. Ofgem’s assumptions would suggest eventually about £13m for each contiguous company (£17m where 3 companies are merged, £19.5m for 4 companies, etc.) and £6.5m (£8.5m, £10m,…) for each non-contiguous company, but smaller amounts when the merger is recent;
• remember that the data observations are uncertain, that many important explanatory variables are omitted, that the significance of the results will be exaggerated by any statistical tests, and that each record on a chart would be better represented by an ink blot than by a point.

The robustness of any conclusion drawn from comparisons between companies should be tested by checking that it remains valid when other plausible a priori assumptions are made as a basis for the comparison.

If, as seems likely, it is not possible to reject the hypothesis that companies are efficient, price controls should be set on the basis that an individual company’s future costs will only differ from present ones in real terms insofar as total industry productivity growth (or factor price change) is likely to differ from that in the economy as a whole.

The main objective of benchmarking is to establish a level of costs that it would be reasonable to expect a regulated company to achieve during the forthcoming price control period.

Originally, the aim was to establish a norm which less efficient companies might reach, thereby achieving greater efficiency growth than that in the norm itself. Companies whose costs were lower than the norm would be expected to earn super-normal profits unless they reverted to the norm. There was separate assessment of the potential for growth in efficiency in the norm, which might be influenced by, say, how recently companies had been privatised¹.

More recently, there has been more emphasis placed on establishing an efficiency frontier, to which all companies might aspire, with less weight put on possible future movements in the frontier. Indeed, some might argue that there would be no reason to expect the movement in the efficiency frontier in a particular industry to exceed that for the economy as a whole, which is already embodied in movements in the RPI, although others would claim that (given slow productivity growth in services) production industry efficiency growth should exceed that in the RPI.

The shift from norm to frontier has involved some discussion of, but no firm conclusions on, the extent to which a company at or near the frontier would expect to earn a rate of profit higher than its cost of capital.

The costs considered have normally been controllable operating costs, since these can be most easily changed in the short term. This originally excluded both capital costs (return and depreciation) and operating costs that are hard to control but the present list of excluded items (network depreciation, network rates, NGC exit charges and gains on the sale of fixed assets) in practice amounts to excluding capital costs only. This is done not only because of the desire to concentrate on costs that can vary in the short term but also because there is considerable disagreement on how to measure capital costs both in general and in electricity in particular, where CCA values have not been accepted by the regulator.

Efficiency is assessed by comparing actual costs with a level of costs that would be expected given the outputs being produced, any other inputs required, and environmental factors. Determining what would be expected requires the specification of some sort of functional form relating outputs to inputs and to external factors, and involving returns to scale that may be constant, increasing, or varying at different company sizes.

The main outputs of distribution – the numbers of customers connected, peak kW demand, kWh distributed, and distance distributed (length of network) – are highly correlated so that it is difficult for statistical methods to be used to assign weights to them as cost drivers. Table 1 of annex 3 of Ofgem’s August 1999 distribution review draft proposals used cost allocation to derive weights to combine three of these outputs into a single variable but the cost attribution appears rough and ready and the weights used differ slightly from the results in the table.

Ofgem’s final December 1999 distribution proposals recognised three further outputs - (lack of) customer complaints, energy efficiency and quality of supply. This last, in particular, is clearly important and the October 2002 Ofgem IIP paper has attempted to construct a single aggregate measure with which to compare companies’ quality of supply output.

However, it remains the case that there are many (about a dozen) different main outputs – the quantity, peak and distance of electricity distributed and the extent to which it is interrupted, all at different voltages – and, for outlier companies at least, results of efficiency comparisons can differ markedly depending on the weights assigned to each. In addition, there are further outputs of energy efficiency, complaints, and performance under the various standards.

Inputs include all costs, including capital costs. Even abstracting from problems stemming from accounting differences between companies’ classification of costs between capital and current expenditure, there will be real differences in capital assets used. This may stem from conscious choice between policies of maintenance or replacement, or between different choices of replacement assets, but will also result from what happened in the nationalised industry – the date of expansion of the network and so the average asset age, the extent of undergrounding etc.

Operating cost efficiency cannot be considered separately from estimates of the contribution of capital assets but little progress has been made in this area.

It may be the case that, if all outputs (kW, kWh and km at all voltages) were considered separately and all capital assets were fully identified (particularly the extent undergrounded), there would be few external factors to take into account. The major external factors commonly suggested do often appear to be proxies for disaggregated outputs. For example, customer mix may be a proxy for the average kWh per customer and the voltages at which they are connected, customer density for the length and nature of line needed to connect them, and customer sparsity for length of line and voltage.

However, these outputs are not considered separately and there is room for debate as to whether any means of combining outputs into a single variable deals adequately with the impact of mix, density and sparsity² or whether these need to be inserted as further explanatory variables. Moreover, other factors may feature as cost drivers – tree growth, coastal corrosion, serving offshore customers, local labour market conditions, etc.

The relationship between outputs, inputs and environmental factors is likely to be complex and may not easily be represented by a simple function relating the variables. If it can be represented, there will be room for debate as to the form of function that will be appropriate. In a situation like that in electricity distribution, where there are few degrees of freedom³ even before the functional form is considered, it may well not be possible to identify the appropriate form from inspection of the data.

Perhaps the most important aspect of the functional form is the degree of economies of scale – the extent to which output rises when inputs are increased and the inverse of the elasticity of costs with respect to output.

This relationship may vary as output increases. For example, the standard economics textbook⁴ perfect competition equilibrium requires each firm to be producing at the minimum point of its long run average cost curve. But this is where returns to scale are briefly constant as they change from having been increasing to decreasing. If the firms were forced to be different sizes and not all driven to the lowest cost size by the forces of competition (as is the case in electricity distribution) they would have not only different average costs but also different marginal costs.

This particular assumption about the shape of the cost curve is not needed in imperfectly competitive markets (where there is a presumption that firms will produce at a point where average cost is still falling) but, even though returns to scale might be increasing, there is no reason to assume that the elasticity of costs with respect to output remains constant.

Having a) decided on an output variable and b) discounted other inputs, Ofgem’s 1999 conclusions on the shape of the efficient production frontier were based on three further assumptions:

1. A fixed cost of around £25 million, supported both by regression results and consultants’ inspection of costs;
2. Constant marginal costs with respect to output;
3. Accuracy in the reported costs of those companies deemed to be on the frontier (and that they were on the frontier).

All three assumptions are questionable.

4. Brief consideration, and inspection of the costs of small water companies in the UK or small distribution businesses elsewhere, will reveal that the non-capital operating cost of serving a small number of customers (say 30) is less than £25 million a year. At best, one might claim that all distribution companies serving at least 500,000 customers will have fixed costs of £25 million a year⁵. It would seem to be an open question whether UK distribution businesses are all in the same size range and will have the same fixed cost level. Finding the answer is made more difficult by the fact that it is in the interests of smaller companies to demonstrate that fixed costs are high, while it is the larger companies (whose costs in the categories deemed to be fixed will be high) that tend to maintain fixed costs are low.
5. Marginal costs may decline with increasing output, for example if larger companies can purchase more cheaply or organise their work forces more efficiently. OFWAT’s disaggregated water models have several different functional forms including some forms with constant returns to scale and other log-linear forms where the marginal cost varies⁶.
6. It is likely that the costs of a company that appears to be on the frontier will contain a negative measurement error. If there were measurement error but no difference in efficiency, Ofgem’s method would produce an appearance of an efficiency difference. The larger the difference in efficiency relative to the measurement error, the less important the impact of error on the gap to the frontier will be, but, even if efficiency differences were large relative to measurement error, it would still be more likely than not that some of the measured gap resulted from measurement error.


Figure 1 on the left below (using Ofgem’s 1997/98 cost data) compares Ofgem’s frontier with one that moves the assumptions towards the other extreme and adopts fixed costs of £5 million, a marginal cost that falls 0.5% for every 1% increase in output and a measurement error of around 10%. The position of most companies relative to the frontier is further improved if Ofgem’s economies of scale assumption is retained, as in figure 2 on the right.



Figure 1 Figure 2

These alternative frontiers are arbitrarily selected but it is not a simple matter to reject them as implausible. Figure 3 below compares the frontier with the results of a simple log-linear regression of 1997/98 costs against "adjusted customers". It is not claimed that this necessarily represents the true relationship but it has an adjusted R squared statistic of 0.54, compared to 0.37 for the linear regression used by Ofgem, and looks more like the alternative frontiers in the charts above.

Figure 3

  

Economies of scope

There are also likely to be economies of scope in combining distribution with other activities. There may be some savings from sharing corporate costs with any other business. There may be further savings from combining electricity distribution with another similar business, such as water distribution.

It can be argued that such economies differ from those of scale in distribution because the latter result from the licence, and so should be treated as an environmental factor, whereas the former are open to all companies and so should be treated in the same way as any other efficiency gain. However, this is not debated in this note.

Mergers

Three DNOs are now independently owned, two are owned by SP, two by WPD, two by SSE, three by the LE group (and so EdF), and two (Yorkshire and Northern) have a high degree of common ownership.

Ofgem assumes that economies of scale are available to merged companies. Ofgem’s merger policy, as described in paragraph 6.40 of the last distribution review conclusions and its May 2002 policy statement, is to assume that half of distribution’s fixed costs ("of the order of £12m a year" in 1997/98 prices) can be saved in one of the two companies as a result of the merger and that, of that amount, half should be returned to customers in the first five years.

No distinction is made between the merger of neighbouring networks and those that are physically separated, although one might expect the savings of fixed costs to differ between the two circumstances.

No adjustment is made for any savings resulting from joint ownership with distribution networks in other countries. However, these might be argued to be similar to those from economies of scope discussed briefly in paragraph 31.

Ofgem’s merger policy document also said, "Following a merger companies will continue to provide some information on a separate basis. This will enable some of the comparisons which were possible prior to the merger taking place still to be made. However, the value of those comparisons is weakened because they no longer reflect the efforts of different independent management teams trying to improve the way they operate in terms of cost and quality."

It follows that the number of independent data observations is reduced and the ability to identify inefficiency is even less than it was at the previous review. It is possible to construct comparable data by continuing to collect data from merged subsidiary companies separately and adding to each an estimate of the costs it has saved by merging but:

1. The data observations will not be genuinely independent and should not be considered to give a full degree of freedom; and
2. The observations will be subject to considerable measurement error.

It is not even possible to treat the new merged company as a single company and accept that there are now only eight data observations because the merged companies will be at different stages on integration and cost saving and none will be in the same position as if they had been a single company for fifty years. Even these data would need adjustment.

There are two main issues to be considered in relation to 14-observation comparisons – how to treat the individual data points given that some of the observations are of parts of a single company and what significance this has for the results derived. The former is considered first.

In order to continue to use all 14 DNOs as observations in a cost comparison it will be necessary to make some allowance for the effect of the mergers that have taken place. This could take the form either of a merger "dummy" or of a priori adjustments to the data.

One response would be to use in the equation to be estimated a dummy variable that takes a unit value for a merged company and a zero value for an unmerged company. This would perhaps result in a coefficient that would be an estimate of the impact of mergers on costs and remove the impact of merger from the estimates.

Unfortunately, there is not a single merger effect. There will probably be different effects where three companies have merged rather than two, where authorised areas are contiguous rather than far apart, and where the merger is long-standing rather than recent. There may also be a different impact on the dominant partner from that on the other partner. Even if one ignores the last point, treats contiguity as a binary feature⁷ and separates mergers into >5 years, 2-5 years, and < 2 years, there would be five dummy variables. This is only one less than the additional observations introduced. One is very close to saying that each merger is special, allowing for its special impact, and so "dummying it out" of the data set.

The alternative is to claim a priori knowledge of the effects of mergers on costs and to adjust each data observation accordingly. However, this is not easy. Assumptions are made when mergers take place but these are ad hoc adjustments⁸ based on an assumption of an imperfectly known cost function. They are a pragmatic solution to a situation where a number has to be found and the null hypothesis is not that there are no savings from merger.

The situation in comparative benchmarking is different. The null hypothesis is that all companies are similarly efficient (or at least that any one company is no more likely to be less rather than more efficient than the others). The aim of benchmarking is to produce evidence that this is not the case and to identify the more and less efficient companies. Uncertain a priori adjustment weakens the power of that evidence.

Some corporate savings can be obtained from common ownership with any other business but the restrictions imposed by the separation condition 39 of the distribution licence limit their size. All DNOs are part of larger groups and so these savings are generally available and no adjustment need be made in respect of them.

More savings are available from common management of two or more distribution businesses, consolidation of call centres, billing etc. Ofgem assumes that half of one distribution company’s fixed costs ("of the order of £12m a year" in 1997/98 prices) can be saved by merger. The assumption when three companies join appears to have been that one set of fixed costs (i.e. two halves) is saved.

If DNOs really do have fixed costs of £25 million, it should presumably be possible eventually to remove the entire sum by merger. It would be surprising if all fixed costs could be removed by non-contiguous DNOs and, even where companies are contiguous, other savings (particularly of capital costs) may take a long time to achieve. However, it does not seem unreasonable to presume that merged neighbouring companies could at some point achieve the operating cost level of a single company of that size. Depot and transport fixed costs, for example, could be reduced by contiguous companies fairly rapidly.

There is a counterargument that these costs are not really fixed because economies of scale could be achieved by outsourcing but this is a difficult argument since most activities could be outsourced in this way. Indeed, it seems unlikely that there would be £25 million of fixed costs that could not be outsourced.

Contiguous merged companies might also be expected to have lower marginal costs than the same businesses before merger.

Merger cost savings would not be achieved immediately. There might even be an initial increase in cost, although this might be recorded as exceptional.

Manipulation of the basic data gives the impression that there is more evidence than is really the case. To take an extreme example it would be computationally possible to divide each DNO into the counties (or major postcodes) it contains and divide the costs between them according to the number of customers of different types. This would appear to increase the degrees of freedom of the statistical tests but, in reality, would be entirely arbitrary and would do nothing of the kind. Division of the costs of merged companies, while doubtless less arbitrary, has something of the same effect.

Statistical techniques implicitly assume a number of properties in the data being analysed. Ordinary least squares estimation – the method used by Ofgem - makes five main assumptions, but they may be affected by this data manipulation.

The technique estimates an equation of the form Y_i = a + bX_i + e_i where e_i is the "error" term, the extent to which the equation fails to explain the data observation i. It does so by choosing values of a and b to minimise the sum of squares of the errors.

The first assumption is that the errors have zero mean. If they do not, the estimate of the constant term will be biased. Clearly, if the adjustment for merger effects is wrong the average error of the equation will be non-zero and the constant term estimate biased.

The error term is assumed to be normally distributed. If it is not, the confidence interval tests for the coefficients are affected. Both the use of adjustments and the methods of cost allocation used within merged companies may well produce sets of errors that are not normally distributed.

If the error variance differs between observations, or is correlated with the explanatory variable, the confidence of the results may be overstated. This is likely with a linear estimation (and is a reason why log-linear forms are often preferred) and is also likely if costs are assigned within merged companies on a pro rata basis.

If errors are correlated with each other the results will also be affected and confidence in them overstated. When dealing with time series, there are standard techniques for changing the method of estimation to allow for the impact of serially correlated errors but it would be difficult to do so with cross-sectional estimation as is the case here. However, there is reason to believe that correlated errors might well be introduced either by the method of adjustment or by the methods of cost allocation within merged companies.

Fifthly, the explanatory variables are assumed not to be correlated with the errors. If cost allocation was done by number of customers etc. some correlation would be introduced and this would bias coefficient estimates. However, this is unlikely to be a major factor.

The use of 14 observations gives a misleading impression of the statistical validity of results found using the data, because the observations are not genuinely independent and are constructed using uncertain adjustments. It may also introduce bias to the results. Perhaps the most worrying feature is that, if 14 observations are used, it would be clear that the results would be affected and their validity significantly overstated, but it would not be clear by how much.

Ofgem’s use of data for two or three companies to define an efficiency frontier in 1999 was contentious. Repeating that exercise now would be even more so. Uncertainties in both adjustment and allocation mean that if subdivided merged companies were used to define an efficiency frontier⁹, a very large margin of error would need to be allowed. It seems unlikely that the margin would be smaller than the gap between the estimated efficiency of the merged company and that of other companies.

Adjustments should probably be made to the data, or dummies used in estimation, whether 8 or 14 observations are used. The five recently merged companies may have costs higher than those of a company of a similar size that had been so for a long time and had set up a single network. However, adjustment introduces a further degree of uncertainty and, for this reason, estimation using dummy variables with the full data set may be less distorting than making adjustments and using either 14 or 8 observations. It would, of course, still be necessary the effect that disaggregated data from merged companies would have on the estimates and on the power of the statistical tests.

As discussed above, there is little objective basis for a choice of adjustment. For illustration we adopt a version of Ofgem’s merger policy assumptions here, that a merged company will save £12.5m (in 1997/98 prices) a year but only half that sum in the first few years. In figure 4, where data for 14 companies are shown¹⁰, £6.5m is added to the 1999/2000 operating costs less depreciation of Scottish Power, Manweb, Southern and Scottish Hydro, £3.25m to Northern, Yorkshire, WPD and Swalec and £4.25m to London, Seeboard, and Eastern. No allowance has been made for the possibility of further gains by contiguous companies.

The data pattern appears to have a large random element and perhaps reflects the uncertainty relating to the observations and the adjustments.

Figures 5 and 6 show data for eight companies, with and without adjustments. These are on the same assumptions as in para 63 and involve subtracting £13 million from the combined costs of SSE and SP/Manweb, £19.5 million from Northern/Yorkshire and WPD, and £39 million from the EdF group.

The data are in three clusters and the effect of the adjustments is mainly to move the two right-hand clusters down a little. It would be possible to use many specifications and functional forms to fit the data. Both linear and log-linear forms using Ofgem’s output variable do so well, but this will probably also be the case for other forms, outputs and additional variables.

Comparisons within clusters provide little more information. The right-hand cluster is a single data point and the ranking in the others is affected by the assumptions used and may be influenced by special factors.

There are now even fewer data observations from which to draw conclusions on efficient costs. Only three companies remain separate. The remainder have combined with other companies, which Ofgem expects to have affected their cost levels.

Comparisons can only be made on the basis of a number of prior assumptions as illustrated in the table below. Since the assumptions are uncertain, the conclusions need to be robust against plausible variations in the assumptions. Suggestions for sensitivity tests are made in the right hand column. However the assumptions are both numerous and important.

Topic	Assumption	Sensitivity
Mergers	Merged company data can be adjusted to be compared with unmerged companies	Different sizes of merger savings
		Further savings from proximity
		Varying time profiles of savings
Data	Accounting data now comparable	SFA analysis with varying efficiency
Outputs	50:25:25 weighting of customers, kWh, km	Use of different weights e.g. 33:33:33
		Checks against voltage levels etc.
		Addition of other outputs (quality etc.)
Capital stock	Operating costs invariant to capital stock	Re-estimate using capital asset proxies
External factors	Absence of other cost drivers	Re-estimate using additional variables
Equation	Functional form	Linear/log linear etc.
		Different sizes of fixed costs
		Other forms allowing variable marginal costs

It is doubtful whether any conclusions will be robust to these sensitivities, but this is not surprising given the lack of data, its poor quality, and the large number of alternative hypotheses that must be eliminated before inefficiency can be presumed.

An alternative approach would be to attempt to examine efficiency at a lower level of aggregation as OFWAT does and as Andersens attempted to do for NGC. That may require fewer assumptions and generate more data points. However, it does require:

• Separability of cost data to the units to be compared
• Similarity of operation and output at the unit level, which may be more the case for sewage plants than parts of distribution networks.

There are substantial difficulties in a disaggregated approach, as outlined briefly below. Given the problems involved it would be difficult to develop data and methodology in time to introduce the method at this review.

There are data problems in aggregate and these may be even greater at a disaggregated level. There will be differences in cost allocation and charging methods and almost certainly still a need to make cost adjustments in respect of merged companies.

An obvious, but widely recognised, danger of disaggregated analysis is that it can be misused to create a supposedly efficient, but in practice unachievable, virtual company by cherrypicking the cheapest cost example in each cost category when that lowest cost result has not been achieved by greater efficiency but by differences in cost allocation.

Unless there is a very high degree of confidence in the data, disaggregated analysis should only be used to assess norms and not frontiers.

Although disaggregated data are often benchmarked in a simple form, e.g. comparisons in cost per customer or per km, there is no reason to believe that costs at this level can be explained by the simple cost function assumed by the comparison. There are likely to be the same sort of problems of defining output and external cost drivers, and of assessing the effects of economies of scale as are present at the total cost level. It may also be the case that expenditure in one category will be affected by how much is spent in another category.

If it is not possible to reject the hypothesis that all companies are similarly efficient (and not less so than in the economy as a whole), price controls should be set on the basis that an individual company’s future costs will only differ from present ones in real terms insofar as total industry productivity growth (or factor price change) is likely to differ from that in the economy as a whole.

Since Ofgem’s last distribution efficiency comparison in 1999 there have been a number of developments in relative efficiency assessment:

• The Dutch electricity regulator, assisted by Frontier Economics, conducted a data envelopment analysis (DEA) of Dutch distribution companies¹¹ and NuonNet engaged NERA to criticise it¹².
• The Civil Aviation Authority reviewed benchmarking and made limited use of it in the recent review of airport price control¹³.
• Oftel commissioned a report from NERA on BT’s relative efficiency¹⁴
• Ofgem developed "an initial version of a model" for "making more robust comparisons of quality of supply performance" in distribution networks¹⁵.
• Ofwat continues to publish efficiency comparisons and has also published further comparisons of the relative performance of the regulated water and sewerage companies in England and Wales against companies in Australia, the Netherlands and the USA¹⁶. The Competition Commission endorsed OFWAT’s method when two companies appealed the 2000 determination¹⁷. This contrasts with the MMC’s conclusions on electricity benchmarking in 1995 and 1997.

Dte chose to use DEA on the rather surprising grounds that "While all methods are less effective with smaller sample sizes, DEA techniques are less data-intensive than econometric methods." The various methods of comparison are briefly discussed in appendix 2 but, put crudely, they can be separated into:

• Regression techniques that use statistical tests to help determine the important cost drivers; and
• A linear programming technique (DEA) that draws a frontier between efficient similar (or peer) companies using preselected cost drivers (and perhaps preselected weights representing their importance) and gives each company a percentage efficiency score relative to its peers.

Dte said, "The data restrictions therefore lead us to adopt a DEA approach as our principal methodology to estimate efficiency. Since model selection is not based upon econometric tests, an alternative process is needed to select the variables for the efficiency analysis." This process was in effect argument with the companies together with some sensitivity analysis of the importance of the assumptions made.

NERA’s response included that:

• The gap to the frontier stemmed not only from inefficiency but also from defects in the model. "DEA as a technique is prone to certain biases and errors. In a large sample of similarly sized observations, such biases and errors are relatively unimportant. However, in a sample of only 18 disparate observations, the problems with DEA are likely to have a major effect on results." ¹⁸
• There were problems in the data that were used. "We found (1) that the data is prone to measurement errors, (2) that it may include biased estimates, (3) that the process of standardisation may be arbitrary (e.g., RORC) or incomplete (LV-connected generation at peak), and (4) that some variables suffer from an aggregation bias against large companies." ¹⁸ ; and
• The specification of the model effectively determined the answer. "With a different and more reasonable specification of the input and output factors in the DEA model, NuonNet would very nearly have reached the frontier, with a DEA score of 99.7 per cent."¹⁸

The CAA’s study has perhaps two main points of interest here.

1. In its "Overview of the use of benchmarking by other regulators" in the December 2000 consultation, CAA does not refer to the 1999 electricity distribution exercise as an example of benchmarking, although the distribution final conclusions document is cited in the bibliography. In energy only the bottom-up benchmarking of NGC commissioned by Ofgem from Andersens is described, although OFWAT’s work, which was contemporaneous with the distribution review, is dealt with fairly fully.
2. In spite of having assembled a substantial amount of international data and commissioning external work on it CAA considered that it was not yet ready to use it. "Overall, the results from the NERA and ESI analyses seem to suggest that, given the limited number of degrees of freedom available and the quality of the data collected so far, it is difficult to establish a robust cost frontier model that can inform on the relative efficiency of the regulated airports and identify the areas and scope for efficiency gains in comparison with the best practice airport(s) in the sample. …. The CAA takes the view that in principle benchmarking can deliver a substantial improvement in regulatory incentives and intends to further explore the applicability of benchmarking to airports by expanding the data set and improving the quality of the data. This process is unlikely to yield results for the current review but should be useful for subsequent reviews."

As briefly described in paragraph 79, there are two main approaches to estimating appropriate or optimal efficiency levels – through regression analysis or linear programming. The former can be further subdivided into more simple techniques such as ordinary least squares (OLS) and more complex approaches such as stochastic frontier analysis.

Ofgem used ordinary least squares in its regression analysis. This technique assumes, inter alia, that the errors in the data distribution are normally distributed and draws a regression line through the data points in such a way as to minimise the sum of squares of the deviations of the data points from the line.

Various statistics can be estimated, such as R-squared - the percentage of the sum of squared deviations from the mean explained by the equation – and confidence intervals for coefficients such as the constant or fixed costs term.¹⁹

However, many properties of the regression line depend on which variables are included and the functional form specified. Ideally one would test the preferred specification against broader specifications in which it is nested but this is impossible when there are few data observations.

Stochastic frontier analysis allows for error in the data and estimates an equation that assumes that the difference of each observation consists of two elements, inefficiency and error. The distribution of the inefficiency component is inserted as a prior assumption.

Data Envelopment Analysis is a linear programming technique that compares companies with others that are similar to it. It makes few prior assumptions and can accommodate any number of inputs and outputs. However, it needs "several hundred observations" and "appears to cope best under the following conditions:

• a data generating function [i.e. a frontier] which is reasonably close to linear;
• distribution of efficiencies (such as the exponential distribution) with sufficient observations close to the frontier for the complexity of the function form;
• if the underlying cost function is not linear, a small variation in cost drivers is desirable." ²⁰

These conditions are not satisfied in electricity distribution and it is difficult to imagine that this technique would be helpful in the analysis.