Quieting the Noise in Economic Impact Estimates | By Thomas A. Hazinski

2022-07-16 00:36:36 By : Ms. Rose Wu

An assessment of the accuracy of traditional approaches to valuing the economic contribution of convention, sports, and entertainment venues reveals significant limitations of economic impact models that tend to produce noisy and sometimes biased estimates. Improved methodologies and the effective use of available data sources that address some of these limitations can provide more reliable estimates. Venue owners and destination market organizations need improved methodologies to estimate the return on public sector investment in capital improvements, marketing and sales, and venue operations.

State and local governments invest in capital development, operations, and marketing of convention, sports, and entertainment venues and events to generate positive economic impacts in their communities. Destination marketing organizations (“DMOs”), venue managers, and governments typically measure economic impacts by estimating the spending of visitors imported into their community. In the United States and Canada, public entities require economic impact estimates to justify the use of public funding sources. The traditional procedure, or “spending approach,” to economic impact analysis seeks to estimate a return on investment of lodging and other tourism taxes that provide the primary funding sources. Return on investment is typically measured in terms of new income imported to the local economy, associated job creation, and new tax revenues.

But tracing the path of a dollar of spending in the tourism economy presents a complex challenge for information gathering and economic modeling. This leads to the question of whether analysts generate accurate estimates of economic impact. Many economists are extremely critical of economic impact estimates citing exaggerated and wide-ranging claims about individual projects.[1]

The output of economic impact models results from multilayered assumptions about the number and type of visitors and the amounts and categories of their spending. The number of new visitors, “heads-in beds,” and the daily spending of visitors attracted by events are typically key inputs. The figure below illustrates a simplified flow of information within an economic impact model.

At best, these models produce rough approximations and noisy estimates of the total spending generated by visitation to events and venues. We know that spending estimates are noisy because different consultants and impact calculators produce widely varying estimates of the impacts of the same venues and events. Even a single consultant may generate a significant variance in estimates when making multiple attempts to model the impact of the same events.

These variations are not necessarily due to a lack of diligence or objectivity on the part of estimators. Rather, wide-ranging estimates are caused by the need to layer many levels of assumptions on top of each other. The formulation of assumptions involves not only the application of objective information but elements of human judgment on how to use available data. A reasonable range of error in estimates at each level of assumption can compound to create large errors in the final estimate.

To evaluate this hypothesis, the author created an impact model that generates a hypothetical annual economic impact of $775 million. The model has nine layers of assumptions needed to estimate the spending of overnight visitors to a given destination. The figure below shows three applications of those assumptions. In two scenarios, estimation errors of plus or minus 10% are applied alternately to each assumption. In Estimate A, a 10% overestimation was applied to the first assumption, a 10% underestimate was applied to the second assumption, and so on for all nine assumptions. For Estimate B, the opposite pattern was applied with a 10% underestimate applied to the first assumption. A third scenario shows a biased estimate of 10% positive error in each assumption.

The actual economic impact of $775 million is the product of the nine perfectly accurate parameters. Estimates A and B are based on alternating 10% positive or negative errors in each assumption. While a 10% confidence interval is considered a reasonable degree of accuracy in any given parameter, the cascading effect of this acceptable error level results in estimates ranging from 120% more or 82% less than the hypothetical actual in Estimates A and B, respectively. In Estimate C, each of the assumptions has a positive bias of 10%, resulting in an estimated 1.89 times the actual economic impact. Small biases in individual assumptions are likely to produce large biases in overall estimates.

Since all economic impact studies are predictions, we do not know actual impacts, which makes it difficult to distinguish between biased and unbiased estimates. What is interpreted as evidence of bias is sometimes just the result of noisy estimates. It is also difficult for estimators to determine their confidence level in the estimates of the parameters they use in their models. When estimating the impact of events that have already occurred, historical data on the visitation and spending generated by the event may be available. Still, when estimating the impact of future events, the use of historical data may be limited or absent altogether. In the latter case, errors in estimates can easily exceed a statistically acceptable 10% confidence interval.

To estimate the level and pattern of noise that may be involved in unbiased economic impact estimates, the author returned to the sample economic impact model shown in the figure above and applied errors that fell within the range of plus or minus 25% to each of the nine assumptions. The error terms were randomly applied to each parameter in 15,000 trials. The estimates are unbiased because the average error in the estimate of each of the parameters is zero. The figure below shows the distribution of the errors in the overall estimates of economic impact.

Figure 3 - Distribution of Random Errors (+ or – 25%) in 15,000 Trials

The error in estimates is not normally distributed. The degree of asymmetry or skewness [2] tends to be negative (approximately –1.4). That is, unbiased random errors in each of the nine parameters result in more underestimating than overestimating economic impact. Only 35% of the estimates fall within a reasonable range of plus or minus 10% of the actual hypothetical impact. At the extremes, approximately one out of ten estimates produce an error of plus or minus 50% or more.

While illustrative, the sample model described in this paper is simpler, with fewer parameters, than most impact models. Many impact models attempt to measure the impact of multiple types of visitation that, in addition to overnight visitation, may include day-trip visits, exhibitors, organizers, sponsors, and others that may bring new income into a destination. Multi-year forecasts also include assumptions about how visitation may change yearly and about inflation and changes in other economic factors. As this model demonstrates, as the number of parameters in the model increase, the potential error in the overall estimate of economic impact increases.

The spending approach to economic impact tests the limits of our ability to accurately measure the change in economic output caused by venues and events. The assumptions used in economic impact modeling are prone to error because of the difficulty of obtaining accurate information. Some common methodological shortcomings compound the errors resulting from incomplete data. Following is a discussion of five important limitations of traditional economic impact methodologies.

These limitations add up to a great deal of noise in economic impact estimates and call into question our ability to accurately measure the economic impact of events.

Improved methodologies and better use of big data can address some of these limitations and improve the accuracy of estimates. Forecasters need to focus on reducing the number of parameters used in their models and reducing noise in each parameter. Following are six recommendations for improving accuracy.

Figure 4 - Difference of Means Test Impact of SDCC on City of San Diego Hotel Sub-Markets

The results show that when the SDCC had an event, downtown occupancy increased by as much as 13% and by 6% or 7% in other sub-markets. The ADR increased by as much as $37.00 on convention center event days in downtown San Diego and as little as $3.00 in the least affected sub-market. A difference of means test also provides information on the reliability or level of confidence in the estimate. In this example, the results for all sub-markets had a less than 1% probability that these differences were caused by random variation in the data.

Using the difference of means tests, HVS has estimated the dollar amounts of impact convention centers have on hotel room revenues in several US cities. The results show how convention centers compress the hotel market and have a ripple effect of increasing room revenues throughout the entire market. This statistical method avoids three limitations of a traditional approach: 1) inaccurate counting of visitation, 2) uncertain estimates of consumer spending, and 3) failure to account for displacement of existing economic activity.

Figure 5 - Estimated Annual Visitation to the David L. Lawrence Convention Center

In addition to these estimates of total visitation Placer.ai generates estimates of the average length of visit (1.97 days), the dwell time (3 hours and 45 minutes), the origin of visitors (30% came directly from home), and the peak days for visits (Saturday and Sunday). Demographic data on visitors is also available, but analysts should recognize the estimates generated by these services involve extrapolation from a small panel to a larger population. In this case, the panel size was 22 thousand, or 4% of the number of estimated visitors. The reliability of these estimates depends on how representative the panel is of the total population. Analysts should not place blind trust in these sources. They should perform reasonableness checks on the estimates they provide.

The application of noise reduction techniques can exponentially improve the accuracy of economic impact estimates. Returning to the sample model shown in Figure 2, a reduction of random error in estimates of individual parameters from plus or minus 25% to plus or minus five or ten percent dramatically improves the accuracy of the impact estimate, as shown in the figure below.

The reduction in random error in the parameters improves accuracy and normalizes the distribution of error. The effects are dramatic.

Variations on the example model used in this study show that if noise reduction techniques reduce the random error in the estimate of parameters to 10%, 73% of economic impact estimates would fall within a reasonable range of plus or minus 10%. A smaller random error of 5% would produce reasonably accurate estimates 95% of the time.

Because they rely on these estimates to foster continued support of public sector investment in their projects, venue owners and destination marketing organizations need and deserve improved accuracy.

[1] Baade, R. A., & Matheson, V. A. (2001). Home run or wild pitch? Assessing the economic impact of major league baseball’s All-Star Game. Journal of Sports Economics, 2(4), 307-327.

[2] Skewness is a statistical measure of how much a distribution deviates from a perfectly symmetrical normal distribution.

[3] Evaluating Tourism’s Economic Effects: New and Old Approaches, Dwyer, Forsyth and Spurr, Tourism Management 25 (2004) 307-317.

[4] The difference of means analysis tests the null hypothesis that there is no significant difference between the means of two groups. Rejecting this null hypothesis indicates that differences in the means are probably not due to chance.

[5] Convention Center Expansion Analysis—San Diego Convention Center, HVS Convention, Sports & Entertainment Facilities Consulting, April 2017.

[6] Noise: A Flaw in Human Judgment, Kahneman, Sibony, and Sunstein, Little Brown Spark, 2021, pps.111-136.