I attended the Transportation Research Board Annual Meeting in Washington, DC in 2024. I'm about two months late, but following is a summary of my "aha" moment during the conference. Because after all, if I didn't have some kind of vaguely profound-to-me series of thoughts at a conference, then was I really there?
This year, my musings were on aviation forecasts. There were two major triggers for my musings this year which happened in quick succession on Tuesday. First was the Aviation Economics and Forecasting Committee (AV040) meeting; immediately afterward was a session sponsored by the Airport Terminals and Ground Access Committee (AV050) entitled, "Persistent Impacts of the COVID-19 Pandemic on Passenger Processing".
The AV050 session was introduced with some background on passenger statistics before and after the COVID-19 pandemic. The moderator consistently framed the discussion of the statistics as still not quite having "returned" to pre-pandemic levels. This discussion was not unique to this session; it is frequently cited as a benchmark when setting the stage for a range of recent changes in aviation. But why, I wondered, is a return to 2019 demand levels seen as a goal that we want to reach again? Why is that the light at the end of the tunnel before things are back to "normal" and we can continue on our forecast growth trajectories?
Aviation forecasts always show perpetual growth for any forward-looking interval examined, even out to twenty years into the future. It is a reflection of the capitalistic mindset: growth forever. But this is not sustainable. Aviation demand cannot grow forever. When, I wondered, will growth simply stop--not caused by exogenous factors, but simply because the market is saturated and the demand for air travel is satiated, or an "equilibrium" of sorts is reached?
I thought back to the AV040 committee meeting immediately prior. The last third of the session consisted of a "roundtable" discussion on forecasting methodologies, especially into the future. Four speakers scattered around the room were asked questions, and despite their detailed answers to questions and the differing systematic approaches that they apply to their forecasting practices, I was reminded that forecasts are always wrong, regardless of the systematic approach used.
Forecasts are always wrong. They are always wrong because they are econometric regression models that only account for a handful of explanatory variables--perhaps six or seven at most. Explanatory variables are chosen that are assumed to have a relationship with aviation demand and result in regression models with reasonably high R2 values. Common variables like economic output and population tend to show up in many forecasts, but these variables have a correlative relationship, not a causal relationship.
Irrespective of the variables chosen, an R2 of 1 is never achieved. That is because the model would need to include an infinite data set and an infinite number of explanatory variables. However, the models used are never robust enough to predict events like the Great Recession or the COVID-19 pandemic. Or, conversely, many forecast models prepared around 2014 to 2016 underperformed in their near-term projections through 2019. Even though the model's R2 value may have been over 0.9, that remaining 10% of explanatory variability can hold a massive sway on how demand actually materializes.
Therefore, there must exist some other variable or set of variables that, if they could be included in the forecast models, should be able to predict massive downturns like the Great Recession or the COVID-19 pandemic. I call these "dark variables", analogous to the dark matter in the universe that we cannot observe but nevertheless accounts for 75% of the total mass in the universe.
Using AI in preparing forecasts briefly came up during the committee meeting. Frankly, the panelists were extremely short-sighted in their views on how AI will influence aviation forecasting. They asserted that AI can be used as a tool in forecasting, but it will never replace the "human element" of aviation forecasting. I don't believe that for a second. This is exactly what AI is good at: identifying patterns in data, often ones that most humans would otherwise overlook. And given that much of the work of preparing a forecast is setting up a regression model, this is a task that almost certainly will be trivially taken over by AI. AI may even have the ability to generate more accurate forecasts, as it may pick up explanatory variables that most forecasters would not consider including. To me, forecasting seems like a practice which is perhaps among the most vulnerable to being displaced by AI. Aviation forecasters would be wise to start getting savvy on other areas.
There were a number of times early in my career when I was asked to pull historical schedule data for an airport. I was also instructed to exclude certain periods of "non-representative" demand patterns--usually the Great Recession. I wondered how much insight was left on the table by excluding these periods, which surely contain valuable information about factors causing declines in aviation activity and how markets behave under such circumstances. I wondered how much data was thrown out unnecessarily. AI, by contrast, benefits with more input data, so this practice of excluding non-representative data must be discarded.
I have reviewed several papers for TRB over the last few years. Many of these papers develop regression models for measuring whatever phenomenon the authors are measuring. The analyses of the resultant regression coefficients discuss the impacts of one variable on the measured quantity. But a fundamental assumption for a regression model is that the explanatory variables are independent of each other. In almost all cases that I have seen, the explanatory variables are not independent, which poses a challenge to the model's validity and its predictive ability. Furthermore, the models often rely on linear multivariate regressions, when other forms of regression (exponential, power) exist. Of course, the problem with any such model is that it presumes that all the terms are subject to the same mathematical operators: addition, exponentiation, or multiplication. What if forecasters or AI were able to develop mathematical relationships for aviation forecasting that combined mathematical operations and was able to correct for dependent variables?
So, after all that, my takeaway was that no one actually knows what they are doing with respect to aviation forecasting. It's all perfunctory and dressed up in a gold sheen of mathematical rigor.
Then what does it mean for the FAA to review and approve a forecast? For airport master plans, the forecast is one of two items that the FAA has approval authority over (the other being the Airport Layout Plan set. Contrary to popular belief, the FAA does not have approval authority of the master plan as a whole.) If even the forecasting experts who prepare the airports' forecasts cannot prepare anything truly defensible, what endows the FAA with the ability and secret knowledge such that they can judge whether a forecast is appropriate?
When the FAA reviews a forecast, they often come back with comments as to why the approach or method was taken over a different one. If the sponsor and consultant agree, or if they just want to get the forecasts done so they can continue the rest of the master planning process, then they may change the approach or method at the FAA's suggestion. But who is to say that the second approach or method is more accurate or valid than the first?
Hint: they're both wrong. Once again, it is cloaked behind a gold sheen of mathematical rigor.
So, we are back to the question: what does it mean for the FAA to approve a forecast, especially if no one can actually get a forecast right?
An FAA-approved aviation forecast represents a level of demand that the FAA is comfortable having inform an airport's facility requirements and, ultimately, its capital improvement plan. All projects in the capital improvement plan need to be supported by an FAA-approved demand basis, i.e., the forecast. So, perhaps the FAA's approval of a forecast should be thought of as a pre-approval of a capital improvement plan. It is a statement of what the federal government is willing to pay for.
And, fundamentally, this is a policy stance. It is similar to the government's historical push for other transportation projects such as, "a freeway must go here". It is a policy stance in that it is an indirect for the federal government to commit federal money to airport capital projects. Of course, these can create jobs, so there is a positive there. However, that may skew the incentive for the FAA to want to always see positive growth in airport forecasts.
Now, who is holding the federal government accountable for this policy decision? What occurred to me was this insane idea: should the FAA's approval of a forecast be classified as a federal action under NEPA requiring review?
I talked about this with a colleague who has worked in aviation environmental for over fifteen years. Her immediate answer was no, because the forecast approval itself does not have effects in the impact categories as outlined in NEPA. The NEPA review occurs at the project stage, when the impact categories can be measured for a specific project or set of projects in question.
However, by that point, the forecasts are considered set in stone. Many projects' justifications rely on the forecast growth in demand, and the need to deliver these projects to meet this forecast growth in demand. So the NEPA process creates accountability for the project itself, but not for the demand assumptions underlying the project.
In short, forecast review and approval by the FAA is a sham. It is subject to the whims of the local ADO or the FAA headquarters and is likely reflective of what capital projects they are willing to help pay for. It thus constitutes a policy decision, and there is no real accountability for this decision.
So, those were my profound-to-me musings at TRB 2024. I'm not really sure where this goes, if anywhere, but it was certainly a fun thought trajectory.
No comments:
Post a Comment