03 August 2010

A Guest Post: No Fluid Dynamicist Kings in Flight-Test

[This is a guest post from Captain Joshua Stults,who is an aeronautical engineer with the US Air Force. Captain Stults is writing here in a personal capacity and his views do not reflect those of the USAF or Department of Defense (the image above was chosen by me). He blogs here. His guest post follows up on a discussion of my book, The Honest Broker, here and at Michael Tobis' blog. Such thoughtful contributions are one of the best things about running a blog. Thanks Captain Stults! -RP]

Dr Pielke's Honest Broker concepts resonate with me because of practical decision support experience I've had, and this post is an attempt to share some of those from a realm pretty far removed from the geosciences. All the views and opinions expressed are my own and in no way represent the position or policy of the US Air Force, Department of Defense or US Government. I am writing as a simple student of good decision making. My background is not climate science. I am an Aeronautical Engineer with a background in computational fluid dynamics, flight test and weapons development. I got interested in the discussions of climate policy because the intersection of computational physics and decision making under uncertainty is an interesting one no matter what the subject area. The discussion in this area is much more public than the ones I'm accustomed to, so it makes a great target of opportunity. The decision support concepts Dr Pielke discusses make so much sense to me now, but I can see how hard they are for technical folks to grasp because I used to be a very linear thinker when I was a young engineer right out of school.

My journeyman's education in decision support came when I got the chance to lead a small team doing Live Fire Test and Evaluation for the Air Force (you may not be familiar with LFT&E, it is a requirement that grew out of the Army gaming testing of the Bradley fighting vehicle in the 1980s, a situation that was fairly accurately lampooned in the movie "Pentagon Wars"). The competing values of the different stakeholders (folks appointed by congress to ensure sufficient realistic testing compared to folks at the service level doing product development) was really an eye-opening education for a technical nerd like me. I initially thought, "if only everyone can agree on the facts, the proper course of action will be clear". How naive I was! Thankfully, the very experienced fellows working for me didn't mind training up a rash, newly-minted, young Captain.

It's tough for some technical specialists (engineers/scientists) to recognize worthy objectives their field of study doesn't encompass. The reaction I see from the more technically oriented folks like Tobis (see how he struggles) reminds me a lot of the reaction that engineers in product development offices would have to the role of my little Live Fire office. A difficulty we often encountered was the LFT&E oversight folks wanted to accomplish testing that didn't have direct payoff to narrower product development goals that concerned the engineers. "What those people want to do is wasteful and stupid!" This parallels the recent sand berm example. The preferred explanation from the technician's perspective is that the other guy is bat-shit crazy, and his views should be ridiculed and de-legitimized. The truth is usually closer to the other guy having different objectives that aren't contained within the realm of the technician's expertise. In fact, the other person is probably being quite rational, given their priors, utility function and state of knowledge.

In my little Live Fire Office we had lots of discussion about what to call the role we did, and how to best explain it to the program managers. I wish I had heard of Dr Pielke's book back then, because "Honest Broker" would have been an apt description for much of the role. We acted as a broker between the folks in the Pentagon with the mandate from congress for sufficient, realistic testing, and the Air Force level program office with the mandate for product development.

The value we brought (as we saw it), was that we were separate from the direct program office chain of command (so we weren't advocates for their position), but we understood the technical details of the particular system, and we also understood the differing values of the folks in the Pentagon (which the folks in the program office loved to refuse to acknowledge as legitimate, sound familiar?). That position turns out to be a tough sell (program managers get offended if you seem to imply they are dishonest), so I can empathize with the virulent reaction Dr Pielke gets on applying the Honest Broker concepts to climate policy decision support. People love to take offense over their honor. That's a difficult snare to avoid while you try to make clear that, while there's nothing dishonest about advocacy, there remains significant value in honest brokering. Maybe Honest Broker wouldn't be the best title to assume though. The first reaction out of a tight-fisted program manager would likely be "I'm honest, why do I need you?"

One of the reason my little office existed was because of some "lessons learned" from the Tri-Service Standoff Missile debacle (all good things in defense acquisition must grow out of historical buffoonery). The broader Air Force leadership realized that it was counterproductive to have product development engineers and program managers constantly trying to de-legitimize the different values that the oversight stake-holders brought (the differences springing largely from different appetites for risk and priors for deception) by wrangling over largely inconsequential, technical nits (like tree rings in the Climate Wars). The wiser approach was to maintain an expertise whose sole job was to recognize and understand the legitimate concerns of the oversight folks and incorporate those into a decision that meets the service's constraints as quickly and efficiently as possible. Rather than wasting time arguing, product development folks could focus on product development.

The other area where I've seen this dynamic play out is in making flight test decisions. In that case though, the values of all the stake-holders tend to align more closely, so the separation between technical expertise and decision making is less contentious (Dr Pielke's Tornado analogy). In contrast to the climate realm where it's argued that science compels because we're in the Tornado mode, the flight-test engineers understand that the boss is taking personal responsibility for putting lives at risk based on their analysis. They tend to be respectful of their crucial, but limited, role in the broader risk management process. Computational fluid dynamics can't tell us if it's worth risking the life of an air crew to collect that flight test data. In that case there is no confusion about who is king, and over what questions the technical expert must "pass over in silence."

23 comments:

  1. Captain:
    Congratulations on a well written piece that takes us beyond the abstractions of Roger's taxonomy and out of the emoitional climate mire.
    Choosing labels for categories in a taxonomy is frquently fraught with unintended consequences. A perfect example is the old 2 x 2 BCG typology for businesses - dogs, cows, ?, stars.
    Perhaps you can illustrate from your experience how an office of your type simply doesn't become another stakeholder with goals and a perspective that limit your ability to ask the penetrating informed questions that help produce better overall decisions and/or alternatives? How much variation is there in the actual behavior and effectiveness of offices that are similar to yours in terms of mission for other weapon systems/platforms?

    ReplyDelete
  2. I hope most of us understand the tension that occurs when people with very different responsibilities view a process within the context of their different objectives. I'm not sure that the argument seen so often from climate scientists fits very neatly in this category.

    Yes, it is true that climate scientists can be focused too narrowly on their own field and fail to appreciate all the competing values and different estimates which inform the cost/benefit analysis that voters and lawmakers have to make. These scientists can resemble the old saw that all problems look like a nail to a man who has only a hammer.

    But the argument that the public should defer to climate scientists to make decisions for us has a much bigger complication which makes it an unlikely fit -- climate scientists want to be considered 'expert' without any accountability. They refuse to allow the public to see the data and code necessary to evaluate their work. They refuse to allow stats experts to look at their stats or software experts to review their code. And, of course, they don't bother to check each other's work. What climate scientists are really demanding is the right to render judgments which impact billions of people without any check, without any way for the public to protect itself. We are being told that we must simply accept their judgment.

    Given all the incompetence which has been exposed in the little bit of climate science work that has managed to receive some scrutiny, deference to these "experts" without some accountability would be insanity.

    ReplyDelete
  3. bernie: Perhaps you can illustrate from your experience how an office of your type simply doesn't become another stakeholder with goals and a perspective that limit your ability to ask the penetrating informed questions that help produce better overall decisions and/or alternatives?
    I think you are asking how we keep the mission ahead of more parochial interests (?). The best I can say is that it takes leadership (sorry if that seems like a cop-out, I don't know of a pat solution to that problem, let me know if you have any clues on that one).

    We certainly did have both goals and a perspective. Our goal was to get a decision about a sufficient test program that met the needs of all the stake-holders, and our perspective was based mainly around recognizing the need and usefulness of the Congressional LFT&E mandate for fielding "good" weapon systems on the one-hand and the limited time and budgets of product developers on the other. That perspective is fundamentally similar to the one underlying the policies Dr Pielke argues for (advocates in fact, see it's not a dirty word) in that it recognizes the legitimate concerns of the various decision makers.

    One of the things that mitigated against our becoming just another stake-holder was that we didn't have any decision making authority or "veto power". We also made it a point to split out funding that was to broker decision making and funding that supported a more traditional test support role, since it could be argued that we'd have an incentive to drive up the amount of testing to get more test-support bucks flowing in, and this would be contrary to the role of honest broker (we'd be just another stake-holder at that point). Test-support was purposefully kept as a marginal addition, the primary fixed costs (mostly professional labor costs) being covered by a "decision support" task. If you do one test or one-hundred we're happy (and fed) as long as all of the stake-holder's needs are met.

    bernie again: How much variation is there in the actual behavior and effectiveness of offices that are similar to yours in terms of mission for other weapon systems/platforms?
    The Air Force approach to the problem is unique so there is wide variation in actual behavior. The effectiveness for the office (based on the goal I mentioned above) is highly dependent on the personalities involved. Some product development managers "get it" and we get to a decision with only minor wailing and nashing of teeth. Others take a path very similar to the Climate Wars, and lots of effort is expended on arguing minor points for little gain. The other stake-holders tend to be involved in multiple such decisions over a longer time-frame. They are already familiar with the realities of the dance, so they are not such a big source of variation. The parallel here would be climate scientist newbies to the big bad world of decision support and public policy compared with seasoned old political operators that have seen issues come and go.

    Stan, I agree transparency is important, but at some point we have to recognize that no amount of transparency will "paper over" the fundamental differences, and that's OK.

    ReplyDelete
  4. I am not sure what you describe matches Roger's "Honest Broker" the way he derives it (twice!), but I understand the role you are describing.

    The problem is that everyone competent to judge climate science has come to the conclusion that the political system is horribly out of synch with the substantive evidence, and that we need to find a way to eliminate most or all net greenhouse gas emissions.

    The IPCC was assigned, agreed, and eventually mistrusted. Then the AGU and the AMS supported IPCC and were assumed part of the conspiracy; then all the national academies that matter, the APS, the ACS, the AAAS. So far pretty much everybody who looks says that the science is sound enough to require policy.

    Implicitly this means on any reasonable value system; the idea that all those organizations are authoritarian is absurd. Who is the king of NAS? The idea is just so alien to scientists as to seem silly.

    As for climate science, the evidence is pretty much in the same ballpark we expected it to be in 1979 when the first serious study was released, or about 1989, when the first serious policy push began.

    But now we are essentially out of competent reviewers!

    What we see is that everybody taking that role is won over by the evidence, and thereby tainted in the eyes of those who don't believe the evidence and prefer to see some vast and dark conspiracy to force freedom-loving people to have windmills.

    So although I firmly support skepticism, and testing, and openness, and tolerance of alternative values, and separation of science from values and all, I really don't see what practical insights you are offering.

    Just postulate for a moment that the world really was twenty years too late in taking this problem on, and looks to just have signed up for another five or ten years' delay. How would that situation look any different from what you see now? What behaviors would you propose to those who think their valid expertise is being ignored and that the problem is beginning to take on epic proportions?

    That's the advice we're looking for.

    Roger's advice seems to be to give up and hope somebody comes up with a magically cheap energy technology. I just don't see that approach in our repertoire of ethically tolerable choices.

    See, there's this tornado coming?

    ReplyDelete
  5. -3-Michael

    I can clearly see my framework through what is described in this post. I can also see your role well described.

    Since you keep going on about "two derivations" even after being corrected, I am beginning to wonder if you actually read my book ;-)

    When you say things like: "the political system is horribly out of synch with the substantive evidence" that is why you get tagged as an authoritarian.

    As far as my policy advice on climate change, you do not understand that either ;-) Perhaps you'll take a look at my book.

    ReplyDelete
  6. Michael Tobis

    Here are some sceptics of global warming science. Freeman Dyson, Ivar Giaever (Nobel Prize), Robert Laughlin (Nobel Prize), Edward Teller, Robert Jastrow, James Lovelock and William Nierenberg.

    Who are we to believe ? They, who are each vastly more intelligent, neutral and trustworthy than any alarmist climate scientist, or the climategate crew, who the world seems to have decided are guiltier than a planeload of OJ Simpsons ? (Apart from the British government of course).

    When I was involved in Guardian discussions, I almost never came across a scientist who wasn't a serious sceptic. That is, someone who was a credible and articulate witness with a good science education.


    James Lovelock wrote some extremely nasty things about the intelligence and integrity of modern climate scientists.


    http://www.guardian.co.uk/environment/blog/2010/mar/29/james-lovelock


    "The problem is that everyone competent to judge climate science has come to the conclusion that the political system is horribly out of synch with the substantive evidence, and that we need to find a way to eliminate most or all net greenhouse gas emissions."

    Every corporation and every government on earth supports global warming. What more do you want ? Unfortunately, the public don't (74% in the UK). That's the problem, isn't it ?

    ReplyDelete
  7. Dr Tobis:The problem is that everyone competent to judge climate science has come to the conclusion...
    One of the points of my example is that a stake-holder can have a legitimate claim on shaping the decision without any particular competence. In fact, (and I mean this in the most respectful way possible), the folks in an over-sight role were generally completely incompetent to judge the merit of design details on any particular system. This had no bearing on the fact that their needs had to be met for a decision to be made. Failure to grasp this simple fact is the cause of plenty of misplaced effort in Air Force weapons development, and is the source of lots of heat and precious little light emerging from online climate policy discussions. To me, the parallel is striking.

    ReplyDelete
  8. Michael-

    you said "Implicitly this means on any reasonable value system; the idea that all those organizations are authoritarian is absurd. Who is the king of NAS? The idea is just so alien to scientists as to seem silly."

    To me, the National Academy and the NRC are part of the Science Establishment. They decide what facts are really facts and which ones are not, currently without a transparent, open process, if we are talking about PNAS and NRC reports. It is more of a royal priesthood than a kingship- perhaps authoritarian is the wrong word.

    However, I have long thought of my visits to their building as pilgrimages to the Temple of Science.

    ReplyDelete
  9. Some people think about climate-scientist conspiracies. With the internet up and running, you can find folks who worry about anything. And conspiracies have always been grist for the mill.

    But, to my knowledge, most scientifically-literate people who have lost much trust in climate scientists--I am one--are not thinking that way.

    The space shuttle Challenger didn't fail because of a conspriracy among engineers (or climatologists). The problems were subtler, though the consequences were severe.

    Perhaps climate scientists who see themselves as unfairly marginalized as conspirators could interact with a broader community. Collide-a-scape seems to be a place du jour with pretty good conversations; I haven't seen the C word used there in recent times.

    ReplyDelete
  10. AMac

    Climategate was a series of conspiracies. Intelligent observers weren't surprised they were happening. Realclimate pretends to be a forum for the discussion of science when it was actually set up to defend the hockey stick (conspiracy pending). Why does James Hansen's junior, Gavin Schmidt spend a great deal of time in partisan activity on the internet. It's outrageous behaviour for a 'scientist'

    In general, most people in jail are there because a jury, who had no direct experience of their crime, decided they were involved in a conspiracy on the basis of presented evidence.

    The biggest argument against laissez faire capitalism is the existence of anti trust laws, designed to prevent various types of corporate conspiracies.


    "If you don't agree with them, they call it a conspiracy". James Kelman, Booker Prize winning novelist.

    Conspiracy theory for dummies.

    Wikileaks dude Julian Assange is the new batman. He was being hunted down by the FBI, so hid in London surrounded by the world's media. Very cool ! The NYT published 90,000 unsigned US military electronic files against the wishes of the US government. Wikileaks co-founder John Young thinks Assange is a a fraud. No kidding.

    Anyone who plays these games risks getting caught and facing the rest of his life in jail. Time to start working for the man. Same with the drugs business. See the ingenious novel 'A Scanner Darkly' by Philip K Dick where a drug dealer is assigned to investigate himself. Now a film with K Reeves.

    ReplyDelete
  11. Regardless of who trusts who, Copenhagen didn't fail because anyone believed any skeptics. That idea is absurd. It failed because it is politically impossible to enact global cuts when China and the other Bric countries have not signed on and when there is no coherent plan of how to do it without doing more harm than good.

    Meantime the EU, individual members of the EU and individual states of the USA have decided to impose their own cuts despite having no idea how and no foreseeable chance of success, except perhaps to just wait for all the lights to go out. At that point I guess we'll have succeeded in Michael's ambitions. The question will be if that certain humanitarian disaster is less than the merely plausible climate disaster that he is so concerned about.

    Never mind skeptics and scientists, what we need is more realists.

    ReplyDelete
  12. I personally find systems engineering issues like this to be fascinating - and this guest post in the best tradition of books like The Mythical Man Month, How Doctors Think, and Think Twice. Thanks - I think you'll have another regular reader.

    ReplyDelete
  13. Michael Tobis,

    "Roger's advice seems to be to give up and hope somebody comes up with a magically cheap energy technology."

    I know little about climate science, and probably less about political science. I know more then most about security.

    A contented population will require 3-6 police per 1,000 residents in order to maintain civil order. A not contented population will require 20 police per 1,000 residents.

    The primary purpose of government is to maintain civil order. Whether or not government policies are popular has an enormous impact on the cost of maintaining civil order.

    Any government that takes it's primary role of maintaining civil order seriously will consult someone with Roger's expertise on every single policy decision.

    ReplyDelete
  14. -4- Michael Tobis,

    Your comment, "...everyone competent to judge climate science has come to the conclusion..." demonstrated, I think, that climate science and its relation to policy is very similar to economics, and its relation to policy. In particular, I'm reminded of the White House press secretary saying:

    " I don’t think there’s an economist that believes there’s a stimulative effect to -- or a good reason in terms of economic growth to extend those tax cuts, particularly given the choice that one has to make about the budget deficit."

    Anyone aware of the spectrum of thought on the subject has to laugh at Gibbs' or Tobis' statements. There are experts on either side of the issue, and even those who don't care about the data or the theory, but think that certain policies are nevertheless good ideas.

    ReplyDelete
  15. Michael Tobis wrote -- "everyone competent to judge climate science"

    Would that be those who never check the accuracy of their instruments? Those who never check the accuracy of their data?

    If no one makes their data and code available and no one replicates anyone else's work, how can competence even be determined? If the quality of scientists' work is never examined, how can you know they are competent?

    The public has gotten enough of a peek behind the curtain to know that the IPCC, Mann, Jones, Briffa, and Rahmstorf have put out some extremely incompetent work. If you include them in your roster of competents, you just lost the argument.

    ReplyDelete
  16. "all the national academies that matter, the APS, the ACS, the AAAS. So far pretty much everybody who looks says that the science is sound enough to require policy."

    These societies exist to sustain themselves. People like to be in clubs. There's no particular tendency towards altruism. Every time I get an invitation from the ACS, I pitch it. I figure they just need more dues. If I really want to look at the publications, there are always a few copies around gathering dust before getting sent to the recycling can. Read through their goals and you will see these kinds of statements: "financially sustainable", "premier advocacy organization", "creating and supporting implementation of public policy statements". I have no problem with their education and research functions, but when your goal is to create public policy statements, that's what you will do, whether needed or not.

    ReplyDelete
  17. Re Michael Tobis:

    "everyone competent to judge climate science has come to the conclusion that the political system is horribly out of synch with the substantive evidence, and that we need to find a way to eliminate most or all net greenhouse gas emissions."

    So then, Lindzen, Christy, Abusamov, Tennekes, Balliunis, Soon, Choi etc, etc are not only 1) wrong about climate sensitivity but 2) Not competent to judge climate science. How in the world can you claim not be an "authoritarian"?

    "thereby tainted in the eyes of those who don't believe the evidence and prefer to see some vast and dark conspiracy to force freedom-loving people to have windmills."

    Sigh, yes all skeptics or anybody who dare have doubts about the CERTAINTY of the MODELED huge net positive feedbacks, or question the ACCURACY of the data on which they are based must be a conspiracy theorist or simply duped by big oil PR.

    Further, and owing to one of your previous posts, neither the "incompetent" scientists nor the "conspiracy theorists" should get to vote on policy.

    ReplyDelete
  18. Stults' piece does not speak to me, but hopefully it opens the eyes of a few more engineers.

    Over at "In it for the gold", Michael Tobis calls for the education of people who disagree and, if that does not help, their marginalization from power. Chilling.

    ReplyDelete
  19. Richard Tol refers to this exchange. Decide for yourself if he has provided a fair summary:

    ===
    Richard: "there is a small group of people who think that climate change is Armageddon, and who are burning is much coal as they can hoping to see the Second Coming during their earthly life time"

    me: "The thing to do about small groups of crazy, evil people is to try to make sure they stay small and try to convince them to be less crazy or evil."

    Richard: straight back to authoritarianism

    Me: So, any attempt to convince any member of the public of anything is authoritarianism?

    Richard: Yes, Michael, you disrespect other people's opinions, and you want to make sure that they are powerless. You returned to this position time and again during this discussion, so I'll just have to assume that you mean it.
    ===

    Apparently I am not allowed to advocate disagreement with people who want to hasten the end of the world, on the grounds that it is authoritarian.

    It's at http://initforthegold.blogspot.com/2010/08/and-in-end_01.html

    ReplyDelete
  20. Michael Tobis

    "Apparently I am not allowed to advocate disagreement with people who want to hasten the end of the world, on the grounds that it is authoritarian."

    Do you really believe that the end of the world is coming if President Barack Obama cannot get a climate deal because of the bad Republicans ?
    Are you really frightened, the way you would be if your car was heading toward a wall ?

    I see the fundamental division being between environmentalists and scientists (physicists). Physicists don't seem to be too keen on global warming . Look at the nasty, underhand goings on at the APS and IOP (UK) to keep honest men quiet. As I said before, almost all physicists on the Guardian forums were serious sceptics.


    Jasper Kirkby is a physicist . He observes a close correllation between cosmic rays and global temperatures.

    Results from his experiments at CERN are expected later this year.

    Video

    http://cdsweb.cern.ch/record/1181073


    P.S. I know environmental physicists can be termed climate scientists.


    Here is one that states the blindingly obvious about climate advocacy, sorry science.


    Petr Chylek


    The fact that the Atmosphere Ocean General Circulation Models are not able to explain the post-1970 temperature increase by natural forcing was interpreted as proof that it was caused by humans. It is more logical to admit that the models are not yet good enough to capture natural climate variability (how much or how little do we understand aerosol and clouds,and ocean circulation?), even though we can all agree that part of theobserved post-1970 warming is due to the increase of atmospheric CO2 concentration.

    Thus, two of the three pillars of the global warming and carbon dioxide paradigm are open to reinvestigation.The damage has been done. The public trust in climate science has been eroded. At least a part of the IPCC 2007 report has been put in question. We cannot blame it on a few irresponsible individuals. The entire esteemed climate research community has to take responsibility.


    Los Alamos National Laboratory

    http://www.thegwpf.org/opinion-pros-a-cons/218-petr-chylek-open-letter-to-the-climate-research-community.html

    ReplyDelete
  21. "Do you really believe that the end of the world is coming if President Barack Obama cannot get a climate deal because of the bad Republicans ?"

    Uh, no.

    I suggest you read more carefully.

    ReplyDelete
  22. Sorry Michael.

    I had assumed that using a phrase like "hasten the end of the world', you had an apocalyptic mindset like many of the crazies at RealClimate. It must be woven into the American psyche.

    My bad.

    ReplyDelete
  23. Roger,

    This discussion has applications to your discussion of the honest broker, Tobis' dsire for rule by experts, and the political nature of the climate debate. http://www.professorbainbridge.com/professorbainbridgecom/2010/08/the-basic-difference-between-liberals-and-conservatives.html

    ReplyDelete