Superforecasting the Art and Science of Prediction Pdf Download

Let me say from the starting time that this is an first-class book to read. Information technology is not only informative, as information technology should exist for a book on forecasting, but information technology is highly entertaining. Information technology provides a review of some of the worst forecasts in history, and also gives a fairly personal account of a number of "ordinary" individuals that Philip Tetlock got to know rather well during various forecasting tournaments and research projects that he was involved in 1. I highly recommend the book not but to practitioners who are dealing with forecasting problems on a regular basis, or academics that are interested in more technical, research questions related scoring rules, the evaluation of forecasts, and the similar, but also to a much wider audience. In fact, the book's opening judgement "We are all forecasters" sums it up nicely that each and everyone of the states, either formally or informally, consciously or subconsciously, is engaged in (some sort of) forecasting on a daily basis. The topic of the volume is thus timely and relevant for everyone. I particularly liked the adopted writing style of the 2 authors, Philip Tetlock and Dan Gardner. The book is structured coherently. Moreover, information technology is written similar a novel (rather than a textbook) in the sense that it gets the reader hooked by telling the story of the book in an unfolding fashion, with glimpses of the results of Tetlock's most contempo forecasting projection, the Adept Judgment Projection (GJP), revealed slowly and progressively in stages throughout the book. Needless to say, the combination of a Professor of psychology and a journalist every bit authors of the book, indeed make it hard to put the book downwards one time you go started.

And so what is the volume about? Well, it is virtually forecasting, only in that location are many such books 2. What makes the book dissimilar from most other standard treatments on forecasting is that it gives a detailed account of the forecasting operation of a big number of "ordinary" individuals that volunteered to take office in diverse forecasting competitions that Tetlock has organized since about 1984. These "ordinary" individuals, forecasting laymen then to speak, would compete with professional, authorities as well every bit bookish, forecasters on a variety of geopolitical and also economic topics of interest iii. Moreover, the book describes the reasoning and the idea process that these so called "Superforecasters" employ to arrive at a probabilistic prediction of the result of interest.

What are "Superforecasters"? Tetlock and Gardner describe them as ordinary, everyday individuals, that are logical thinkers with an eagerness to "share their underused talents" [i]. Superforecasters have a cracking interest in understanding the mechanism behind things, every bit opposed to just getting accurate forecasts without knowing why. They question their line of reasoning. They are willing to test new ideas. They never settle on one and only 1 forecasting machinery, just rather remain in a "perpetual beta" mode, realizing that life, as well as the world around them, is complex, that nothing is for certain, that everything is constantly evolving. Moreover, they are well enlightened of the fallacies and tricks that the heed can play on us (humans) to "brand something wait certain", leading to the temptation to come up to the "obvious" conclusion too rapidly, without second guessing it, and re-analyzing the evidence that is presented over and over again.

A practiced part of the volume deals with outlining the typical graphic symbol traits that Superforecasters take. In this context, Tetlock and Gardner attempt to emphasize and convince the reader that "forecasting is not a 'yous have it or you lot don't' talent", but that Superforecasting can be learned. Role of the book's purpose is thus to illustrate to the reader how to become a Superforecaster. On page 191, under the heading of "Pulling it all Together", Tetlock and Gardner list some specific key characteristics that a typical (modal) Superforecaster possesses, using the following four headline points to group them:

(1)

philosophic outlook,

(2)

abilities and thinking style,

(3)

method of forecasting, and

(iv)

work ethic.

In that location are a number of themes in the book that I particularly liked and found extremely relevant, but allow me highlight only a few here. I accept an econometric or model based forecasting background. Although the type of forecasting that is described in this volume is quite different, as it is based on subjective or judgmental forecasting, I see many parallels between the key characteristics that Superforecasters possess when amalgam judgmental forecasts and those utilized by Econometricians, when building statistical prediction models. For case, Tetlock and Gardner talk about "Foxes" existence better forecasters than "Hedgehogs", with "foxes knowing many little things, while the hedgehog knows one large thing." Since conditions in the real earth are rather complex and constantly changing, Tetlock and Gardner find in their research that it is ameliorate to motion away from an "I know 1 big thing" approach to forecasting, and rather adopt an "I know many little things" approach. In an econometric (model based) forecasting world, this means that one needs to move away from a single model (or predictor) framework, to i that utilizes instead a potentially large number of (small) models, which are then combined or averaged to produce a unmarried forecast of the target variable of interest. Such a modelling strategy is adequately mutual today (encounter for example [3], for an application of this approach in the equity premium forecasting literature, or [4,5] in the context of exchange charge per unit forecasting and equilibrium credit modelling).

Although the gains from a "diversified" prediction framework, where the average of the predictions from many individual forecasts (i.e., "the wisdom of the crowds") is used, seem obvious, Tetlock and Gardner talk over at length the primary obstacles they faced when forming "Superteams", that is, teams of Superforecaster to implement such a strategy in the GJP forecast contest (see Affiliate 9 and besides the discussion that follows). There were 2 main concerns. The first was "groupthink", that is, everyone in the group converging to the "average views" of the grouping, without questioning the logic or motivation behind the arguments to come to that determination, or the sources of information used to class the prediction. The 2nd was rancor and dysfunction. The overall conclusion from this chapter is that consensus is non ever good, and disagreement not always bad. Moreover, one should focus on constructive confrontation. What is interesting to indicate out here is that a similar loss in the forecast gains from model averaging can exist encountered in an econometric forecast combination framework, if the individual forecasts that make upward the aggregate are also like.

An illustrative visual case that highlights this issue is seen from Figure i below, which shows forecasts of the U.s. unemployment rate 12 months into the future from the Survey of Professional Forecasters (thin calorie-free bluish lines), together with the actual realized unemployment rate (thick nighttime ruby-red line). There are a number of features that are intriguing from this plot, but let me talk over just a few. First, notation that the overall predictions from the professional forecasters have the same trajectory, that is, turning points in the series, despite showing some variation in them. The predictions are dispersed, so there is variability in the forecasts of the individuals, merely the turning points no 1 seems to be able to predict. This is particularly evident effectually 2001 and 2008. 2nd, the forecasts lag by most one year (4 quarters) behind. This suggests that all professional person forecasters seem to employ, to some extent at least, the aforementioned type of dynamic econometric model to generate the forecasts. 3rd, all prediction models that are used by professional forecasters seem to be based on the thought of mean reversion, where the series today is constructed every bit a weighted average of the (unconditional) mean of the serial and the deviation of the serial from its mean. If the hateful of the series was higher in the recent past, the prediction from the model volition tend to revert back to this higher mean. This property of the model based forecasts is clearly and systematically visible from Figure 1. For instance, at the kickoff of the sample, the mean was between 6 and 7. Over the menses from 1995 to 2001, nearly all forecasts fabricated are systematically to a higher place the realized value. Similarly, betwixt 1999 to 2001, the mean had dropped, resulting in systematic under-predictions of the unemployment rate from 2001 to 2004. This design continues. Combining forecasts from such similar individual predictions is not going to lead to a great bargain of improvement, and tin hence be viewed as the econometric or model based analogue to "groupthink".

Since this is a critical review, I should say a few words about what I did not like so much in the volume. All assessments of forecast accuracy and comparisons across forecasters in the book (and the GJP as a whole) are based on the Brier Score (BS) 4. Evidently, when wanting to assess how well forecasters are doing, one will demand to settle on a functioning measure. The Brier Score is a proper score function designed to measure the accuracy of probabilistic forecasts. It is thus perfectly suitable for the type of predictions that are considered in the volume and the GJP. The Brier Score is formally divers as:

$$\mathrm{BS}=2\left[\frac{1}{N}\sum _{i=1}^N{({\widehat{p}}_i-y_i)}^{\mathrm{two}}\right],$$

(i)

where Due north denotes the number of predictions that are made,

${\widehat{p}}_i$

is the predicted probability of event i, and

$y_i$

is the realisation or outcome of the effect, where

$y_i$

takes the value of 1 if the upshot occurred, and 0 if it did not 5. A BS of 0 means that the forecaster is doing really well, in fact, as well equally is possible, correctly predicting the binary outcome every time, while a BS of 2 ways the forecaster is getting it wrong every time. A BS of 0.5 is the score you lot would get from random guessing (the dart-throwing chimp metaphor in Tetlock and Gardner), or alternatively, assigning a

$50\%$

probability to every event that is forecasted.

So what is the problem here? Because the Brier Score uses a quadratic loss function, the penalisation from getting forecasts very incorrect increases not-linearly. With getting forecasts very incorrect I mean the scenario of assigning a predicted probability

${\widehat{p}}_i$

of 1 (or close to one) for an event to occur, but the realised value

$y_i$

is 0 (the consequence did not occur). Or conversely, my prediction being the event is incommunicable to occur (i.e.,

${\widehat{p}}_i=0$

or close to 0), nevertheless it occurs. These are grave forecasting errors and there is nothing wrong with penalizing them harshly. It makes sense to do that from a utility based motivation as well. However, if I am aware of this every bit a forecaster (and I am sure many forecasters are), I will be more inclined to provide weak predictive signals, in the sense that I will accept a preference for my forecasts to be close to the

$50\%$

uninformative prediction most of the fourth dimension, unless I feel extremely confident. This is exactly the type of forecast that nobody wants to have. Recollect of the predictions that the President's advisors were providing regarding the compound in Pakistan, discussed at length in Chapter vi of the book under the heading "The tertiary setting". On folio 135 it says: "... the median estimate of the CIA officers — the "wisdom of the crowd" — was effectually $70\%$. And however Obama declares the reality to be "fifty-fifty". Thus, the apply of scoring rules that encourage weak predictions provide weak information for policy relevant actions. In an econometric or model based prediction setting, obtaining forecasted probabilities that are nearly ever close to

$\mathrm{l}\%$

would raise doubts about the informativeness of the model, or, to put it in econometrics jargon, the goodness of fit of the model to the data. Moreover, the predictor variables that get into the model, i.eastward., the workout variables used to generate the forecast, would be considered equally irrelevant for the issue variable.

Another outcome with how BS is used throughout the book is that only point estimates are provided, without any mention of the dubiousness surrounding these estimates. Because the outcomes

$y_i$

are realisations of a random variable, the Brier Score in (ane), which depends on

$y_i$

, will also be a random variable. Making statements of the form: "their collective Brier score was 0.25, compared with 0.37 for all other forecasters" on page 94 is uninformative, as at that place is no measure of variation reported in the deviation between the ii groups' Brier scores 6. From a statistical point of view, one is inclined to ask: "Is this difference in BS significant?" or "What are the confidence intervals of the 2 scores?" In theory, these can exist computed quite easily; the authors of [half-dozen], in fact, derive standard errors and confidence intervals for the Brier Score and Bramble Skill Score. However, one crucial assumption in the derivation (see the motion from line 2 to iii in equation 16 on page 994 in [6]) is that the pair of predicted probabilities and outcomes of events are random, i.e., independent across i (the events to be predicted). In general, this may non seem like a strong assumption to make. But in the given context, it would depend on the type of forecast questions that are existence asked and would crave us to recall about questions like "What is the likelihood of two (or more than) of the predicted events occurring jointly?" If the set of questions to be answered in the forecast competition include: "Will the Kurdistan Regional Government hold a referendum on national independence this year?" and "Will either the Swiss or French inquiries discover elevated levels of polonium in the remains of Yasser Arafat's body?", and then information technology probably is easy to argue that these two events tin can be regarded every bit independent of each other. However, consider the case where the set up of questions includes: "Is Britain going to leave the European Union?" and "Is the Pound Sterling going to depreciate past

$\mathrm{xx}\%$

against the Euro over the next iii months?", or "Are borrowing costs in Britain going to increment over the next six months", and the like. Then, it becomes much harder to justify the supposition of independence of the events being predicted. Computing the variance of BS is thus more complicated in practice.

As Nelly Furtado said (sang actually) and many people earlier her, "All expert things come to an end". So permit me conclude this review with the words that the book is a nifty read, very informative, entertaining, and, above all, provides the reader with a summary of the experiences of forecasters in a life long project that Tetlock has been involved in since about 1984.

Conflicts of Interest

The author declares no disharmonize of interest.

References

1. P. Tetlock, and D. Gardner. Superforecasting: The Fine art and Science of Prediction. New York, NY, USA: Crown, 2015. [Google Scholar]
2. G. Elliott, and A. Timmermann. Economic Forecasting. Princeton, NJ, Usa: Princeton University Press, 2016. [Google Scholar]
3. D.E. Rapach, J.K. Strauss, and G. Zhou. "Out-of-Sample Disinterestedness Premium Prediction: Combination Forecasts and Links to the Real Economy." Rev. Financ. Stud. 23 (2010): 821–862. [Google Scholar] [CrossRef]
4. D. Buncic, and G.D. Piras. "Heterogenous Agents, the Financial Crunch and Exchange Rate Predictability." J. Int. Money Financ. 60 (2016): 313–359. [Google Scholar] [CrossRef]
5. D. Buncic, and M. Melecky. "Equilibrium credit: The reference point for macroprudential supervisors." J. Bank. Financ. 41 (2014): 135–154. [Google Scholar] [CrossRef]
6. A.A. Bradley, Southward.S. Schwartz, and T. Hashino. "Sampling Uncertainty and Conviction Intervals for the Brier Score and Bramble Skill Score." Weather Forecast. 23 (2008): 992–1006. [Google Scholar] [CrossRef]

• ^1. On the commencement page of the book, Nib Flack, a retired regime employee of the US Department of Agriculture in Arizona is introduced, who took part in Tetlock's forecasting tournaments and who has achieved the condition of a "Superforecaster".

• ^2. The most contempo book in an economic or econometric setting is by [2].

• ^three. The types of questions forecasters would have to reply were questions similar: "Volition Bharat or Brazil get a permanent member of the United nations Security Council in the next 2 years?" or "Will Democratic people's republic of korea detonate a nuclear device before the end of this year? and the like. Participants (forecasters) would have to give a probabilistic forecast, that is, the likelihood of the upshot occurring.

• ^4. BS seems to exist the usually used abbreviation for the Brier Score (see for case https://en.wikipedia.org/wiki/Brier_score). I am thus using it hither in accordance with standard practice in the literature, rather than as an expression of contempt or dislike with the measure.

• ^5. Note that $y_i$ is thus a realisation from a Bernoulli random variable $Y_i$, with success probability $p_i$. Also, Tetlock and Gardner use the original scaled (multiplied past ii) version of BS (they define the lower and upper bounds of BS to be 0 and ii on page 93).

• ^six. I should say here in defence of the authors that, commencement, they are probably very well aware of this, given their background, and 2nd, that they do study results of the Bramble Score from various years of the same Superforecaster being ranked, which gives more acceptance to the difference not just being due to chance. Notwithstanding, the issue still remains.

© 2016 by the author; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and atmospheric condition of the Creative Commons Attribution (CC-Past) license (http://creativecommons.org/licenses/past/4.0/).

noltebutersomprom.blogspot.com

Source: https://www.mdpi.com/2227-9091/4/3/24/htm

Share this post