I am picking up on an earlier discussion of skill in prediction (
Part I is here). Above is a dataset. What it is does not matter, it could be wheat prices on the Chicago exchange, it could be the stock price of BP in recent weeks or it could be the number of hits on this blog. The reason that the dataset does not matter is that I would like you to suggest a forecast -- a naive forecast. The less you know the more naive your forecast of what the next few dozen point might be in the dataset.
So in the comments please suggest a forecast of what will come next, and please explain the basis for your naive forecast. To aid the more quantitatively inclined, I will reproduce the data in the first comment below. After I get some suggestions for a naive forecast, I'll move to the next step.
45 comments:
119.4
99.1
89.9
99.6
96.7
88.5
92
80.6
83.4
78.9
70.8
74.8
100
91.3
65.2
115.4
64
92.5
76.1
82.9
101.7
95.9
109.8
125
97.1
136.3
108.8
100
112.1
111.4
133.4
101.9
118.6
122.1
126.3
120.5
115.7
134.7
106.7
115.7
148.4
136.3
121.2
132.9
157
151.5
149.6
145.2
129.8
162.2
105
137.1
Here is my naïve data. My data would change depending on it mean, but they would remain the same regardless of the three suggestion above.
In each of these scenario the value should go up in the future at about the same rate.
So my forecast would add these values:
151
164
147
158
167
170
160
173
164
168
176
180
Let's try a value of approximately 160 at point 101. It's worse than we thought.
Numbers are alternating up and down 75% of the time. The mean is 110.21 and one Stdev = 24.48. I'll forecast the next number will go down. If you want a number, my WAG is 113.
Oops.
That 75% alternating should be 65%.
I shall cheat and make three forecasts.
(1) The WUWT forecast. The use of 52 datapoints is highly suggestive of 52 weeks in the year. The data is obviously cyclic, and my best prediction is that the cycle will repeat. Tack a copy of the dataset on the end and calculate a five point moving average.
(2) The RC forecast. The data is well described by a linear fit y=1.215x+78 and will continue rising without limit. It will soon be worse than we thought.
(3) The Blackboard forecast. The data is indistinguishable from a random walk, so the best prediction for future values is the current value of 137.1 for ever.
It will roughly repeat. There are 52 weeks in a year.
Without any great statistical knowledge (although I like playing with numbers, and have a background in quantitative geosciences where some number crunching is required), I come up with two very simplistic alternatives using excel charts (projecting forward to x=65):
1) Linear regression (r2 = 0.566, so there is a correlation). Range of random error is about +/-40. Hoping the format stays reasonable, the forward projection trend (excluding the random errors) is:
53 142.408
54 143.623
55 144.838
56 146.053
57 147.268
58 148.483
59 149.698
60 150.913
61 152.128
62 153.343
63 154.558
64 155.773
65 156.988
So by this reckoning, the final point (taking into account the random error) would be between about 117 and 197.
(2nd order polynomial has r2 only slightly higher than linear [r2 = 0.59] and will give projection about 20 higher than does linear, with a slightly reduced error range)
2) Third order polynomial - picks up the apparent (by eye) cyclicity of the data, and offers a very different projection:
r2 = 0.684, so is a 'better' correlation (at least offers a closer fit with a significant jump in r2 - no statistical test performed to verify whether this is overfitting). Random error is reduced to about +/-30
53 126.0401
54 123.3656
55 120.2835
56 116.7776
57 112.8317
58 108.4296
59 103.5551
60 98.192
61 92.3241
62 85.9352
63 79.0091
64 71.5296
65 63.4805
So with the random error, the final projected value would probably be in the range between 33 and 93 when including the random error.
I also looked at 4th and 5th order polynomial fits - both offered slightly improved r2 (obviously) and gave very different patterns - 4th order followed a slight upward trend, 5th order a massively downward trend (such that the end of the projection would have been negative).
Without having applied any form of statistical testing, I would say:
1) The linear projection is probably too simplistic, in that visually the data appears to have sections with different characteristics (downwards for first 10 or so points, then a fast increase from about point 15 to 25, slowing to about point 40 and then plateauing). As such, I think it possible that the linear projection is too high.
2) The third order polynomial may be over-influenced by the last couple of points being low relative to their preceding points, and so may decrease too quickly. This would result in the forward projection being unrealistically low.
Of course, it could also all just be a series of random numbers based around 110 +/-50, so the projections may be meaningless.
137.1 = most recent value; the most up-to-date value.
121.05 = average of last two values; because the actual values are hardly ever the same as the preceding value.
105 = the value before last; because the values go up then down a lot so this is the most up-to-date of alternate values.
Simon sends this is by email:
"I reckon it should go as follows:
Reasoning - looks slightly sinusoidal with a +/-20 whatevers, randomly distorting the signal.
139.7808
119.678
140.9939
108.8713
111.4878
135.8748
114.4297
108.6223
116.972
129.7606
107.6586
114.7645
96.63316
114.6549
100.6507
94.30316
84.25568
94.77068
80.2658
107.2726
70.64402
89.35555
100.5096
100.8226
78.75906
104.8564
89.3879
76.27638
80.78541
99.3016
102.8589
88.07086
91.67681
81.35547
107.9862
104.3152
104.0878
79.64331
80.46514
75.77563"
Somebody's using a different definition of "naive" to me. Fitting a sinusoid or a 3rd-order polynomial is "naive"?
You did say "naive". I have always remembered from an exercise in my econometrics course (Wonnacott & Wonnacott) that the slope of a line joining the first and last oberservations is an unbiased estimator! So my answer is:
137,4
137,8
138,1
138,5
138,8
139,2
139,5
139,9
Looks to me like Jonathan's got it! He only missed the AMac answer, that the series was contaminated by Norse bridge-building for the last 15 readings, and so the whole series is unusable.
Ian - I'm amazed by your r2 stats - are you sure? When I do this stuff on prices I use the % change, not the values (since that is the return), and the r2 of that is pretty much zero here.
To be clear, I am looking for predictions of the future trend in this dataset (not individual points) for at least 20 points into the future.
Good stuff so far, I can easily convert to trend. I'll let this run for much of today to allow for further naive forecasts and then proceed to step 2.
Roger:
I can't seem to do the naive forecast. My mind resists.
In looking at the chart I immediately thought of a yearly cycle, and with that assumption my forecast is that if you take a vacation your bloodpressure will go down, if you do not it will continue to approach an unhealthy level.
In other words one has to start with an assumption of whether the data points map a difficult to control or relative easy to control phenomena. The latter leads to sharp discontinuities.
-15-bernie
It is naive, not a real forecast, so don't over think it ;-) We'll get to the real forecast soon!
An AR(1) process applied to the first difference with a multiplier of -0.6.
Naive enough for y'all?
:-)
Roddy
All I did was copy the data into Excel and use the trendline function to give me R2 (so it would be inaccurate to say I calculated it ;-) )
As the functions I used are simply those in the spreadsheet, I'd consider them 'naive' in the sense that the only human consideration was whether a trend line looked (by eye) to give a reasonable fit, and then I simply lifted the equation to calculate the forward projection. There was no real consideration of the form of the data beyond that.
As I said in the first post, the visual impression is of most of one cycle in the data, so the question is what was the simplest form that kind of reflected that (I also considered the possibility of a sinusoidal fit, but that would have been more effort to calculate - indeed, I suspect this is the best model, as I think the linear trend goes too high and the polynomial too low).
I included both the linear and polynomial trends because it really shows how one simple assumption can make a huge difference to the forecast.
How about
119.4
99.1
89.9
99.6
96.7
88.5
for the next six readings?
This assumes that the process is periodic with a period of 52 time units (weeks/year) and does not attempt to filter out noise.
Chuck
I get the average to be about 110.21, standard deviation about 24.48. About 68% of future values will lie between 85.73 and 134.70. About 95% of future values will lie between 61.24 and 159.18.
-11-Rich
I suspect that is the purpose of this exercise. Also, I think you might get different results if you say you'll give a prize.
As I understand it, a naive forecast need not be a simple one - merely one that assumes that something in the data will continue, or be the best guess, for the future. For a stable and trendless process, the best is likely to be the historical mean. A simple moving-range control chart reveals that this process is not stable in that sense, although the plot for the moving range looks well-behaved suggesting that the amount of short-term variation is reasonably stable. The data plot suggests autocorrelation, and this is confirmed by more formal analysis. To get my naive forecast, I just tried some of the options in a software package (Minitab) over a few minutes, and plumped for one which gave results with the smallest or in the vicinity of the smallest of three errors (MAPE, MAD, MSD). Of the ones I tried, this turned out to be an additive models with a linear trend and a seasonal effect (the assumed period of which I did not play with, leaving it set at my first guess of 26 data steps - all assumed equally spaced in time - another thing to check for a non-naive analysis!). Here is a forecast for the next 48 values (that's also naive in the everyday sense - to make such a long forecast from such data, but you did ask for a few dozen values). The autocorrelations of the forecast values are weaker at lag 1 than the original set, but otherwise the forecast looks somewhat similar to the training data:
144.684
134.971
146.226
145.180
166.459
133.782
148.764
151.179
154.666
148.690
143.946
163.235
136.223
155.907
131.149
182.061
131.163
159.887
143.679
150.866
169.513
163.246
176.677
191.370
162.611
201.686
174.744
165.031
176.285
175.239
196.518
163.842
178.823
181.239
184.726
178.749
174.005
193.294
166.283
185.966
161.209
212.121
161.223
189.946
173.739
180.926
199.572
193.305
Roger:
Judging by other comments, overthinking is clearly in the eye of the beholder. ;)
P.S. When I retired, my bp fell 35 pts.
I'll bite ...
The end of the series looks like the collapse of an up-trend. If I assume the thing is cyclical, then I can view the beginning of the series as a forecast of how this collapse will progress. So I say the next 20 points will look like the first 20 points shown.
Naive enough? :)
Predictions of the future trend, not the future data points? Frankly, that's weird. For one thing, it's not a well-defined predictand until you provide a specific definition of "trend".
I think the standard naive forecast for the future data points in this instance would be "persistence", or repeated occurrence of the last value (Jonathan's "Blackboard" forecast). This would have the trend asymptoting to zero.
The other conventional naive forecast for the future data points is climatology, or the long-term mean. Depending on the definition, this would have the trend gradually becoming negative, then asympoting back to zero.
Both of these naive forecasts assume stationarity. I'm hesitant to consider a "climatology of the trend", or OLS fit to the existing data, as a naive forecast, because it abandons stationarity and seems to therefore be a step beyond naive.
Roger--
I posted my naive forcast plus uncertainty intervals. :)
http://rankexploits.com/musings/2010/lets-give-roger-a-naive-forecast/
I'm going with a linear fit and explain why.
Jonathan-- By the Blackboard forecast, do you mean me? My naive is linear trend -- that's pretty much what I tend to expect for "naive" unless the trend is not statistically significant. :)
I hope my predictions are clear, but if not I include the next twenty points for each prediction below. I'll also hedge my bets completely by making a composite prediction made up as a simple average of the three underlying "models".
All done using standard tools in excel.
WUWT:
110.1
109.02
100.94
94.76
93.34
91.48
88.24
84.68
81.14
77.7
81.58
83.16
80.42
89.34
87.18
85.68
82.64
86.18
83.44
89.82
RC:
142.40
143.61
144.83
146.04
147.26
148.47
149.69
150.90
152.12
153.33
154.55
155.76
156.98
158.19
159.41
160.62
161.84
163.05
164.27
165.48
Blackboard:
137.1 (for ever)
Composite:
123.97
117.24
111.89
113.47
112.43
109.48
109.98
105.39
105.55
103.31
102.31
104.57
112.47
112.94
103.93
120.57
102.83
113.91
107.94
112.73
Lucia #27, the Blackboard is of course yours, but the labels are meant to refer to the typical (IMAO) readers of the blogs, not their authors.
Jonathan--
Question-- Why do you think The Blackboard would predict unchanging?
Since unlike markets, courtrooms or race tracks the actual outcome of this exercise (like the one we all assume it refers to) does not reward or punish the predictor as much his or her relative adherence to the consensus of pending predictions, I have to decided to go with whatever lucia does plus 10%.
I'm sorry I got to this late. Jonathan's astute observation that there are 52 points led me to conclude this was weekly averaged data over the period of a year. I therefore multiplied the ordinate by 2*Pi/52 to get a function periodic over the range 0 to 2*Pi. Fitting to a function A+B*Cos[x+phi] led to values of {A -> 110.212, B -> 26.4278, phi -> 7.59545}. . You can obviously subtract 2*Pi from phi, if you like. Eyeballing the residuals, there appears to be no statistical justification for adding a second cosine with half the period.
The next dozen points are 113.839, 110.657, 107.469, 104.321, 101.258, 98.3263, 95.5678, 93.0228, 90.7284, 88.7182, 87.0213, 85.6627. Which, of course, are the best fit to the first dozen points.
On what basis is being assumed that the x-axis is something other than a number for the corresponding data point?
Cycle? Linear trend? Discontinuity? Or a combination there of.
This is like choosing heads or tails in a coin toss.
If one must make a naive prediction, then I plump for a string of zeroes in honor of BP sealing that blowout.
Hi Roger,
127
146
134
143
133
139
150
139
154
144
139
136
151
146
153
155
142
153
162
148
Hi,
My naive estimate was based on just eyeballing the data, and estimating that a linear extrapolation would take the data from a mean of about 140 at the end of the data to a mean of about 150 twenty measurements later. Also, by eyeball, the standard deviation of the existing data looked like about 8.
I hadn't really read any of the other responses at the time I made my forecast. Obviously, as pointed out by others, the "naive forecast" might change, dependent on knowledge of what the data actually represent (e.g., if they were stock market composite values, they might be expected to continue to rise).
These forecasts are all based on models. Unless those models reflect reality, the outcome is reflective of the angle of the dangle. Does my model give a better line than your model. Fierce bar arguments erupt at this point over the measuring stick.
-Gerard Harbison-
Looking at the frequency plot does suggest a strong signal at f = 1/52 weeks. After that point the response is extremely noisy. Have you ever worked with FFTs or discrete time cosine transforms? At that frequency I get a phase angle of 1.4331, which is slightly larger than yours. Then again it might be because I am doing the complete frequency transform and picking the peak, while you are fitting to the best cosine, but I suspect that the difference would just be noise if you are disregarding higher frequency responses, which I agree with.
Just manually fitting a sine wave ...
o <- 112 # offset
a <- 28 # amplitude
p <- -28 # phase
f <- 28 # frequency
a * sin((seq(1:52)+p)/f * pi) + o
My next dozen:
121.24781 118.23059 115.13501 112.00000
108.86499 105.76941 102.75219 99.85126
97.10310 94.54229 92.20101 90.10872
------
I see I'm not the first with a sine solution.
Everything is particle or wave, line or sine. (inclusive or)
This looks like (some noise plus) a piecewise linear function with inflection points that are fairly well marked by spikes in the square of the third difference -- ie the segments where the data oscillate with period 2 in the original. see:
http://autogeny.org/blog/wp-content/uploads/2010/08/image.png
the data are smoothed with a window of 4, making the linearity of the segments between inflection points more obvious. The height of the peaks in the square of third diff seem to correspond with the magnitude of the slope change at that point.
The peak at 50 seems a lot like the one at 15, so unless we had good reason to believe the curve was really cyclical with a period of 52, we'd have to guess that 50 represents a sharp downturn like the upturn at 15.
The piecewise linear model would make sense if we assume some process that gets occasional sharp perturbations, with the oscillations at the inflection points being ringing in response to the inputs.
So I'd propose a prediction that was just the mirror image of the curve with the inflection point at 15 superimposed on the one at 50 ... of course we have no real way of knowing what the next inflections will be, but we can guess that there will be a big one followed by successively smaller ones as shown in the diff curve.
Averaging a 4-point window again:
136. , 144.125, 134.7 , 133.85 , 125.425, 114.9 , 116.05 , 105.95 , 106.2 , 112.45 , 108.7 , 114.925, 108.775, 108.3 , 106.675, 104. , 105.775, 101.125, 101.85 , 98.7 , 103.6
Ignignot.
I did a Fourier, and didn't see anything other than the primary frequency; that's why I truncated the series. It turns out the number of data points was a red herring; Roger is evil. :-)
I also tried power series; truncating at the quadratic gave something like Hansen's prediction. Truncating at the cubic term give divergent negatively growing predictions.
I know what conclusions Roger drew. My conclusion (well hardly a conclusion; more a reinforcement of existing prejudice) is that interpreting a data series is never 'naive'. Knowing that the x variable is cyclic/pseudo-cyclic or acyclic, for example, is important. Year-to-year predictions tend to discount cyclicity, and without much justification. I would be interesting to check the s.d. based on sinusoidal vs. power series fits, but I'm not going to bother.
Duh!
136.1
144.1
141.1
145.1
116.1
126.1
134.1
136.1
133.1
145.1
140.1
127.1
137.1
151.1
129.1
132.1
121.1
149.1
138.1
145.1
155.1
163.1
141.1
163.1
146.1
142.1
149.1
163.1
168.1
156.1
174.1
172.1
149.1
150.1
160.1
174.1
166.1
176.1
193.1
169.1
170.1
185.1
193.1
192.1
185.1
199.1
191.1
194.1
180.1
194.1
Steve's nature trick.
Gisstemp global anomaly*100. splice 1960 to present to the butt end of rogers graph
What will come next:
For time, n, after last time, the output will be a log-normal distribution with parameters (mu = 0, sigma^2=0.0216*n) times 137.1
The expected value of the output at time n (n=1 is the first time after the last given sample) is 137.1*exp(0.0216*n) (note: this looks like a line with slope 1.02 for small n), the sd of the output at time n is 137.1 * sqrt(exp(0.0433*n)*(exp(0.0433*n)-1))
Method:
- Assume that the model is s(n) = exp(Xn)*s(n-1) where Xn is a Gaussian with mean 0 and variance extracted from the data.
- Estimate the variance by computing b(n) = log(s(n)/s(n-1)) and then compute Var(b(n)).
- Validate: Does b(n) look Gaussian? By an eyeballed order statistic test it fits very well. Could do a further test but I won't. The mean of b(n) is 0.00271 which is close enough to 0 given the sample sd.
A plot of b(n) shows that it oscillates a little more than is seemly for a Gaussian but for a naive fit this will do. Also from b(n) it looks like someone stirred up the system around the 12th sample in the original series.
140.069, 29.3077
143.102, 42.8058
146.201, 54.1487
149.367, 64.5849
152.602, 74.592
155.907, 84.4155
159.283, 94.2037
162.732, 104.057
166.256, 114.047
169.857, 124.232
173.535, 134.659
177.293, 145.367
The first number is a mean, the second is a standard deviation
Joseph, I'm pretty sure that a log normal distribution with an average return of 0 will simply forecast the last value to continue on. Other than that it looks like I won't be putting my 401k into this series!
Hi Ignignot,
I gave the formula for the prediction and it most certainly does not predict the last value. Notice the sigma^2 parameter changes with n - The expected value of a log-normal distribution is exp(mu+sigma^2/2).
The model I gave is not a Martingale.
Joe
Post a Comment