Failed predictions and gravity models

Discussions about serious topics, for serious people
Post Reply
User avatar
jdc
Hilda Ogden
Posts: 1925
Joined: Wed Sep 25, 2019 4:31 pm
Location: Your Mum

Failed predictions and gravity models

Post by jdc » Sat Oct 23, 2021 7:57 pm

jdc wrote:
Sat Oct 23, 2021 5:22 pm
sheldrake wrote:
Thu Oct 21, 2021 8:54 am
To be fair they probably used the same gravity models with mis-set parameters that generated all the wrong predictions before the referendum.
I'm in the market for a comprehensive list of failed predictions if you have one to hand.

I hadn't been thinking of them, but now that you've mentioned gravity models with mis-set parameters being responsible I'd be very interested to see how that list of failures breaks down by use of a gravity model or not. If you could also explain what's wrong with the parameters used and what the correctly-set parameters would have been, that would be great but obviously I appreciate I'm asking a lot here and you only have so much time to spare.

PeteB
Clardic Fug
Posts: 205
Joined: Wed Nov 13, 2019 1:02 pm

Re: Failed predictions and gravity models

Post by PeteB » Mon Oct 25, 2021 7:56 am

It's not really answering your question but there is an article here on gravity models that was quite interesting

https://oxfordre.com/economics/view/10. ... 5979-e-327

eta wiki entry

https://en.wikipedia.org/wiki/Gravity_model_of_trade

But they seem to think gravity models accurately predict trade flows

PeteB
Clardic Fug
Posts: 205
Joined: Wed Nov 13, 2019 1:02 pm

Re: Failed predictions and gravity models

Post by PeteB » Mon Oct 25, 2021 8:01 am


PeteB
Clardic Fug
Posts: 205
Joined: Wed Nov 13, 2019 1:02 pm

Re: Failed predictions and gravity models

Post by PeteB » Mon Oct 25, 2021 8:10 am

There was this from 2017 - don't know how it stands up now ?

https://ideas.repec.org/p/cbr/cbrwps/wp490.html

PeteB
Clardic Fug
Posts: 205
Joined: Wed Nov 13, 2019 1:02 pm

Re: Failed predictions and gravity models

Post by PeteB » Mon Oct 25, 2021 8:15 am


User avatar
El Pollo Diablo
Stummy Beige
Posts: 3323
Joined: Wed Sep 25, 2019 4:41 pm
Location: FBPE

Re: Failed predictions and gravity models

Post by El Pollo Diablo » Mon Oct 25, 2021 8:31 am

Busy morning, Pete?
If truth is many-sided, mendacity is many-tongued

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Mon Oct 25, 2021 10:26 am

Sorry was slow getting to this, wanted to construct something with good references.

There are several kinds of failed predictions w.r.t Brexit, and I'd like to cover them all, but in separate posts. First I'll break down the categories of failed prediction I see: -

1) Failed predictions concerning macroeconomic indicators like GDP growth, levels of unemployment, investment, exports and imports stemming from Brexit.

2) Failed predictions concerning specific logistics problems, e.g. 'food and medicine shortages'.

3) Failed predictions w.r.t. the movement of people and a cultural tone of xenophobia or racism

4) Failed predictions about the UK's ability to close trade deals

5) Failed predictions from the pro-brexit camp.

I will try and keep all of these things in separate posts as much as possible for clarity, using the following format

<Some person or organisation making a forecast>
What did they predict?

How did they make this prediction?

What was the outcome as of October 2021?

Discussion of the accuracy of the prediction and if it was very wrong, what might have gone wrong.

Does this work for you, JDC ?

User avatar
El Pollo Diablo
Stummy Beige
Posts: 3323
Joined: Wed Sep 25, 2019 4:41 pm
Location: FBPE

Re: Failed predictions and gravity models

Post by El Pollo Diablo » Mon Oct 25, 2021 10:50 am

Do you think it would work better as a podcast series?
If truth is many-sided, mendacity is many-tongued

User avatar
Woodchopper
Princess POW
Posts: 7057
Joined: Sat Oct 12, 2019 9:05 am

Re: Failed predictions and gravity models

Post by Woodchopper » Mon Oct 25, 2021 11:14 am

When people are discussing this subject it would be good if they could distinguish between predictions and scenarios.

A prediction takes the form of a statement that something will happen, perhaps qualified by likelihood. For example, there is a 80% chance of rain on Wednesday. Its therefore possible to assess the accuracy of a prediction (eg did it rain on Wednesday).

A scenario takes the form a statement that if certain conditions occur then something will happen, perhaps qualified by likelihood. For example, if it isn't raining on Wednesday then there is an 80% chance that I'll go for a walk in the park. Analysis of the accuracy of the scenario needs to take into account both the weather and whether I went for a walk.

This is relevant as most, of not all, of what I've seen on Covid or Brexit have been analysis of scenarios. For example, if people do not change their behaviour then there will be 400 000 deaths by the end of the year.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Mon Oct 25, 2021 11:54 am

El Pollo Diablo wrote:
Mon Oct 25, 2021 10:50 am
Do you think it would work better as a podcast series?
I cannot tell if you're teasing me here EPD :) A podcast would be very hard to do without doxing myself. Plus, I have a boring flat voice and it's hard to present graphs etc.. in a podcast.

User avatar
dyqik
Princess POW
Posts: 7527
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: Failed predictions and gravity models

Post by dyqik » Mon Oct 25, 2021 1:32 pm

Woodchopper wrote:
Mon Oct 25, 2021 11:14 am
When people are discussing this subject it would be good if they could distinguish between predictions and scenarios.

A prediction takes the form of a statement that something will happen, perhaps qualified by likelihood. For example, there is a 80% chance of rain on Wednesday. Its therefore possible to assess the accuracy of a prediction (eg did it rain on Wednesday).

A scenario takes the form a statement that if certain conditions occur then something will happen, perhaps qualified by likelihood. For example, if it isn't raining on Wednesday then there is an 80% chance that I'll go for a walk in the park. Analysis of the accuracy of the scenario needs to take into account both the weather and whether I went for a walk.

This is relevant as most, of not all, of what I've seen on Covid or Brexit have been analysis of scenarios. For example, if people do not change their behaviour then there will be 400 000 deaths by the end of the year.
Additionally, all predictions are scenarios made with an assumption that the background conditions will remain roughly the same as when the prediction was made. A prediction of an 80% chance of rain in 9 days time* is making the assumption that a giant meteorite impact does not occur, that a terrorist attack doesn't shut down global air travel, a volcano doesn't produce a giant ash cloud over the Atlantic, that global thermonuclear war doesn't occur, etc.

Any prediction or scenario about Brexit in 2021 that was made before CoVID was understood is likely to be off because no one could reasonably model a global pandemic, with big disparities in how different countries were affected and responded, big effects on travel, and complex effects on supply chains. The UK and EU have had different pandemics, with different lockdowns, vaccine roll-outs, and exposure to global conditions.

It will be possible to test detailed predictions and scenarios over longer time periods, but the recent 18 months have had very different background economic conditions to those expected or reasonably foreseeable, and it's not yet fully understood how the pandemic has changed economic conditions.

*2 days time is probably a little too close to be affected by all of the options I list.

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: Failed predictions and gravity models

Post by plodder » Mon Oct 25, 2021 1:49 pm

Is anyone interested in a long list of categorised wrong predictions with someone then giving their own homilies about why? Sounds a bit strawman monologue to me. It's also veering away from whether gravity models are any good.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Mon Oct 25, 2021 6:22 pm

Treasury made 2 kinds of forecast, ones for the short term based partly on the effect of 'uncertainty' on the economy, and longer term forecasts.

see here https://assets.publishing.service.gov.u ... eu_web.pdf
The analysis in this HM Treasury document quantifies the impact of that adjustment
over the immediate period of two years following a vote to leave. Such a vote would
trigger a redefinition not only of the UK’s economic relationship with the EU and the rest
of the world, but also of much of the UK’s domestic economic policy, regulatory and
legislative framework. A vote to leave would cause an immediate and profound
economic shock creating instability and uncertainty which would be compounded
by the complex and interdependent negotiations that would follow.
The central conclusion of the analysis is that the effect of this profound shock would
be to push the UK into recession and lead to a sharp rise in unemployment.
Emphasis mine: This prediction was definite in timescale, and not contingent on us fully leaving the EU (this timescale is within the specified article 50 negotiation period even if article 50 had been notified immediately).

These short-term predictions partly came from a vector autoregression (VAR) model which tries to model the impact of
increased uncertainty on overall economic activity, but they also incorporated changes in trade: -
Treasury immediate impact analysis wrote: The immediate economic impact would be driven by three key factors:
1 the ‘transition effect’: the emerging impact of the UK becoming less open to
trade and investment under any alternative to EU membership

2 the ‘uncertainty effect’: the rise in uncertainty following the referendum and
the impact that has on economic decisions
3 the ‘financial conditions effect’: the extent of financial market volatility
The section I've bolded would take gravity modelling as an input.

We can see now that the short-term (2 year from vote) predictions were false through the period 2016-2019. The economy continued to grow, and unemployment continued to fall until 2020. Unemployment is already back below 2016 levels now.

This does however leave me with a bit of a puzzle; The treasury appears to have used changes in trade with the EU in its short-term scenario modeling, even though the time period described is too short for the rules we trade with the EU under to have actually changed. It's not clear to me why they would do this and their paper does not spell it out. They repeatedly state throughout their document that they expect changes to happen because other actors throughout the economy will adapt to a new future; and it seems they think everybody else will be expecting the same future as them: -
2.14 The scale of the short-term impact of this transition to a permanent
reduction in trade, FDI and productivity would depend crucially on the sort of
relationship the UK seeks with the EU.
2.15 Reflecting the shock of a vote to leave, there are likely to be immediate
consequences to reflect the adjustment to becoming permanently poorer in the long
term, particularly in financial markets. Businesses and households would start to adjust
to being permanently poorer in the future by reducing spending immediately. For some
businesses and households, this effect may only take hold over time as the difficult
negotiations begin and it becomes evident that the outcome will lead to a UK economy
that is less open to trade and investment.
2.16 The modelling of both scenarios assumes that these impacts would build
up over 15 years from the time of a vote to leave the EU. The analysis in the longterm document was modelled over a period of 15 years, which is considered to be a
sufficient time horizon for the UK’s future relationship with the EU to be clear and the
economy to have adjusted to a new ‘steady-state’ outside of the EU.

2.17 The size of these effects on trade, FDI and productivity in the shock scenario
reflect the transition to the central estimate of the negotiated bilateral agreement
alternative. As set out in the long-term document, leaving the EU under this alternative
would result in Gross Domestic Product (GDP) per household being £4,300 lower every
year after 15 years, than if the UK remained in the EU.
Unfortunately the Treasury document does not quantitatively break down the different components that would produce the short-term collapse in growth and employment they expected, at least not in a clear way.

This means that whilst I can say confidently the the Treasury forecasts for 2016-2018 are falsified (directionally wrong during the whole period), gravity models were only one of the inputs used and I'm going to have to trawl through their long-term impact paper to look at the effects that those predicted over time for trade to see if they specifically were directionally wrong w.r.t to EU trade immediately after a vote to leave.

User avatar
jdc
Hilda Ogden
Posts: 1925
Joined: Wed Sep 25, 2019 4:31 pm
Location: Your Mum

Re: Failed predictions and gravity models

Post by jdc » Tue Oct 26, 2021 12:43 am

I can't see that the failures of the short-term predictions have anything to do with use or misuse of a gravity model. Looks to me like it's down to them making some weird assumptions.

User avatar
jdc
Hilda Ogden
Posts: 1925
Joined: Wed Sep 25, 2019 4:31 pm
Location: Your Mum

Re: Failed predictions and gravity models

Post by jdc » Tue Oct 26, 2021 12:45 am

plodder wrote:
Mon Oct 25, 2021 1:49 pm
Is anyone interested in a long list of categorised wrong predictions with someone then giving their own homilies about why? Sounds a bit strawman monologue to me. It's also veering away from whether gravity models are any good.
My fault - I should have said "comprehensive list of failed economic predictions (macro and micro)".

Soz.

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: Failed predictions and gravity models

Post by plodder » Tue Oct 26, 2021 7:04 am

jdc wrote:
Tue Oct 26, 2021 12:45 am
plodder wrote:
Mon Oct 25, 2021 1:49 pm
Is anyone interested in a long list of categorised wrong predictions with someone then giving their own homilies about why? Sounds a bit strawman monologue to me. It's also veering away from whether gravity models are any good.
My fault - I should have said "comprehensive list of failed economic predictions (macro and micro)".

Soz.
If you’re going to be grandiose then why limit it to economics? There are all sorts of predictions from loads of other disciplines that are demonstrably wrong. Have some ambition, man.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Wed Oct 27, 2021 11:31 pm

The gravity modelling is explained in Appendix A in this document https://assets.publishing.service.gov.u ... ip_web.pdf

Some of the assumptions called out earlier in the document have already been falsified, e.g that 15 years after voting to leave we might just have reproduced the deals with non-EU countries we had as a result of our EU membership, but we'd have no deals that went further than this. We already have deals that go further in the case of Australia, Japan, Canada and New Zealand.

The modelling is heavily dependent on assumptions about FDI dropping which has also shown to be untrue since 2016.

What's left is based on estimating the uplift in UK trade as a member of the EU by looking at data going back to 1950 for country-pairs and showing how trade increased.

This is critiqued in this paper from the Judge Business School in Cambridge
https://www.cbr.cam.ac.uk/wp-content/up ... /wp490.pdf

tldr; Treasury modellers neglected the following: -

1) The impact of EU membership on trade has diminished over time
2) They don't model the positive effect of our currency devaluation on exports
3) They use these gravity models in the 'immediate' scenario above when it doesn't make sense to. I think the only part of the long-term analysis that would really make sense to use starting in 2016 would be FDI effects.

My interpretation is the following: -

1) It's too early to really be sure the gravity models are wrong, only to say that it was inappropriate for the Treasury to use them in their short-term forecast (which nonetheless was endorsed by a knighted economics professor).

2) It's fair to be skeptical about them based on the way they were set up and ignore currency devaluation effects.

3) The VAR technique used to estimate the impact of uncertainty gave very wrong results

4) The predictions about FDI were very wrong

5) The treasury made very overly-pessimisstic assumptions about how hard it would be to sign new trade deals

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: Failed predictions and gravity models

Post by plodder » Thu Oct 28, 2021 8:48 am

tldr:

GIGO

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Thu Oct 28, 2021 8:58 am

Note, the OBR are currently doing a news cycle where they rebroadcast the results of these models.

PeteB
Clardic Fug
Posts: 205
Joined: Wed Nov 13, 2019 1:02 pm

Re: Failed predictions and gravity models

Post by PeteB » Thu Oct 28, 2021 2:13 pm

sheldrake wrote:
Thu Oct 28, 2021 8:58 am
Note, the OBR are currently doing a news cycle where they rebroadcast the results of these models.
I may have misunderstood but now we've left the EU the OBR can track their modelling against real world data and they think the GDP impact is 4% (rather than 6%) ?

PeteB
Clardic Fug
Posts: 205
Joined: Wed Nov 13, 2019 1:02 pm

Re: Failed predictions and gravity models

Post by PeteB » Thu Oct 28, 2021 2:40 pm

https://obr.uk/efo/economic-and-fiscal- ... ober-2021/

page 64 of the pdf - numbered p58 at the bottom of the page

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Thu Oct 28, 2021 3:00 pm

PeteB wrote:
Thu Oct 28, 2021 2:13 pm
sheldrake wrote:
Thu Oct 28, 2021 8:58 am
Note, the OBR are currently doing a news cycle where they rebroadcast the results of these models.
I may have misunderstood but now we've left the EU the OBR can track their modelling against real world data and they think the GDP impact is 4% (rather than 6%) ?
They still think trade will end up 15% lower (in line with Treasury gravity models), and that productivity will be 4% lower.

User avatar
jdc
Hilda Ogden
Posts: 1925
Joined: Wed Sep 25, 2019 4:31 pm
Location: Your Mum

Re: Failed predictions and gravity models

Post by jdc » Thu Oct 28, 2021 7:52 pm

Thanks for the links Sheldrake.

Unless I've horribly misread, the Treasury's assumptions included rolling over none of the EU trade deals with non-EU countries which looks like it's wrong by about £180bn-a-year-worth of trade deals from a quick eyeballing of the list https://www.gov.uk/guidance/uk-trade-ag ... -in-effect

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: Failed predictions and gravity models

Post by sheldrake » Thu Oct 28, 2021 8:39 pm

jdc wrote:
Thu Oct 28, 2021 7:52 pm
Thanks for the links Sheldrake.

Unless I've horribly misread, the Treasury's assumptions included rolling over none of the EU trade deals with non-EU countries which looks like it's wrong by about £180bn-a-year-worth of trade deals from a quick eyeballing of the list https://www.gov.uk/guidance/uk-trade-ag ... -in-effect
They said they thought it might be possible by the end of the 15 year timescale, but that it would be impossible to do any custom deals that go further than rollovers from EU deals in that timescale. We already have deals with 4 countries which go further. They (along with many other pro-Remain 'opinion formers') massively overestimated how hard it was to do new trade deals when you don't have to reach consensus with 27 other countries each time.

Post Reply