Predicting Scottish Councils elections: an impossible task?

By Jeremy Blackwell

A few words about me

Jeremy BlackwellJeremy Blackwell, almost 26 now, known on Twitter as @WeAreThe59. My friends usually call me Jez or JB but Jeremy is OK—actually I like it better. Born in France and studied statistics and political science there.

Living in Craiglockhart since late 2015 with my French boyfriend Alban and our beloved dog Bosco. Also member of the SNP, so no ambiguity about where I stand. Started seriously working on polls and projections during the 2015 General Election campaign; loved it and carried on during the 2016 Scottish Parliament campaign. And now giving a try at the 2017 Scottish Council election.

Hits and misses

Psephology is not an exact science. Sometimes polls are wrong. Sometimes your method is wrong. It’s closer to ‘trial and error’ than to rocket science, whatever the pundits might tell you about it.

Like everyone I totally missed General Election (GE) 2015 because UK-wide polls were so off the actual result. But I got Scotland right because Scottish polls were really accurate. Got the 56-1-1-1 part right; only got the exact seats wrong. I thought the Labour seat would be Glasgow North East; not Edinburgh South. I also thought that the Tory seat would be Berwickshire, Roxburgh and Selkirk, not Dumfriesshire, Clydesdale and Tweeddale. Still, I got the Lib Dem seat right. That was the easiest one.

I did better on Scottish Parliament (SP) 2016. Everybody got it wrong because the polls were wildly overestimating the SNP vote. But I was the ‘least wrong’ of all, crediting the SNP with only 66 seats while others went up to 70 or even more.

Now to the heart of the matter

The main question is: can Scottish Council elections be predicted with the same level of accuracy as Westminster or Holyrood elections? And the answer is a resounding no. Standard or sophisticated statistical methods that work for Parliamentary elections mostly fail for Council elections because there are many more variables that interact in sometimes unpredictable ways.

The first factor is that there are many more independent candidates in Council elections than in Parliamentary elections, and they get a much bigger share of the vote:

  • 0.1% in the 2015 General Election in Scotland
  • 0.3% in the 2016 Scottish Parliament election
  • but 11.8% in the 2012 Scottish Council election

The second factor is that Council elections definitely follow a wholly different pattern from other elections, like Westminster (WM) and Holyrood (HR) elections did before 2015 when they finally aligned. Council elections are still very far from aligning on the WM/HR voting patterns. One reason is that we have 59 MPs, 73 constituency MSPs but 1,219 Councillors after the last Boundary Review. So personal factors have a heavier influence on the Council vote, while Parliamentary votes are closer to a divide purely on party lines that can be statistically projected.

The third factor (and probably the most important) is the voting system: STV (Single Transferable Vote). There are many technical explanations of STV to be found (just Google it and make your pick). All of them are technically right but all are wrong on one major point. STV is not and has never been PR (Proportional Representation). Quite the opposite, in fact. STV is the perfect vehicle for tactical voting, and is even worse in many cases than FPTP (First Past The Post). Partly because FPTP is basic and simple while many people find STV extremely complicated and don’t fully grasp its limits and implications, and also how it can be used to maximise tactical voting.

So why do I even try to predict the outcome of Scottish Council elections?

Because Council elections are the most difficult to predict, and it’s the first time I’m trying it. Besides, it’s fun and I like it.

Because it’s part of a learning process. Whether I get it totally wrong or get it spot on is irrelevant. In both cases I’ll learn from it and be better next time.

Because I have a lot of followers on Twitter who really appreciate what I’m doing. Sometimes I bring bad news and sometimes good news. I truly am humbled and grateful for all the support I get. Including ‘for your eyes only’ information (you know who you are).

Because I think that, as an SNP member, I can help the party and all local branches with my work. Part of it being giving the most accurate data and analysis I can; part of it being boosting morale when required.

So yes, I’m partisan and will never deny it. But no, I’m not biased, meaning I’m always as professional as I can be about all this.

Firstly because crunching numbers and making sense out of them is my day job. I work on these elections stats as seriously as I do on the highly confidential corporate data I have to handle on a daily basis at work. The same methodology applies whatever the data you want to analyse. From basic stats to elaborate multi-criteria models, everything works if you have the right hunch at the right time. And part of it is also just a lucky guess. I’m quite good at that too.

Secondly because in the end it’s all about interpreting data that have been made public already. So everyone can check them and form their own opinion regardless of mine. The analysis is mine and mine only, but the underlying data are as solid as can be, most of it from official sources.

Numbers never lie except when they do, which happens more often than you’d think. And my SNP friends all know that when I have a choice between a rosy prediction for us and a darker one, I always go for the darker. That’s part of being a ‘watcher of the skies’ for my party.

And, more important, how can I do it with a reasonable level of reliability?

I guess you have all understood that there is no foolproof way to predict the outcome of an unpredictable election. That’s why I crosstab data from four different sources:

  • Opinion polls
  • Results of Council by-elections since 2012
  • Trends seen in earlier SP and House of Commons (HofC) elections
  • ‘For your eyes only’ information I get from SNP Twitter friends

There have been only two Council polls so far. Both point to a strong SNP victory. But both are underestimating the Independent vote. So I don’t take them as the most reliable source. I think Independents will do better than polls say. They’ll probably take 2–3% from the SNP and 1% each from Labour and the Conservatives.

*

By-election results are definitely good for the SNP, especially after September 2014. But the average gain/loss also shows the danger zones and possible gain zones for the SNP, some of which are truly counter-intuitive compared to what happened in the 2016 SP elections.

*

During the last term the SNP lost two seats overall in by-elections. But they gained seven in by-elections held after the independence referendum. That’s the kind of result every governing party in Europe would love to get in local by-elections—you can take my word for it.

The trends we saw in previous elections (2015 and 2016) are not a true predictor for Council elections, because all elections have their own logic and their own voting patterns. But what we see this time is that the Scottish vote is roughly aligning for all elections. There are still three discriminating criteria: independent candidates, tactical voting, and personal profile of candidates. But with all data pointing the same way, I have to think that factoring in other elections and their own polling makes sense.

Finally, there is information I get directly from SNP members (again, you know who you are). I obviously can’t go public with all this information. But it helps taking into account local situations that are not known nationally. And it factors in as the final fine tuning of my predictions.

Will the snap General Election have an impact on Council elections?

Of course it will. All parties in Scotland are already campaigning for Councils, and the snap GE can only make them more determined. Especially the two main players, the SNP and Conservatives. Both know the outcome of Council elections will be a good predictor of the GE’s outcome in Scotland. So both already have to allocate resources (or re-allocate them) to areas that will see the hardest fights.

Conservatives and Liberal Democrats have already ‘leaked’ to BuzzFeed a list of 13 Westminster seats they think they can gain from the SNP.  Which I think are not seats they actually think they’ll win, but rather a distraction to make the SNP think twice about how they choose priorities and allocate resources. Not just for the snap GE, but already for Council elections. It’s a common practice known as ‘ratfucking’ in the USA: confuse your opponent and make him think he’s making the right choices while he’s in fact making the wrong ones.

The Conservatives also have already started campaigning on ‘vote for us on both 4 May and 8 June.’ I just hope both the voters and the SNP won’t fall for it. They’re different elections fought on different issues.

A few words about the SNP’s strategies

An example of a sound electoral strategy is what Americans call the ‘Fifty-state strategy’. Theorised by Howard Dean when he was chairman of the Democratic Party in 2004, the basis of the strategy is that you field candidates everywhere and campaign everywhere regardless of the anticipated result. That way you can benefit from unexpected upsets. It worked for the Democrats when they regained control of the House of Representatives in 2006. Then Barack Obama used the ‘campaign everywhere’ part during the 2008 primaries, winning many delegates in small Republican-leaning states that Hillary Clinton totally overlooked, and so clinched the nomination.

Can a variant of such a strategy apply to our Council elections? I believe it can, and it definitely looks like the SNP basically adopted it. Below is the number of SNP candidates in each Council, and two important ratios: candidates per seat, and candidates per ward.

*

A candidates/seat ratio above 0.50 means you have enough candidates for a majority. Between 0.40 and 0.50 means you can’t have a majority, but a minority administration still remains feasible. Below 0.40 means that even the possibility of a minority administration is remote and you implicitly already conceded.

On average the SNP has the highest candidates/seats ratio of all parties (0.51) and remarkably is the only party with an average ratio above 0.5. The SNP also has the highest candidates/ward ratio (1.79) and is the only party with a ratio above 1.5. Both ratios show—even before any vote has been cast—that the SNP is the only party that can hope to get majorities or solid minorities in most councils.

The three islands councils have a life and a logic of their own, and will have an Independent majority no matter what. So, of the remaining 29 councils, the SNP implicitly conceded only two. And the SNP is also ready for a majority in 18 (62% of the mainland councils) and could get a workable minority in nine (31%). Which is remarkable when compared to the 2012 results: two SNP majorities, and seven SNP minority administrations.

So all available evidence says the SNP have chosen a strongly offensive (in the military sense) strategy and don’t want to miss any opportunity to gain both seats and control of Councils. Detailed ward-by-ward data (available here) also show that the SNP have chosen the number of candidates per ward to maximise gains. Some choices look counter-intuitive at first, like fielding only two candidates in Anderston-City (Glasgow) or three in Sighthill-Gorgie (Edinburgh). But a closer look at voting trends shows both make sense.

The former is typical ‘play it safe’ to avoid a split vote and make sure both candidates are elected. The latter is typical ‘play it bold’ because the overall result in Edinburgh will be tight, and there is a credible possibility in that ward to gain one seat from Labour.

And what of other parties’ strategies?

I won’t go into as much detail as I did for the SNP. Full detailed data about the 2,572 candidates are online here, and my own analysis of it is here.

What is most striking is that raw data show Scottish Labour have already accepted a major defeat. They just don’t field enough candidates. Only 0.37 candidates per seat on average, which is far below what you need to hold control of your Councils. And resources are clearly concentrated on only nine Councils at best. But even these nine are clearly within the danger zone and ready to switch to the SNP.

*

More unexpected is the Scottish Tories’ lack of true ambition. On a purely statistical approach they had opportunities for control of six or even eight Councils. But now the number of candidates they are fielding reduces that to, at most, four and possibly only three (the same as in 2012). Even Dumfries and Galloway, which once was a nearly sure gain, now looks like a very remote possibility. A striking fact is that the Conservatives are fielding fewer candidates in D&G than the SNP, so a major upset in not impossible.

*

The Liberal Democrats have already accepted they’re minor players. Their highest candidates/seats ratio is 0.35, in Dundee of all places. The average is 0.20 candidates per seat, and they’re not fielding any candidate in eight councils. That’s a sure sign they know they’ve lost most of the strongholds they once had in Scotland.

Then the wildcard is Independents, as usual. Their share of the vote is the most unpredictable. Many of them are actually ‘closet Tories’ and this year also ‘closet Labour’. And by definition they don’t have any unified strategy. As such I think they deserve to have the full table displayed. It definitely proves they have a totally counter-productive approach. In many wards they have too many candidates and that ensures none will get elected because of a split vote. The best they can expect is holding control of the three Islands councils. But their two minority administrations on the mainland are clearly threatened by the SNP.

*

And what does all of this say about the outcome?

Lots, but with a wide margin of error. The SNP will win, but will it be 520 councillors or 580? Nobody can tell for sure. Labour will lose but will they hold control of at least one Council? Nobody can tell for sure. Not me, not anyone else.

The best I can offer is my ablest approach, which is part classic statistical analysis and part best educated guess. In this case ‘best educated guess’ is the main component. Don’t hang me if I’m wildly off.

So let’s go. Vote share and change from 2012 first:

*

*

Then how I think it will translate into seats and change from 2012:

*

*

Clearly both the SNP and Labour benefit from the Conservatives’ lack of ambition in this election. The Conservatives just don’t not field enough candidates and I estimate it will cost them 20 to 25 seats, roughly half to the SNP and half to Labour. But more importantly it will also cost them control of several councils (best estimate 2 or 3).

And finally how I expect it will translate into Council control:

*

*

*

Conclusion (sorta, for now)

I know my projection will look astounding for many. But I think it’s right and just in line with major realignments we’ve seen in Scotland recently, namely Scottish Parliament 2011 and House of Commons 2015. Whether you like it or not, the SNP is now the dominant party in Scotland. The 2017 Council elections are just the last stage of Scottish Labour’s demise.

My projections just show the cumulative effect of what we already saw in 2011, 2012, 2015, and 2016. The center-left vote has definitely switched from Labour to the SNP.

This election is also the first evidence that the Scottish Conservatives ‘surge’ is not a given. Obviously national issues also have had an impact and lowered support for the Conservatives. Think ‘rape clause’. Their lesson to learn.

The Councils and snap General Election are probably the last major elections before the second independence referendum. Whatever the result of these Council elections, they will have a great impact on Scotland’s future. For better or for worse, and anyway more than any previous Council elections ever had.

My point here is not to campaign for the SNP. It’s just to tell you all to make the best choice according to your strongest beliefs. Make that choice on Council issues only—nothing else is at stake in these elections. Make an informed choice and never believe the lies, wherever they come from.

So we’ll be a better nation.

Saor Alba gu bràth.

Jeremy Blackwell, 23 April 2017

7 thoughts on “Predicting Scottish Councils elections: an impossible task?”

  1. Excellent. I look forward to your General Election analysis, especially as the MSM are pushing the Tory revival in Scotland angle.

    Unfortunately, I think that will be the case, mainly because Labour are quite correctly being cast into irrelevance by the electorate in Scotland.

  2. Thanks Jeremy, really enjoyed your detailed, thorough and well reasoned analysis. The data on candidates : wards ratio fascinating. Enjoy your Twitter very much also, keep up the good work.

  3. Really thankful to Jamie for the opportunity to get an article online here.
    Next one will be on the same lines and also a bit more provocative.
    About what polls and statistics say and what they don’t.
    Saor Alba gu bràth

Leave a Reply

Your e-mail address will not be published. Required fields are marked *