Privacy campaigners have expressed alarm after the government revealed it had hired an artificial intelligence firm to collect and analyse the tweets of UK citizens as part of a coronavirus-related contract.

Faculty, which was hired by Dominic Cummings to work for the Vote Leave campaign and counts two current and former Conservative ministers among its shareholders, was paid £400,000 by the Ministry of Housing, Communities and Local Government for the work, according to a copy of the contract published online.

In June the Guardian reported Faculty had been awarded the contract, but that key passages in the published version of the document describing the work that the company would carry out had been redacted.

In response to questions about the contract in the House of Lords, the government published an unredacted version of the contract, which describes the company’s work as “topic analysis of social media to understand public perception and emerging issues of concern to HMG arising from the Covid-19 crisis”.

A further paragraph describes how machine learning will be applied to social media data.

Silkie Carlo, the director of the civil liberties campaign group Big Brother Watch, which discovered the updated contract, expressed alarm at the details. “This is effectively AI-powered mass political surveillance, and it’s been done in a very secretive way, apparently to inform policy,” she said.

“It seems from the contract this social media monitoring has been going on for three months, and machine learning has been used for that time. We don’t know what the impact is.”

A Faculty spokesperson declined to comment on the criticism, but said that the project analysed public posts from Twitter containing words relating to the pandemic, such as “covid” or “coronavirus”, as part of an endeavour to help the government detect emerging issues relating to the pandemic in local communities.

They said personally identifiable information, such as Twitter usernames or profiles, was stripped out from the data at the point of collection, meaning it would be impossible to use the information to profile any individual or group of people.

Carlo expressed scepticism that social media posts would provide the government with an accurate insight into public attitudes. “Twitter is not representative of public opinion as a whole,” she said. “I think there are a lot of questions to be asked about the premise.”

An MHCLG spokesperson said: “We are satisfied that the service provided by Faculty was of a high standard, and delivered on value for money.” They added that the contract expired in July.

The MHCLG social media work is one of several government contracts awarded to Faculty relating to coronavirus. In March an existing contract to support the NHS with AI initiatives was expanded so that the firm could help create a “coronavirus datastore” for the government, visualising NHS data to help inform ministerial decision-making.

The company has several links to figures in the Vote Leave campaign and the Conservative party. Two of its investors, Theodore Agnew and John Nash, are current or former Conservative ministers.

In 2016 the firm was recruited by Cummings to provide data science and machine learning technology for the Vote Leave campaign. The work was carried out by Ben Warner, the brother of Faculty’s CEO, Marc Warner.

Ben subsequently worked on the Conservative party’s 2019 general election campaign, and was later recruited as a data science adviser to Downing Street.

Last month the Guardian reported Cummings had paid more than £250,000 to Faculty via his private consultancy firm, Dynamic Maps. Both parties have declined to explain what the payments were for. Faculty has previously said it would no longer accept political work.

The Scott Trust, the ultimate owner of the Guardian, is the sole investor in GMG Ventures, which is a minority shareholder in Faculty.

The major overhaul of the system follows warnings that it is now reaching less than half of contacts of those who test positive for coronavirus. On average, those working for call centres are reaching just one case a month, research shows.

Councils have warned that many of those being called reject attempts to contact them because they assume the unfamiliar “0300” number is a cold caller. There is also concern that translation services have not been made available to those for whom English is not their first language.

Those working for the call centres, run by Serco and Sitel, have said the job was akin to being “paid to watch Netflix”. Under the overhaul, around 6,000 of the 18,000 call handlers will be axed.

Instead, councils will be encouraged to send their own workers out to chase up the contacts of infected cases who have failed to respond to phone calls.

The changes follow warnings that the safe reopening of schools next month depends on improving the performance of test and trace.

Research by the Independent Sage group of scientists suggests that the test and trace workforce of 25,000 staff – which includes around 7,000 clinical case workers and health protection staff – reached fewer than 52,000 close contacts of coronavirus cases in two months.

Under the overhauled programme, council teams can track down anyone who cannot be reached by the national system after 48 hours to tell them to self-isolate. Every council will also be told they can have “dedicated ring-fenced teams” from the national service to help with local contact tracing.

Councils in areas with some of the highest infection rates had already taken matters into their own hands amid concern that the national system was failing to stop the spread of the virus.

Last week, Blackburn with Darwen (see video below) followed Sandwell in sending out its own staff amid frustration at failings in the national system.

I recently spent a weekday afternoon in the centre of Bristol, chasing the small comforts of retail therapy, and trying to do my bit for a few of my favourite shops. The experience was dreamlike, and unsettling: businesses smattered with hazard tape, half-deserted streets, and a rush hour that seemed to amount to little more than a few momentary traffic jams. Most of the city’s office workers, it seemed, were either furloughed or working at home. Normality – whatever that is – was nowhere to be seen; perhaps the most forlorn sight was that of empty buses adorned with faded adverts for films that came out before the pandemic.

The same day, alluding to both Rishi Sunak’s “eat out to help out” scheme and the near silence of our business districts, the front page of the Daily Mail featured the headline “We’ve had our lunch, now let’s get back to work!”. Returning to workplaces in the midst of a possible second wave is a self-evidently dire prospect; big companies from Google to Mastercard have recently told their employees they can remain at home. But here was another example of the idea – indicative of the reflexive Tory belief that we are at heart a nation of idlers, and lately voiced by such fans of the work ethic as Iain Duncan Smith and Kirstie Allsopp – that we might be able to return to the pre-Covid world by a sheer act of collective will.

The truth, of course, is that making that argument in the midst of such a grave crisis is not just cruel and condescending, but a denial of huge changes to working lives that might turn out to be a lasting social shift. Besides, how can you report for work if your job has gone? As the Bank of England projects a fall in GDP of nearly 10%, the list of high street names who are either shedding staff, or warning that they soon will, now extends into the distance: WH Smith, Pizza Express, Dixons Carphone, Boots, Pret a Manger. In the first two days of this month, 4,500 jobs were cut by major British employers.

Independent businesses, too, are staring into the void: I know of cafe proprietors who reckon they are doing no more than 25% of their pre-Covid trade, and pub owners who say that a combination of staffing costs and dismal customer numbers means that they will soon have to give up.

There is another aspect of our mounting jobs crisis that is still overlooked. Contrary to the idea that we are now a nation of shoppers and service industries, Britain still has a manufacturing sector, a lot of it located beyond our major cities, and a rare source of skilled, well-paid employment. And here, the news is not only just as depressing, but replete with the sense of a country at risk of losing something whose absence will make the future even more perilous. Jobs in aerospace, automotive production and much more besides are disappearing at speed; both management and unions are crying out for dedicated help from the government, apparently in vain.

Between its retailing and manufacturing wings, the UK car industry is reckoned to have already lost 11,000 jobs. Given that the prime minister affects a fondness for buses, he may or may not be aware that Alexander Dennis, a manufacturer of double-deckers whose key site is in Falkirk, are set to cut hundreds of staff. Given the awful state of demand for air travel, in such places as Derby, the south Wales valleys, the parts of north Wales just beyond Chester and the industrial cluster that sits on the northern edge of Bristol, aerospace is in equally dire straits. Airbus intends to shed 1,700 British jobs; the figure for Rolls-Royce is 3,000.

The fates of troubled workplaces are closely linked to those of scores of smaller businesses – from the makers of all-important parts, to hotels and caterers. Just about all of the key firms, moreover, are unionised, with cultures far from the kind of flimsy, insecure practices increasingly common in other parts of the economy. Perhaps the most vivid example is the cluster of aerospace factories in Northern Ireland, which extends from the transport manufacturing giant Bombardier to two major producers of aeroplane seats. Prior to March, Brexit was these businesses’ biggest worry; now, what remains of 2020 will fuse those concerns with the misery and anxiety of immediate job losses.

Even if many jobs are shed via voluntary redundancy schemes, that still means a huge loss of opportunity for future generations. Besides, many of the people involved say they expect an initial round of job losses to be followed by more. And in places that have already been through 40 years of deindustrialisation, what will replace the work that is lost? People may just about be able to find work as security guards, Amazon “associates” and delivery drivers, but that is no foundation for the future.

Even if most of the businesses in question are still based on carbon-intensive technologies, if Britain is to get anywhere near a Green New Deal, their workers’ skills will be indispensable, particularly if these companies are to be made more environmentally sustainable. You can’t engineer a greener future if there are no engineers around to do it.

By way of parrying our burgeoning economic disaster, the government has its across-the-board job retention scheme, but after the tapered phase that began at the end of July, it will soon come to a close.

Last month, the government repromoted its plan to give £200m in grants to boost research and development to make air travel “safer and greener”; manufacturers have also drawn on state-backed loans and help with exports. But all this pales in comparison to what is happening elsewhere. France has launched a €15bn (£13.5bn) support plan for its aerospace industry, exchanging government help for faster progress towards greener aviation; Germany has established a €50bn industrial fund aimed at “climate change, innovation and digitisation”.

Both countries, moreover, have systems in place that will continue subsidising short-time working to protect jobs – something the Labour party is now loudly proposing, but the Johnson government still refuses to do.

The prime minister and his allies still profess to be driven by the idea of “levelling up” the parts of the country that are still defined by the economic disasters and Conservative cruelties of the past. But whatever he says is increasingly drowned out by the sound of doors closing and redundancy payments dropping into bank accounts. Between thinly populated town and city centres and factories frantically cutting back their payrolls, we are now in a watershed moment that will decide our economic future, and whether we will retain the industrial strength necessary for a green future, and at least reduce the damage caused by our exit from Europe.

Cries of “back to work!” sound as moronic as the old Tory exhortation to get on your bike and look for a job – not a meaningful answer to an unprecedented emergency, but yet another indication that the gap between rhetoric and reality is now so huge that it is starting to look downright absurd.

• John Harris is a Guardian columnist

The Teignbridge Local Plan, adopted in 2014 and already in action, demands that 621 new homes are built across the district each year until 2033.

You might have even missed the start of an online consultation on flood risk management in Carlisle, the £450,000 spent repairing a flood wall in Hereford, or chancellor Rishi Sunak’s visit to Stokesley, North Yorkshire, to learn about flood alleviation. Meanwhile, 127 employers were given awards for supporting the armed forces, transport secretary Grant Shapps announced £589m to “kickstart rail upgrades across the north”, plans for “congestion-busting” near Swindon were “unveiled” and a monument to the battle at Gallipoli restored.

Twenty years ago, Conservatives accused New Labour of “initiative-itis” and “legislative incontinence”, endlessly moving from one gimmicky announcement to the next. The current government appears to suffer from a similar affliction. Young governments, like young people, are eager to prove themselves and be approved, but short on patience – and ambitious personnel in government departments are keen to demonstrate their “on-message” competence.

This government’s initiative-itis is, of course, also about politicking. (No 10 – not for the first time in its history – is staffed by campaigners in place of policy specialists.) The pandemic and Brexit are both complex policy challenges that our government has proved uniquely ill-equipped to address. Governing by announcement is a way to distract the press and the public from the truth of this incompetence.

Bashing judges, praising veterans and publicising investment in “The North” is red meat to throw at one part of the electorate. Appearing serious about diet, exercise and green transport is the vegan meat substitute for another.

This permanent campaigning mindset is how we do government now. The phenomenon of the “permanent campaign” was named in a 1980 book by the American journalist and political adviser Sidney Blumenthal. His argument was that government decision-making was increasingly based on assessments not of what might be good policy, but of what would look good, keep sponsors on board and appeal to key voters. Government was reduced to “an instrument designed to sustain an elected official’s popularity”. The tail was wagging the dog.

Boris Johnson’s permanent campaign is symptomatic of a deeper change in our politics. Even before the Tories won the 2019 election, Britain was becoming a “post-democracy”. One of the founding propositions of 20th-century mass democracy has been that government should act not for the few, but for everyone. Inherent within this idea is a concept of “the public interest”, or “the common good”. We might not be sure what that is – and we certainly won’t be able to agree on it. But in a democracy, so the argument goes, we can come together politically and our representatives will broker a settlement between the competing interests of various social, economic and political groups.

Such groups are now more complex and less stable. People’s occupational, economic and even geographic locations are more likely to vary over a lifetime. Connections between distinct social interests and their political representatives have been attenuated. In our post-democracy, aggregations of individual tastes and preferences, expressed through opinion polls and surveys, have displaced parties and other civic forums as the means by which government comes to understand “the people”. Instead of public interest, we now find public opinion. Indeed, this government spent more on polling between January and May this year than in the whole of 2019.

In place of elections or collective deliberation, opinion polls serve as an ongoing referendum on the government, providing constantly updated information about people’s attitudes and reactions, vital for the formation of campaign strategies. There is an argument that this has been a good thing. It means that government is continually apprised of and attentive to public opinion, able to incorporate public concerns within policy-making. If people want a war memorial repaired, and if, in search of their support, politicians repair it, then perhaps democracy is working well.

A counter-argument is that governments are no longer experts in governing. Instead, they can think only according to short-term campaign strategies, choosing to do what is most popular rather than what is best for people. As the academic and former government adviser Patrick Diamond puts it, this damaging short-termism becomes endemic. A government focused on “eye-catching” initiatives, as we have seen, is not preparing for or equipped to deal with a pandemic.

In our post-democracy, “public opinion” is not only a source of policy. It’s also an object of policy. Public attitudes, outlooks and behaviour are now a central focus of government policy – the goal of which is not to change the situations people find themselves in, but to change people’s feelings about their situations. Post-democratic government is not unconcerned with what you think or want or need. But it doesn’t require your participation. It just needs to know how you might respond to the various things it might do to you.

What the prime minister’s chief adviser, Dominic Cummings, understands that most politicians do not is that polling, surveys and focus groups don’t tell you what people think or what public opinion definitively “is”. They provide a rough map of possibilities and probabilities of what people might come to think, depending on various potential scenarios and actions. In the leave campaign, different versions of adverts were tested, refined and retested in what Cummings has called a “constant iterative process” and combined with feedback from polls and focus groups. In government, this approach has turned the Tories’ permanent campaign into a permanent experiment, the endless measuring and testing of means for managing and shaping political attitudes.

We are the subjects of this ongoing experiment, the single data points in a larger firmament. That, in post-democracy, is what constitutes citizenship. And while it won’t save summer, it might – as you enjoy your half-price Sunak-branded meal next Monday, Tuesday or Wednesday – change how you feel about its loss.

• Alan Finlayson is professor of political and social theory at the University of East Anglia

Four years ago an unholy alliance of St Nicholas Hospital, a Christian alms house in Salisbury, and the Earl of Radnor’s Longford estate just outside the city announced plans to build 100 houses on a meadow beside a canalised branch of the River Avon. According to the Environment Agency’s flood map of Salisbury, the meadow does not flood.

The Salisbury Journal columnist Annie Riddle gamely published a photograph of the meadow. It showed a lake. The Barbour’d masses revolted. The planning application was withdrawn, although the thwarted developers’ PR operative, a delusionist with a promising political future, continued to insist that there was no flood. Under the new zoning system, another splendid gift from the impressive Robert Jenrick, the housing secretary, he would get his way without a fight.

Such events are doubtless a British norm. There is an increased awareness that the back yard in Nimby is not mine alone. The more land that is sacrificed to infrastructural monoliths, to buildings, to aggregates, to tarmac and concrete, the more pervious will these watery isles become to floods, bursting dams and associated disasters, unnatural disasters, micro-Anthropocene disasters. The back yard with cars in trees, collapsed bridges, floating furniture and societal chaos belongs to everyone. It’s ours. Just as local knowledge is ours — not that it counts for anything.

The peculiarities of places are to be obliterated by startlingly crude standardised criteria devised by infantile vandals in their think tank sandpit. A levelling is to be applied with disregard for specifics. This is not merely political swinishness. It is an aesthetic crime. The absolute disregard for genius loci is a triumph of philistine stupidity.

If, the “thinking” goes, it’s good for a Cumbrian fell, then it’s obviously good for a Cambridgeshire fen. Not that fells and fens will be targeted; they are too far from the southeast, where uncontrolled development or “regeneration” will achieve a level of overcrowding and overheating yet unseen because, for the past seven decades, statutory regulations have struggled to prevent it.

This government of all talents will one day no longer be with us, reduced mercifully to dust. But its squalid legacy will remain. Not simply in boorish schemes and mediocre designs, but in ruined lives, in the RAF favelas, Brit bivouacs and bidonvilles that will grow up to accommodate the disenfranchised and unsheltered who can seldom reach even the first rung of the “housing ladder” — the very epithet reeks of the expectation of entitlement. They may have been born in Blyth and made in the Royal Navy and served their country selflessly, but the nearest they’ll get as veterans to that ladder is a sleeping bag beside a hot-air vent.

And, supposing they do grapple with it, they will very likely be looking at a windowless “home” of 20 square metres, which is about two-thirds the size prescribed by the Parker Morris committee nearly 60 years ago and adopted by both main parties in an era when not to build social housing was unthinkable. There existed a cross-party consensus favouring a sort of nationalised noblesse oblige. That consensus was destroyed by Margaret Thatcher. It was in the long term the most divisive of her many divisive acts. The right to buy was the right to buy votes and steal someone else’s hope.

It also signalled the capitulation of environmental dirigisme to the god of the market. The UK’s rules were both stricter and more rigorously applied than those of the southern European countries it now seeks to emulate. They were also applied in a different manner.

Because, for many decades from the 1860s onwards, the wealth in English (though not Scottish) cities tended to live outside those then reeking cities, they had no particular investment in them domestically. With the decline of heavy industry in town centres and the gradual gentrification of London from the 1960s onwards and, later, of the main English cities, things changed. Those cities became centripetal in the way that Paris and Madrid are. Class clearance freed up space for celebrity baristas and lifestyle-size influencers.

Laissez-faire planning was the sole form of planning, bridge fetishes excepted, that appealed to the former mayor of London. If a volume builder wants to do it, let it be — build away the blues: that evidently was the mayoral mantra. The volume builder knows his stuff and he is a very great contributor, a valued donor. Laissez-faire planning — you might call it no planning or sloth planning — is this government of talents’ speciality and is to be non-applied to everything, everywhere.

And what London thinks today, Manchester thinks tomorrow. Save that it won’t work like that. Giving free rein to volume builders is a route to uglification on the most astounding scale. These people are the barrow boys of the built environment. They are enemies of architecture. Indeed, they seldom employ architects: do they get treated by amateur dentists and Sunday oncologists? All that matters is the bottom line. The little people who will subsist in the valued donors’ creations don’t count. Indeed, they would do well to get littler still, to fit into their undesigned crates and hutches.

Jonathan Meades has written and performed in more than 50 television programmes on urbanism, architecture and topography. His next book, Pedro and Ricky Come Again, will be published next spring

Nimbys may have lost the battle of the back yard
When Nimby, an American acronym for “not in my back yard”, entered the British lexicon it quickly became shorthand for the tensions between selfishness and the public good, writes Nicholas Hellen.

Margaret Thatcher’s environment secretary, Nicholas Ridley, declared his support for a swathe of new towns: “Our English countryside is one of the most heavily manmade habitats in Europe. To make it into a green museum would be to belie its whole history.”

It swiftly emerged that he had objected to new homes near his Cotswolds rectory and was a Nimby too.

Now as Robert Jenrick, the housing minister, takes charge of Boris Johnson’s pledge to “Build, build, build”, Nimbys are once again in the crosshairs.

In the 30 or so years since, the population of the UK has grown by more than 10 million to 67.9 million.

This time round, the battles will not be confined to fields, hedgerows and herbaceous borders. Nor will Nimby remain code largely for white middle-aged Tory voters keen to preserve property values and to exclude people different from themselves.

Under Jenrick’s proposed planning reforms, land across England will be divided into three categories. New homes, schools, shops and offices would be automatically permitted in “growth” areas, and in urban and brownfield sites, known as “renewal” zones, plans would be given “permission in principle”.

The most affluent areas, often Tory-controlled, will be ordered to release the most land for housing, and local control over the rate of building will in effect be removed. Disused commercial premises can be turned into homes without the need for planning permission, creating fears that empty shops will become the slums of tomorrow.

New alliances are being created. Rich developers may pose as friends of the homeless, while wealthy homeowners make common cause with environmentalists. In the era of identity politics, we are all Nimbys now.

With the Town and Country Planning Act 1947, the Attlee government nationalised development rights. That is, whatever benefits that might come from developing land belong in principle to the public. When planning permission is granted, a portion of those rights is transferred to a landowner.

This gives government a lever that can be used for the public good. A vast quantity of latent wealth comes from the fact that planning consent can increase the value of land – according to one study, by a stupendous 275 times. The most realistic way of funding the genuinely affordable housing that this country needs is from sharing this windfall.

Last week, the Ministry of Housing, Communities and Local Government published a white paper, Planning for the Future, which promises, in the prime minister’s words, to “tear down” the system created by the 1947 act and “start again”. The crucial question is whether it will, as the government promises, end the generational divide between housing haves and have-nots.

Its most eye-catching proposal (for England only, as planning is devolved to Scotland, Wales and Northern Ireland), is to designate three types of land, labelled “Growth”, “Renewal” and “Protected”. These would be specified in the plans drawn up by local authorities. Where land is designated for “growth”, outline permission would be automatically granted for development of the type set out in the plan.

This would reduce the public debate and scrutiny that currently come with planning applications and focus on the drawing up of local plans, which is currently a slow, sometimes opaque process. The white paper promises to make the preparation of these plans both faster and more open to public consultation, at the same time as their significance is greatly increased.

The proposals include a “fast track to beauty”, whereby local authorities, with some direction from national government, produce design guides that translate the “basic characteristics of good places into what works locally”. Applications that conform to these guides would be speeded through the planning process. Digital technology would make planning more accessible to the public, basing it on “data” rather than “documents”.

Perhaps most importantly, section 106 agreements, which are negotiated between planning authorities and developers and are the main vehicle for obtaining affordable housing and other public benefits, would be replaced by an enhanced community infrastructure levy (CIL), a cash sum that would be fixed according to national rules. Viability assessments, whereby developers can plead for reductions in their obligations in order to make their schemes stack up, would be abolished.

The idea of these changes, says the government, is to speed up the planning process by slashing red tape. Much of the current system relies on opinion and interpretation, which adds uncertainty, risk and complexity. A clearer, more rules-based system would benefit smaller building businesses, currently squeezed out by the construction giants.

The communities secretary, Robert Jenrick, promises that these proposals will help “recreate an ownership society in which more people have a dignity and security of a home of their own”, where “all pay a fair share of the costs of infrastructure and the affordable housing existing communities require and where permissions are more swiftly turned into homes”.

Looked at neutrally, some of these aims seem valid. If local plans are truly to be exposed to fuller public debate, that would be a good thing. The discretionary nature of the planning system is indeed wasteful, does not always lead to the best possible results and tends to favour those with the resources to shout most and argue hardest. The abolition of viability assessments would be positive. In theory, the improved CIL could be the lever by which those still-nationalised development rights could benefit the public.

But Jenrick saved the Tory donor Richard Desmond millions of pounds in CIL by overriding a planning decision made by the London borough of Tower Hamlets. This episode doesn’t give confidence that Jenrick is the best man to preside over a more equitable CIL, nor that the Conservative party is not primarily driven by its property business supporters.

The white paper also assumes a high degree of administrative competency, for which this government has not so far been notable. The new super-powered local plans will require resources and organisation that are not yet in evidence. The design codes will have to address those tricky issues that really make places successful, such as layout, density, construction quality and mixtures of uses and tenure.

It is possible to see how council estates could be designated as areas for “growth” and therefore vulnerable to comprehensive redevelopment. The CIL proposals come with exceptions that could easily be used to erode their effectiveness. The promise of more equal housing would therefore fade away.

It is a persistent Tory fantasy that deregulation of planning will, through the operation of supply and demand, reduce prices. It does not and will not: volume housebuilders slow down their rate of building if the price is in danger of falling. The European countries with more balanced and successful housing markets than ours, such as Germany and the Netherlands, have more proactive national and local governments, which do such things as assemble land and initiate building. Even if this white paper works, which is doubtful, it will take a lot more to address the housing crisis.