David Cameron’s pledge to build a property-owning democracy is called into serious question by a landmark survey revealing that almost four in 10 of those who do not own a home believe they will never be able to do so.

According to an exclusive poll for the Observer on attitudes to British housing, 69% of people think the country is “in the throes of a housing crisis”. A staggering 71% of aspiring property owners doubt their ability to buy a home without financial help from family members.

More than two-thirds (67%) would like to buy their own home “one day”, while 37% believe buying will remain out of their reach for good. A further 26% think it will take them up to five years.

With affordable homes in short supply and demand for social housing rising, more than half of Britons cite immigration and a glut of foreign investment in UK property as factors driving prices beyond reach.

The findings cast doubt on the prime minister’s claim before last year’s general election that Tory housing policies would transform “generation rent” into “generation buy”. In April last year, as he launched plans to force local authorities to sell valuable properties to fund new “affordable homes”, Cameron said: “The dream of a property-owning democracy is alive and well and we will help you fulfil it.”

The poll – which found that 58% of people want more, not less, social housing as a way to ease the crisis – comes as the government’s highly controversial housing and planning bill returns to the Commons on Tuesday.

The bill will force councils to sell much of their social housing and curb lifelong council tenancies, introducing “pay to stay” rules that will force better-off council tenants to pay rents closer to market levels. Described by housing experts as the beginning of the end of social housing, the bill has been savaged by cross-party groups in the Lords. They have inflicted a string of defeats on ministers and forced numerous concessions.

The government’s flagship plan for “starter homes” has also been widely attacked on the grounds that the properties – which in London will cost up to £450,000 – will not be affordable.

With local elections and the London mayoral election on Thursday, ministers now face the dilemma of whether to back down and accept many of the Lords’ amendments to the bill or face legislative deadlock.

On Saturday night Labour’s candidate for mayor of London, Sadiq Khan, who is putting plans for more affordable housing at the heart of his campaign, described the bill as “the most extreme in terms of housing in a generation”.

The party’s housing spokesman, John Healey, said: “Opposition to this bill now comes from across the board: from housebuilders, housing experts, charities and even Conservative ministers’ own council leaders, MPs and peers. It seems that government ministers are alone in thinking their bill is fit for purpose when it comes to tackling our housing crisis.

“Despite the string of concessions the government was forced to make, this remains an extraordinary and extreme bill that will lead to a huge loss of affordable homes to rent and buy. Ministers need to listen to the opposition coming from all sides and back down on their damaging housing plans.”

Khan’s rival, Tory candidate Zac Goldsmith, has pledged to build 50,000 homes a year by 2020 and to ensure that a significant proportion of new homes will be available only for rent.

The research by Opinium was conducted only five days after the Panama Papers revealed how a substantial portion of London’s most expensive properties are now owned by foreigners via offshore companies.

When asked what measures they would like to see implemented to restrict foreign ownership of British properties, 71% backed a ban on foreign owners buying British properties as investments or as buy-to-let; 60% backed higher taxes for foreign buyers involved in buy-to-let schemes.

A spokesman for the Department for Communities and Local Government said the government had unveiled the “most ambition vision for housing in a generation”, doubling the housing budget and investing £8bn to deliver more than 400,000 affordable homes.

“There are billions of pounds locked up in local authority housing assets so it is only right that when higher value homes become vacant they are sold to build new homes that better meet local needs,” the spokesman said. “It means every home sold will be replaced with at least one new affordable home and two for one in London.”