Firms have no right promising to guarantee staff lifetime incomes when they cannot guarantee their own solvency

"Defined benefit pension plans are the bedrock of retirement security", wrote a group of Democratic senators to President George W Bush in 2006. "This policy", referring to a Department of Education directive to restrict contractors' use of defined-benefit pension plans, which guarantee a level of lifelong annual payments to retirees, generally calculated as a percentage of their final salary, "is a direct attack on Americans' retirement security".

It's easy to see why repelling such an attack has become a political priority. According to a bulletin of the United Federation of Teachers in 2004: "This benefit is guaranteed by the employer who assumes all of the investment risks... That's why UFT members with their defined-benefit plan, social security and a supplemental defined contribution plan have become the envy of their friends and family." Indeed.

And it's not just union-backed Democrats who want to see DB plans expanded. Republican Senator Chuck Grassley said in 2003: "We ought to be looking at ways to encourage the sponsorship of defined-benefit pension plans. Funding rules that are too harsh will not encourage DB plans."

Fast forward to 2009, and US, as well as UK, private and public DB pension plans are in crisis.

Whereas Senator Grassley attacked DB plan funding rules that were "too harsh", reality has shown that they were anything but. The US Pension Benefit Guaranty Corporation, a government body that insures private DB schemes, collecting fees to (in theory) cover the liability, reported a massive $33.5bn deficit in May, which is set to soar past $40bn with the collapse of General Motors auto parts supplier Delphi. While pensioners typically take a haircut in the process of a PBGC takeover, United Auto Worker pensions at Delphi will receive unprecedented special treatment - they will be made whole by government-controlled GM, Delphi's former parent. The stench of politics is overwhelming.

When created in 1974, PBGC's congressional backers of course assured the public that no taxpayer dollars were at risk. DB plans were thus to be miraculous. They were to be invested in all manner of risky stocks, bonds, venture capital, private equity, real estate and hedge funds, yet nonetheless offer generous retirement annuities that were riskless to everyone involved - the retirees as well as their fellow citizens insuring them.

State-run DB plans are in calamitous shape. Asset values at the two largest such funds in the US, the California Public Employees' Retirement System and the California State Teachers' Retirement System, fell by over $100bn in the year to June 30. Calpers, whose asset values plummeted 23.4% over that period, had estimated last autumn that a 20% decline would take its funding status - assets divided by liabilities - down to 68%. With Calpers' credit rating now under threat, so is its income from fees charged to local governments to guarantee their borrowing. States and their pension funds are joined at the hip.

Calpers is arguably the most politically active sovereign wealth fund in the world, routinely steering investments on the basis of explicit political agendas - including, in the words of former California state treasurer and Calpers board member Phil Angelides, "positive pressure" on foreign governments. The Los Angeles Times and New York Times have both raised questions about venture capital and other investments which appeared to be motivated by political donations.

Governor Arnold Schwarzenegger tried in 2005 to push California public employees into a defined-contribution pension plan, in which retirement benefits would be determined solely by future returns on contributions, which the employees and the state would make into their individual accounts, but retreated in the face of powerful union opposition.

Other state public pension schemes nationwide have become "pay-to-play" political cesspools, with firms vying for state investment business using highly paid lobbyists and middlemen. In May, New York State Attorney-General Andrew Cuomo reached a settlement with private equity firm Carlyle Group under which the latter agreed to cease such activity and pay the state $20m.

The practice of "pension spiking" - state and municipal bodies granting outsized pay increases to employees a year before retirement in order to super-size their pension benefits - has become commonplace, contributing to the growth of unmanageable liabilities. Professors Robert Novy-Marx and Joshua Rauh estimated that the 116 largest US state and local-government DB pension schemes were underfunded by an astounding $3.12 trillion, or well over three times the states' combined municipal debt.

The situation in the UK is no brighter. Consultancy Watson Wyatt put the funding status of UK local-authority schemes at between 50% and 60%, which would make Calpers look positively healthy.

Big US and UK companies, such as IBM and BP, have been addressing their own funding challenges by replacing DB schemes with DC schemes for new employees. Unions have naturally reacted with anger. Peter Skyte, national officer at Britain's largest union, Unite, accused companies of "using the pretext of the economic crisis to railroad future pension benefits". He predicted "a race to the bottom" among companies to phase out DB schemes.

I hope he's right, and that the public sector follows suit - which will require acts of unprecedented political courage, bankruptcies, or both. Companies have no right promising today's employees guaranteed lifetime incomes when they have no way of guaranteeing their own solvency. States are different, naturally, in that they can always inflate away their promises by printing money, which makes the problem even worse.

But should individuals be expected to bear the full financial risks of supporting themselves in old age? In rich nations like the US and UK, providing a basic state pension is not the problem. The problem is subjecting one part of the citizenry, generally non-public-sector and non-union employees, to the risk that the investments intended to support the other part in retirement comfort won't pan out as wished. Such a system clearly encourages excessive risk-taking and political corruption, and undermines personal responsibility.

The argument remains that individuals are not sophisticated enough to manage their own investments. There are two responses to this. First, DB pension providers that can fob off losses on to the wider public certainly don't make the challenges of prudent investment disappear - in many cases they magnify them. Second, individuals who can't or don't wish to make investment decisions can be herded into low-risk funds. From society's perspective, there is no free lunch in having a DB fund manager promising them higher returns with higher risk investments - the costs of the failed promises just get passed on to neighbours not fortunate enough to be in on the racket.