The union Unite which organized the protest and has 180,000 members working in financial services, has complained of a "deafening silence" from the Conservatives over how to tackle the current economic crisis.
Unity says the Tories are cashing in on the credit crunch.
The row has been sparked by disclosures that a number of Tory donors have been involved in short-selling banks which have either had to be taken over or nationalised by the government.
Unite Joint general secretary Derek Simpson said today: "This exposes the Tories for what they really are. They are funding their party and maybe even their election campaign on the misery of thousands of British families, who could lose their homes and jobs because of the spivs and speculators.
"Thousands of staff at banks like HBOS and LloydsTSB fear for their jobs but the Tories seem more interested in taking money from the culprits of the credit crunch than helping the victims.
The GMB - Britain's General Union meanwhile demanded that the Financial Services Authority, the regulator, launch an inquiry into the Tory donors who are connected to the short selling of bank shares.
Paul Kenny, the GMB's general secretary, said: "GMB is aware that there is a major inquiry under way in the US to establish if rules were broken during the recent meltdown in the financial sector. GMB would like the FSA to establish what meetings these donors had with Tory politicians and what they discussed at the meetings.
"Did these meetings influence the recent Tory defence of short trading which brought down the Bank of Scotland?"
The following is from the BBC.
Bank staff protest at conference
Bank workers from across the West Midlands have protested outside the Conservative Party Conference.
The protesters, backed by the union Unite, claimed that financiers caused the economic downturn and were the same people who funded the Tory party.
Gerard Coyne, the regional secretary of Unite, said people were worried about losing their jobs.
The protesters gathered outside the ICC in Birmingham on the last day of the conference.
The protest followed Tuesday's announcement that US finance firm GE Money would close its Wolverhampton office with the loss of about 100 jobs.
Of the 2.5m workers across the Midlands, about 425,000 people work in the finance sector.
Mr Coyne said: "We're not at the end of this yet and in these circumstances it's really hard to predict the number of job losses.
"It could be in the number of tens of thousands."
He added: "The state of the market is such that any rumour about the future, particularly of a banking institution or financial institution, can absolutely run rife."