"As leader of my party I must accept that as a judgment on me," Brown said, referring to Labour's poor showing in the election.
Brown's startling news conference came as the Conservatives, who won the most seats in the election but not a majority in Parliament, were already holding talks with the Liberal Democrats. Lawmakers had said those talks stalled over differences on key issues, including reform of the voting system, a Liberal Democrat demand.
In a statement outside his office at 10 Downing Street, Brown said Liberal Democrat leader Nick Clegg had asked to begin formal coalition talks with the Labour Party and the two could form a center-left alliance.
Clegg had previously said Brown's departure would likely be a condition of any deal with Labour.
"There is a progressive majority in Britain, and I believe it could be in the interests of the whole country to form a progressive coalition government," Brown said.
Cameron's center-right Conservatives won 306 of the 650 seats in the House of Commons, 20 short of a majority. Brown's center-left Labour won 258 and the center-left Liberal Democrats took 57 seats. Other smaller parties took the rest.
Brown said he hoped a new Labour leader would be appointed at the party's annual convention in September. Foreign Secretary David Miliband and Education Secretary Ed Balls will likely be leading contenders to succeed Brown as party leader.
The pound fell nearly 1.5 cents against the dollar after Brown's statement, trading at $1.4866 late Monday, reflecting some fear of Labour's continued presence in the government.
Britain has a record 153 billion-pound ($236 billion) deficit that the Conservatives have pledged to tackle faster than Labour.