Obama promised in his victory speech earlier this month to engage with Romney following their bitter campaign and consider the Republican's ideas.
"In the weeks ahead, I also look forward to sitting down with Gov. Romney to talk about where we can work together to move this country forward," Obama said at the time.
Obama aides said they reached out to Romney's team shortly before Thanksgiving to start working on a date for the meeting. The two men will meet in the White House's private dining room, with no press coverage expected.
In the days after his loss, Romney told top donors that the president was re-elected because of the "gifts" Obama provided to blacks, Hispanics and young voters, all of which are core Obama constituencies.
"The president's campaign, if you will, focused on giving targeted groups a big gift," Romney said.
Many Republican officials, eager to move on quickly after the loss, disputed Romney's comments and urged the party to focus on being more inclusive.
White House spokesman Jay Carney said Obama was looking forward to having a "useful discussion" with his former competitor. But he said there was no formal agenda for the lunch.
While in Washington, Romney will also meet with his former running mate, Wisconsin Rep. Paul Ryan, according to a Romney campaign aide. Ryan is back on Capitol Hill, where he's involved in negotiations to avert a series of automatic tax increases and deep spending cuts that have come to be known as the "fiscal cliff."
Much of that debate centers on expiring tax cuts first passed by George W. Bush. Obama and Romney differed sharply during the campaign over what to do with the cuts, with the Republican pushing for them to be extended for all income earners and the president running on a pledge to let the cuts expire for families making more than $250,000 a year.
The White House sees Obama's victory as a signal that Americans support his tax proposals.
Obama and Romney's sit-down Thursday will likely be their most extensive private meeting ever. The two men had only a handful of brief exchanges before the 2012 election.
Even after their political fates became intertwined, their interactions were largely confined to the three presidential debates.
Romney has virtually disappeared from politics following his loss in the Nov. 6 election. He's spent the last three weeks largely in seclusion at his family's southern California home. He has made no public appearances, drawing media attention only after being photographed at Disneyland in addition to stops at the movies and the gym with his wife, Ann.
Former aides confirm that Romney is expected to move into an office at the Boston-area venture capital firm Solamere Capital. The firm was founded by his oldest son, Tagg Romney, and Spencer Zwick, who served as his presidential campaign's national finance chairman.
It's unclear what role, if any, Romney will play at the firm. Former aides said Romney was subletting office space from Solamere.