It was like being a fly on the wall, a silent party to a chat between friends, moderated by presidential historian Jon Meacham.
"Members of Congress are primarily motivated by keeping their seat and it's getting worse and worse," Obama said.
"The responsible center in American politics has disappeared," Baker said.
The two men talked politics and policy in a way you don't hear in Washington, and the difference between 1980s and now.
"Things were different," Baker said. "I think that perhaps a few more things got done on a bipartisan basis."
WATCH: Reporter Tom Abrahams takes you inside the gala tent
"By the time I take office, what you increasingly have is a media environment in which if you are a Fox News viewer, you have an entirely different reality than if you are a New York Times reader," Obama said.
"You have the advent of the internet, and that really makes it easy to be divisive," Baker said. "Divisiveness sells."
The former president said the biggest challenge over the next 20 years may be a return to civil conversation.
While he wasn't in attendance Tuesday night, former President George H.W. Bush also had some time with Obama this afternoon at his residence, rekindling what spokesman Jim McGrath called an already warm friendship.
Follow Tom Abrahams on Facebook and Twitter.