On Thursday, the Congresswoman declined the request while poking fun at Perry's last debate gaffe when he blanked on the third government agency he planned to shut down.
"He did ask if I could debate here in Washington on Monday. It is my understanding that such a letter has come in. Monday I'm going to be in Portland in the morning and I'm going to be visiting some of our labs in the afternoon, that's two, I can't remember what the third thing is I'm going to be doing," Pelosi said.
The governor isn't catching much of a break these days as his campaign continues to hit some rough spots. But the rocky road seems to be pushing Perry to attack.
Perry responded on Thursday to Pelosi with a couple tweets. In the first, he asked Pelosi if she had forgotten ongoing insider trading - referring to a 60 Minutes story alleging she made money on insider Congressional information. Forty minutes later, he tweeted again wondering if she'd like to forget passing a 2,000 page health care reform without reading it.
Americans haven't forgotten Perry tweeted. But it seems Americans are having a tough time forgetting Perry's gaffes too.
Perry isn't just fighting with Pelosi these days. He's taking aim at the president pretty forcefully too in New Hampshire Wednesday.
"He's called us soft. He's called us lazy," Perry said.
We heard the same line in Iowa.
"Americans do not lack ambition or imagination -- it is Washington that has failed us," Perry told voters there.
Even if the lazy comment is taken somewhat out of context, it's classic Perry to fight against DC instead of other Republicans, and he's doing it on TV too.
"You believe that? That's what our president think's wrong in America -- that Americans are lazy? That's pathetic," he says in one TV ad.
It's the latest in a blitz of Perry ads in Iowa, getting heavy air play and good reviews but not driving up his poll numbers and there's probably a reason why.
"They're really good ads. But that's very scripted as we know and he's very much at ease in that context, but that seems to clash with what we see on a debate stage. And if you get to the point where you have trouble on a debate stage or in any other context like that, it becomes not just an exception to the rule, but confirmation of an existing narrative," Drake University Iowa Political Scientist Dennis Goldford said.
He makes light of it.
"I am the first to admit I am not the most polished candidate out there," Perry said.
But it's the one thing almost all voters know about him.
"This is honestly actually the first I'd heard of him," Iowa voter Jamie Adamson said.
But even Adamson, who came to a Perry event, admitted he'd heard of the flubs.
"Yeah, I did hear about that," he said.
"When you're at four or eight percent in the polls, you don't have any political capital to attack Mitt Romney or Herman Cain, so you're going to attack Obama or Nancy Pelosi -- that's safe and try to be more strident than your opponents," KTRK political consultant Dr. Richard Murray said.
He says it's not just polls. Perry miscalculated on some of the issues too and is still recovering in his estimation from his immigration stance.
