Leaving her in her place would be "very dangerous for the country," Gingrich added.
Last Thursday Pelosi denied that she knew the CIA employed waterboarding, a harsh interrogation technique that simulates drowning. The speaker said the CIA "misinformed" her when they briefed her in September 2002, the month after Guantanamo Bay detainee Abu Zubaydah was subjected to waterboarding.
Gingrich said on "GMA" that Pelosi is the one who misinformed the American people.
"What she said Thursday was a stunning dishonest statement about a major American institution that has a key role in our survival," he said. "I think the Democrats should get a new speaker."
In his weekly newsletter to supporters published in the conservative publication Human Events, the prominent GOP figure said Democrats "owe it to their country" to replace Pelosi as the speaker.
"The person who is No. 2 in line to be commander in chief can't have contempt for the men and women who protect our nation. America can't afford it," Gingrich wrote.
Pelosi's office called Gingrich's suggestion "ludicrous" and noted that no Democrats are calling for Pelosi to be ousted.
But Gingrich is just one in the growing chorus of Republicans lashing out at Pelosi.
The Republican National Committee has even produced a video ridiculing the press conference where Pelosi accused the CIA of lying. Set to the James Bond theme music, the video picks on several of Pelosi's statements and ends with the line "Lack of leadership. Democrats galore."
Republicans Lash Out at Nancy Pelosi
Rep. Peter King, R-N.Y., also echoed Gingrich's statements.
"For the person second in line to be president of the United States to be making this type of charge is really shameful, and to me, crosses any boundary I know," he said.
Rep. Steve King, R-Iowa, also joined the bandwagon, saying Pelosi is "actively undermining our national security," and demanded that she provide proof that she was misled.
House minority leader John Boehner, R-Ohio, said Pelosi should either produce evidence that the CIA was indeed lying, or apologize.
"She owes our intelligence community an apology," Boehner said in a Fox interview. "I think the ball is in the speaker's court. I think she needs to come forward, either present evidence or do an apology and let's get this behind us."
But unlike Gingrich, Boehner didn't call for Pelosi's resignation.
Democrats -- including the most powerful person in the party, President Obama -- have thrown their support behind the embattled speaker.
While announcing new emissions standards for U.S. automakers, the president singled out Pelosi in his praise.
"Speaker Nancy Pelosi, who has just been cracking the whip and, you know, making Congress so productive over these last several days. We are grateful for her," Obama remarked.
The No. 2 Democrat in the House, majority leader Steny Hoyer, also jumped to Pelosi's defense.
"Let me be clear, so there is no misinterpretation of my view: I believe the speaker," Hoyer said.
Pelosi Says CIA Didn't Tell Her About Waterboarding
Pelosi's involvement came into the spotlight after Obama's Department of Justice last month released the memos written by Bush administration officials providing legal justifications for harsh interrogation techniques that are considered torture by the United Nations and by Obama.
Pelosi has maintained that she was briefed only once about the techniques, but that CIA officials did not tell her waterboarding was actually being employed. Terror suspect Zubaydah was subjected to the waterboard 83 times in August 2002, the month before Pelosi was briefed about enhanced interrogation techniques.
A Director of National Intelligence report released in May directly contradicted Pelosi's statements, saying the speaker was indeed briefed about the techniques used on Zubaydah.
Last week, CIA director Leon Panetta sent a note to employees in response to Pelosi's accusations.
"Let me be clear: It is not our policy or practice to mislead Congress. That is against our laws and our values," the memo said. "As the agency indicated previously in response to congressional inquiries, our contemporaneous records from September 2002 indicate that CIA officers briefed truthfully on the interrogation of Abu Zubaydah, describing 'the enhanced techniques that had been employed.'"
The controversy is energizing Republicans at a time the party is struggling to find a unifying message. RNC chairman Michael Steele proclaimed yesterday that the party is on the way back.
"The honeymoon is over, and it's time for us to speak truth to power," he said. "The Republican comeback has begun."
Slideshow archive | ABC13 wireless | Help solve crimes