Carter said "too many people are at ease" with black unemployment rates that exceed the national average and schools in some places that he described as basically still segregated.
Carter, 89, was the first president to speak at the Lyndon B. Johnson Presidential Library in Austin, which is holding the three-day summit to mark the anniversary of the landmark 1964 law that banned widespread discrimination against racial and ethnic minorities and against women.
"We're pretty much dormant now," Carter said. "We accept self-congratulations about the wonderful 50th anniversary - which is wonderful - but we feel like Lyndon Johnson did it and we don't have to do anything anymore."
The unemployment rate for blacks was 12 percent in February, compared with 5.8 percent for whites.
Carter, who grew up in Georgia, recalled being influenced by black culture and calling for the end of racial discrimination after he was elected governor of that state in 1970. But four decades later, Carter expressed regret at racial and gender inequalities that he says are persistent.
The 39th president touched on wage gaps between women and men and reiterated his support for gay marriage. During a wide-ranging interview to a packed auditorium, Carter also chalked up loosened rules on political campaign contributions as partly the reason for a new era of gridlock in Washington.
"What happens is that the political environment is flooded with money since the Supreme Court made that stupid decision," Carter said, a reference to the high court's 2010 Citizens United ruling.
"A lot of that money pours into the campaigns is spent on negative commercials. ... So by the time the election's over, you have a polarized Texas or polarized Georgia, red and blue states. Then, when people get to Washington, they don't trust each other," he said.
President Barack Obama is scheduled to give the keynote address Thursday. Bill Clinton will speak Wednesday, and George W. Bush will be the event's final speaker Thursday.
George H. W. Bush, 89, is the only living president not attending the summit. He said in a statement that he regretted that he couldn't attend.
Johnson's presidency is often viewed in the dark shadow of the Vietnam War, but the library believes his legacy deserves as much attention for the Texan's victories on civil rights.
The summit began with former Republican Mississippi Gov. Haley Barbour and San Antonio Mayor Julian Castro, a fast-rising Democrat and top surrogate of Obama, urging Congress to tackle immigration reform before the end of the year.
"The stupidest thing we can do economically is make them leave. We don't have anybody to replace them," said Barbour, referring to the estimated 11 million immigrants who are in the country without legal documentation. "So the impracticality of sending them home should be obvious to everyone."
Their discussion was interrupted by a woman in the crowd shouting she was a so-called DREAMer - a young person who immigrated illegally into the United States - and calling on Castro to urge Obama to stop deportations of families.
No one removed the woman, who began shouting again when the panel was over.
Castro, the keynote speaker at the 2012 Democratic National Convention, did not respond to the woman but later said he was troubled by families who are deported after minor crimes such as traffic stops.
"My hope is that his administration will go about it in a different way. I'm not comfortable with the number of deportations," Castro said.
The library also has a "Cornerstones of Civil Rights" exhibit that features the original Civil Rights Act of 1964 and Voting Rights Act of 1965, both signed by Johnson, and a copy of the 1863 Emancipation Proclamation signed by President Abraham Lincoln that declared all slaves in Confederate states free.
Take ABC13 with you!
Download our free apps for iPhone, iPad and Android devices
