But now there's music in those same streets even as the country faces many years of rebuilding, the Academy Award-winning actor said Tuesday.
Penn said "extraordinary" changes have happened since the Jan. 12, 2010, natural disaster killed more than 300,000 people and left about 1.5 million homeless.
He also called the Haitian people resilient in his remarks in a forum at Harvard's Kennedy School of Government.
The actor is an ambassador-at-large for Haiti's president and CEO of aid group J/P Haitian Relief Organization, which started with a goal of bringing painkillers to earthquake victims. It became an agency that manages a camp for displaced people and works to resettle them. It also does other aid work such as clearing rubble, repairing damaged homes and running a community center and clinics.
Former Haitian Prime Minister Michele Pierre-Louis and Ken Keen, the Army lieutenant general who commanded the U.S. military relief effort in Haiti, joined Penn as panelists to discuss the progress in Haiti since the earthquake.
Pierre-Louis spoke of the strength of the Haitian people but also of promises, including from the government, that haven't been kept.
"Today there is reason for hope, but at the same time, there is a lot to be done," she said.
The former prime minister also spoke about the need for the international intervention when it comes to helping middle-class Haitians who lost homes get loans so they can rebuild.
Keen said stabilizing security will be a major factor in the country's recovery, with everything else becoming more difficult without effective policing.
Penn said investment in manufacturing and jobs in Haiti would help solve the challenges because displaced people need work. He also said relief organizations can make a difference by helping with education initiatives because the first thing parents ask before resettlement is where their children will be going to school.
The actor dressed casually in cowboy boots and jeans, his dark straight hair combed back and falling below his collar. He also wore a scowl for some of the evening and criticized some media coverage he said misrepresented what was happening in Haiti.
In particular, Penn took issue with some reporting on his organization's work to demolish the National Palace, the presidential home in Port-au-Prince that became a symbol of the scale of devastation and government inertia following the natural disaster.
He said that it wasn't him and a bunch of "white guys" with "jackhammers" who did the demo and that nearly all his agency's workers are Haitians.
But the actor grinned after answering an audience question about how to bring visibility to Haiti's reconstruction effort. He said the country wouldn't triumph over its problems next year, but it would happen.
"It's coming in 15 years," Penn said, "and I hope I see you there."
Take ABC13 with you!
Download our free apps for iPhone, iPad, Android and Blackberry devices