Tokyo's Ueno Zoo said the male panda, who had not been named yet, died of pneumonia Wednesday morning. A zookeeper found the baby, who was born last Thursday, lying belly up, without breathing, on his 7-year-old mother's chest.
He was pronounced dead an hour later, after resuscitation efforts failed.
In a news conference broadcast live, Yutaka Fukuda, the zoo's chief panda keeper, said milk had accidentally entered the baby's airway while his mother, Shin Shin, was breast-feeding. Autopsy results found traces of milk in the baby's bronchial tube.
The baby panda had been kept in an incubator for three days before being sent back to Shin Shin on Tuesday.
"They peacefully spent the night and the baby was doing fine just this morning," Fukuda said, tears welling up in his eyes. "It happened so suddenly, and it's such a pity."
The panda was the first to be born at the zoo since 1988 and was conceived naturally. Giant pandas have a low birth rate, and artificial insemination is common in captive breeding programs.
His mother was brought from China just before Japan's tsunami and earthquake disasters last year. The much anticipated baby, the first born to Shin Shin, had been celebrated across Japan, and the news of his death topped afternoon television news Wednesday.
The zoo said it would set up a space for visitors to lay flowers and pray for the dead panda.
Less than half of newborn pandas survive more than a week, Fukuda said, citing Chinese panda experts. The rate is even worse for pandas born to first-time mothers, he said.