The men used a kayak and a blanket to get to the dog and were able to safely bring the animal back to shore.
The Good Samaritans told Action News there were times when they felt like the ice was going to give way but they still felt compelled to do something.
Fire crews, who discourage people from going into icy and unstable waterways, were on scene as a precaution.
The dog, scared and disoriented, was navigating the river's rocky ice near Trenton for hours.
Earlier, concerned residents and police lined the banks of the river and tried to lure the animal to safety.
"I just think he's disoriented out there and doesn't' know where to go. He's scared, he's frightened," said Amy Mostrangeli, Ewing Township Animal Shelter.
"We are kind of helpless. There is a boat here but we couldn't get a boat out there and he or she just keeps roaming the middle of the same area. It's sad but we are trying," said Cecilia Gardano.
The temperature in Trenton has been in the low 20s on Sunday and the chill below the dog's paws are much colder.
Many people felt helpless watching the animal wander.
At times, the pit bull mix came close to coming ashore but each time the dog retreated to the middle of the river.
"He probably doesn't have much longer. Someone has to rescue him and soon," said Mostrangeli.
Shortly after being brought to shore the animal was taken to an area veterinarian hospital for treatment.
The Good Samaritans who rescued the animal were not harmed.
