He's eager to erase those memories with his new team, the Houston Rockets, who acquired him in a four-team trade last week that sent swing man Trevor Ariza to New Orleans.
The Rockets formally introduced Lee at a news conference on Wednesday.
"I feel it's a good fit for me. There are a lot of quality guys on this team that put winning first," Lee said. "I fit in that category also. I'm very excited and happy to be here."
The Rockets have kept their eyes on Lee since the 2008 draft. They considered taking him with the 25th pick, but Orlando nabbed him with the 22nd selection.
Houston saved money by trading Ariza, who has four years left on his contract and is due about $6.3 million next season. Lee will make only about $1.3 million next season.
But general manager Daryl Morey said the Rockets merely jumped at their first opportunity to sign Lee.
"It's a guy we've been targeting for a long time," Morey said. "We think he'll really fit in and give us a lot of quality minutes this year."
Lee started 16 playoff games for the Magic in 2009 and guarded Kobe Bryant when Orlando lost to the Los Angeles Lakers in the NBA finals.
Two weeks after the series, the Magic dealt Lee to New Jersey in a five-player deal that brought veteran All-Star Vince Carter back to Orlando. The Nets went 12-70 last season, nearly the worst record in NBA history.
Playing for a losing team was a new concept for Lee, who won a state high school championship in Indiana and led Western Kentucky to the regional semifinals of the 2008 NCAA tournament.
"It definitely was tough, because I'd never been a part of anything like that in my life," he said. "It took a lot of adjusting to get used to that."
The Nets avoided matching the worst record in NBA history by three victories, but still became the fifth team to lose 70 games in one season.
"The one thing I went into every day with was the mindset to get better as a pro, and continue to try to help my team, so we won't be in a position to set that record," he said. "Instead of playing to lose, we went out there and just played for each other and tried to get better as much as we could, even though the record said differently."
Coincidentally, Lee was working out in Florida with current Rocket Kevin Martin when he got the call that he'd been traded to Houston.
"It's definitely a big relief," he said. "I get to contend and compete in the Western Conference and chase a championship."
The Rockets feel like the 24-year-old Lee is a good fit for coach Rick Adelman's free-flowing offense. Lee will likely back up Martin, who knows the system well after playing for Adelman in Sacramento.
Morey said he and Adelman both like Lee's ability to move without the ball.
"He's a very unselfish player," Morey said. "He's got great offensive skills, when he's focused on scoring. But his ability to make the right pass and cut hard, not necessarily to be the receiver, but to open things up for teammates -- that's a big thing coach Adelman values, and one of the things that really stood out to us."
Lee says he's not concerned about his statistics this season, not even his minutes. He just wants to play for a successful team again.
"Winning is everything," he said. "Everything goes a lot better when you're winning. You feel good, you're happy when you wake up to come to practice and compete. The whole aspect of winning makes everything better.
"Once you're playing the right way, scoring and rebounds, outnumbering the next man, it really doesn't even matter," he said. "You put all that aside to chase a championship, and that's what I feel is going to take place here."