Athletic Director Bill Byrne announced the move in a statement.
Sherman went 25-25 in his four seasons. He was hired at the end of the 2007 season three days after Dennis Franchione resigned.
His best and only winning season came last year when the Aggies won their last six regular-season games and lost in the Cotton Bowl to finish 9-4.
"I appreciate Coach Sherman's selfless service to Texas A&M as our head football coach and his tireless efforts in building leaders of character," Byrne said in the statement. "He is truly one of the great offensive minds in football, both collegiate and professional, and I know that he has much to offer the game of football in the future."
Byrne said he will talk with the assistant coaches to decide who will serve as the interim coach for Texas A&M's bowl game. He also hopes to meet with the team and remaining staff Friday.
Texas A&M is moving from the Big 12 to the Southeastern Conference next year.
The Aggies entered this season with 18 returning starters, a top 10 ranking and were expected to contend for the Big 12 championship and be a factor in the national title hunt.
Instead they lost early games to Oklahoma State and Arkansas after holding double-digit halftime leads to fall to 2-2.
They won three in a row after their first skid, but a three game losing streak, which included two overtime losses, ensured the Aggies of a mediocre season. The low point of the season came when Texas A&M ended their more than century-old rivalry with Texas with a 27-25 loss at home on Thanksgiving.
At times, Sherman seemed to be grasping for ways to deal the team as the close losses piled up. Four of their six defeats were by a combined 10 points, including a 53-50 four-overtime loss to Kansas State.
"I've never experienced a season like this and I don't plan on experiencing a season like it again," Sherman said after the loss to Texas. "This was a very difficult season to swallow. We have good kids, they work hard, but for whatever reason the ball bounced funny for us sometimes, and we didn't make the play when we need to make it to win the games we didn't."
Sherman came to Texas A&M after spending two seasons as an assistant head coach with the NFL's Houston Texans. It was his first college job since he worked as an assistant with the Aggies from 1995-96.
After leaving A&M he worked as an assistant with Green Bay and Seattle before becoming the Packers' coach in 2000. Green Bay went 59-43 under Sherman and won three NFC North titles in his six seasons, but he was fired after the Packers finished 4-12 in 2005.
The change gives the Aggies a chance for a fresh start when they move into the SEC next season. They end their time in the Big 12 after a decade filled mostly with disappointing finishes. The team had more than seven wins just twice since 2002.