Bernard James added 10 points, keying a go-ahead run in the second half, and the Seminoles (22-10) advanced to play Notre Dame on Sunday.
Florida State, which leads the nation in field-goal defense, held Texas A&M (24-9) to 31.4-percent shooting on the way to its first NCAA win since they beat TCU in the first round in 1998. The Seminoles had dropped three straight NCAA tournament games, losing to Wisconsin in overtime in 2009 and Gonzaga last season.
Chris Singleton struggled in his return from a broken right foot, finishing with five points and four fouls for Florida State.
Khris Middleton scored 16 points for Texas A&M -- 11 in the first half. Nathan Walkup added 11, David Loubeau scored 10.