With the new "X Factor" judges at his side Monday, the reliably acerbic Cowell said that Spears is "quite mean" and Lovato is "a brat." But, Cowell added, he's long wanted to work with Spears and said "there's really something likeable" about Lovato.
The pop stars are joining "X Factor" for its second season.
Spears and Lovato replace judges Paula Abdul and Nicole Scherzinger, who left after the show's debut year fell short of the spectacular ratings Cowell forecast. Antonio "L.A." Reid and Cowell will return as judges.
With taping under way in Miami, the "X Factor" panel spoke to a Beverly Hills, Calif., meeting of the Television Critics Association by satellite. Earlier Monday, Fox announced that Cowell's former stomping ground, "American Idol," has hired Mariah Carey as a judge for next year.
Spears was asked why she decided to join "X Factor," which is paying her a reported $15 million.
"I feel like being able to be on a show where you can give back and help people achieve their dream is just really interesting to me. ... It's different from anything I've ever done," she said.
She demurred when asked if she hoped it would boost her record sales.
"It's purely on the fact I love the show," said Spears, who said she'd seen "X Factor" a "couple times" before signing on.
The 19-year-old Lovato was queried about her concerns for younger contestants confronted with fame.
In 2011, the singer-actress entered rehab for what were termed "emotional and physical issues." She said that as a child who faced bullying, she had an eating disorder and later started cutting her wrists to vent her despair.
Lovato takes that issue into consideration as a judge and in regard to her younger sister, Madison, whom she said she worries about but is "doing great." Madison, 10, played the daughter of Eva Longoria's character on ABC's "Desperate Housewives."
But the perils of fame can be overblown, Lovato said.
"It adds some pressures, sometimes it makes the problem a little bit worse. ... But for me, I had been struggling with those issues before," she said. "Unless you're in a good place when you start working, I think it's kind of it's inevitable for these things will happen."