And when 2001 Heisman runner-up Rex Grossman graduated last year, he heard the murmurs.
Florida's SEC East reign is dead. Looking for a quality SEC hotspot? Better head north on I-75 for Athens.
Then high school phenom Chris Leak, the brother of UT quarterback C.J. Leak, commited to the Gators in January in front of a live ESPN audience.
What had been Martin's job to lose instantly became Leak's for the taking.
So a lot of media-types want to talk to him. They want him to rip his home-state school. They want him to whine to Ron Zook. They want him to say something about little Leak.
They want controversy.
And he's not giving them any.
"I quit reading the papers and watching the TV during the spring of my freshman year," Martin said. "Everyone was writing me off then and I figured it wasn't very good to be reading all that.
"I found out that the media doesn't really know as much as they claim to know."
Even after Martin's performance in Florida's first three games - 69 percent passing, 449 yards, and a 3-to-1 touchdown-to-interception ratio (fifth best QB rating in the nation) - many of the doubters still say Leak should be taking more snaps.
Here's what Florida Today columnist Peter Kerasotis said about Martin's performance (14-for-23, 196 yards, a TD and no INTs) in UF's convincing 65-3 win over San Jose State:
"Ingle Martin does not look like the quarterback of the future," Kerasotis said. "Heck, he doesn't even look like the quarterback of the present.
"Not every Ingle Martin pass was underthrown or overthrown. It just seemed that way."
And this week it's just gotten worse.
"I've gotten e-mails and calls all week from people saying, 'Go Vols,' he said. "It's happened all three years I've been here."
But we're not talking about some Class A high-school hero who happened to fall into a scholarship because some uncle of his played for the Gators 40 years ago. Martin is a chiseled 6-foot-3, 217 pounds.
While at MBA, Martin led the Big Red to an unprecedented 32 consecutive wins, three straight state championships, and a spot in the USA Today National Top 25 final rankings twice.
I forgot to mention the two state basketball championships he won as point guard.
And, oh yeah, his best sport was soccer.
So why did the two-time Tennessee Mr. Football recipient travel southward to join the man that represented everything Tennessee fans love to hate?
"I had to make a decision and I didn't really worry about what other people thought," he said.
"To have the chance to play for coach Spurrier in the Vacation State, or the Sunshine State, or whatever you call it - think about it, I mean I'm only an hour and a half from the beach."
So keep the criticism coming.
And then ask him about the time he played quarterback, wide receiver and punted all in the same game for the Gators last year.
He once even kicked a 56-yard field goal in high school, with no time on the clock to win a game.
So the accomplishments are phenomenal. The development is impressive. The motivation is evident.
But Mr. Leak probably wasn't the No. 1 recruit in the nation just for kicks.
The true freshman will almost certainly take over eventually. And Martin will almost cerainly re-unite with a clipboard eventually.
Or least that's what everybody's saying.
Ha.
As if Martin was listening.

-Scott Simmons is a junior in finance. He can be reached at ssimmon5@utk.edu.