On Sunday's Game of Thrones episode, Daenerys is most suspicious of Varys's loyalties because in the past, he sides with Robert Baratheon against Mad King Aerys, then supports Dany's brother Viserys (who was a weak, unintelligent man). Once Viserys is dead, Varys then helps carry out Robert's order to have Dany murdered — which was obviously unsuccessful.
Daenerys wonders what kind of a servant conspires to crown the next monarch when the one in power no longer suits him. Varys shoots back that "incompetence should not be rewarded with blind loyalty."
He then reminds Daenerys that he has no house loyalty. He is only loyal to the people.
"I wasn't born into a great house. I came from nothing. I was sold as a slave and carved up as an offering," says Varys. "When I was a child I lived in alleys, gutters, abandoned houses. You wish to know where my true loyalties lie? Not with any king or queen, but with the people. The people who suffer under despots and prosper under just rule. The people whose hearts you aim to win. If you demand blind allegiance, I respect your wishes. Grey Worm can behead me or your dragons can devour me, but if you let me live, I will serve you well. I will dedicate myself to seeing you on the Iron Throne because I choose you. Because I know the people have no better chance than you."
It's a beautiful speech and also, he's not lying about his motivations. Varys was born in Lys, one of the free cities. He was born a slave, forced to work in a troupe of actors — which is not a terrible situation, until he is sold to a sorcerer who cuts off Varys's genitals to use in a ritual, later casting Varys out into the street.
Varys did everything he could to survive: begging, stealing, selling himself. It was during this time on the streets that he starts accumulating his network of spies and informants, which he calls his "little birds." The network is actually made up of poor, homeless children, much like Varys was as a child. But in exchange for telling him what they learn, Varys gives the children protection, food, and treats. He has always shown great compassion toward children.
The "little birds" are also what let Varys become the "Master of Whispers," which eventually puts him on the Mad King's small council as an adviser. He has been working for the kings ever since, helping usurp the ones he thinks are failing the people. Daenerys likes what Varys stands for, but she wants his word that if he thinks she is failing her subjects, he must tell her to her face, not conspire behind her back.
And if he ever betrays her, she'll burn him alive.
"I would expect nothing less from the Mother of Dragons," he replies with a small smile.