True about the tower, but many of the common people believe (or at least repeat) the incest "rumor". I think they were willing to tolerate Jaime while Robert was in power, but, after his death and the obvious power plays by the entire Lannister clan, commoners seem to be over it.