Yes, of course it wears them, the movement is running as if you wear the watch all time. But putting them in a drawer doesn´t save a service after some years, it´s the oil which must be replaced. It´s a personal decision to use a winder or not. Some experts say the abrasive wear is too much, other experts say it´s better for the movement to stay in motion.
Some people wear their one and only watch for decades without problems, so for a Swiss movement of high quality and in good condition, with enough oil in the bearings, it shouldn´t be a problem IMHO.
I own just a single winder and almost use it for day/date watches without quickset to keep them up to date when I want to wear them again inner a few days.