A statement which is essentially derogatory is forbidden, regardless of the speaker’s opinion regarding that statement. Conversely, it is forbidden to utter a statement which, essentially, is not derogatory, if either the speaker or the listener considers it derogatory.

An illustration of this would be where an individual’s mode of dress is being discussed. Although there may be nothing wrong with the way the person dresses, nevertheless, it is forbidden to say that the individual dresses in that manner, if either the speaker or the listener has an unfavorable impression of those who dress that way.

Sefer Chofetz Chaim, 6 Tishrei, page 60

dyjst