The Tur cites the following story in Sefer Chasidim:
After his death, a certain pious man appeared to another pious man in a dream. As the apparition’s face was distinctly yellow, the other man asked him, “Why is your face so yellow?” The apparition answered, “because I used to speak during Kaddish.” We must try our best not to pay full attention during Kaddish so we can properly answer Amen.
(taken from Praying With Fire by: Rabbi Heshy Kleinman)