There is no reason as such but people are tempted to avoid those who are not similar to them in terms of physical beauty. Odd man out is always targeted. It depends on the upbringing, the society to which they belong, the friend circle and everything around. If the behavior is natural we can at least forgive but if they are given instructions by their parents to avoid ugly people, then it is a matter of concern.
Check the history and you will find many examples who were not good by appearances but have achieved milestones.
Here I would like to bring to your notice, there are people who don’t like the company of fat and ugly but because they are rich, they forcibly try to prove that they enjoy spending times with people who have a personality clash.
Example : Boney Kapoor tying knot with Sridevi.