In 2005, people felt sympathy for him. In 2006, people still somewhat did. By 2007, people were like "WTF, this guy is nuts." By 2008, he was getting fat and his workrate declined as he made increasingly strange Youtube videos. And it has only gotten worse since then...
You know you're fucked up when Jeff is the more "normal" of the two Hardy brothers.
I believe that besides the fact he has taken permanent residence on planet Hardy is that he isn't and never was as charismatic as his brother. One comes out with freakish dancing, weird hairstyle/color, face paint, flies around the ring etc. and the other comes out and just kind of stands there. It isn't really hard to see why the majority of people prefer Jeff.
Matt Hardy was like George W. Bush. At one time, he had the support of everybody on his side, but he didn't know when to stop and eventually those same people turned against him.
To be fair, I always thought it was bullshit that WWE would let his druggie brother become a three time world champion while not giving him, the superior worker, a run with the belt and no I do not consider the WWECW belt a world title.
Even when he was in the Hardy Boyz with Lita, Jeff and Lita were the bigger names in the group. No one really cared much about Matt.
He just isn't the performer Jeff is. I am not a Jeff Hardy fan but Jeff, whether you like him or hate him, was great in the ring and put on an entertaining show back in the day. Everyone loved when Lita would get in to it and Matt....he was basically just 'there'.