That very much depends on the situation.
I have often been called crazy and sometimes I have taken it to heart and felt really hurt whereas other times it has been sent almost like a term of endearment.
What the person probably meant, and I was told this once, was that they were saying that I was quirky, or crazy funny.
So it makes a big difference on the context of how and about what this saying is said.
Perhaps if a person really has a mental problem, I could understand that it would be a hurtful thing to say, and therefore insulting.