Yes Mathematics is difficult because it is full of logic and according to researches different people have different types of brain, Logical Brain( for Maths and Logic), Muslical Brain, Language Brain.
So may be Maths is difficult for some people because they have other type of brain that's why we can do best in those things in which we are interested.Similarly not everyone have ability of writing and singing.
But it doesn't mean that we should stop doing those things in which we are not good, by constant work we can learn everything, if not very perfectly may be little good.
Mathematics may be defined as the subject in which we never know what we are talking about, nor whether what we are saying is true.
Bertrand Russell. 1872–1970
British philosopher, mathematician.
Mysticism and Logic
All science requires Mathematics. The knowledge of mathematical things is almost innate in us... This is the easiest of sciences, a fact which is obvious in that no one’s brain rejects it; for laymen and people who are utterly illiterate know how to count and reckon.
Roger Bacon. 1214-1294
English philosopher, scientist.
BTW I like Maths not because I am very good in it but because it is something very strange.
I'm not sure if it's difficult for everyone. I am more interested in language than numbers, and always have been, so I tend not to think in mathematical terms, so when I have to, it can be difficult for me.