Math isn't hard per se, you dertermination (and just a bit of logic) are what can make a subject hard or easy. For example, I try my best in my Calculus class and do well in it. As for AP European History, I hardly put ant effort and do not as great.
I think that the main reason people find maths hard is that it is always buiding on the knowledge that the student is expected to already have. Unfortunately people often do not have the pre-knowledge that they need in order to understand and achieve.
I coach kids privately and I hardly ever do what they are doing at school. I build their basic knowledge that they are usually expected to already have and in so doing their confidence is boosted and all of a sudden they realize that it is worth listening in class. They really can understand what the teacher is talking about. Then they can start really improving!
So it is mostly about pre-knowledge and confidence.