Although I love math and have always been a strong math student/teacher, the reality is that many people (teachers included!) struggle with or dislike math. The good news for ELL students is that math is a universal language. Yes, there are cultural differences in the way math is taught, but numbers can transcend spoken and written language. The bad news is that the academic vocabulary involved in mathematics often holds people back, especially our English language learners.
This link has several articles about the importance of teaching math vocabulary and strategies for doing so.