Magic comes from the Ignorant of the Immortal Realm just by virtue of their existence, and they aren’t even aware of this. It’s the power that created the material reality of the Mortal Realm, and permitted the accelerated evolution of multiple sapient species instead of just one.
Though technically everyone can practice magic, not everyone is sensitive to this energy, and the difference in talent for casting spells between the sensitives and non-sensitives is significant. Just like anyone can run, the difference between the sensitives and non-sensitives is like that of an athlete and an obese person.
Sensitivity to magic comes at a high cost, however: constant nightmares, proneness to madness, obsessive impulses, headaches, nausea, selective eating, suicidal tendencies, paranoia, and schizophrenia.
It’s extremely hard for one to understand the concepts of magic described by someone of a higher sensitivity level.
Though powerful, sensitives are very unstable and so, cases of sensitive people ending up as rulers are rare and far between, with the exception of tribal societies. Through most of recent history, they’ve been controlled by more powerful groups with different levels of oppression, but never exterminated, as they are very useful.
Alter the surroundings, but not create new things. Yes, one can throw a fireball, but by manipulating gases in the ambient and raising their temperature to combustion, and then changing their vectors.