Magic systems generally come in two flavors: Arbitrary or Ruled.
Arbitrary magic systems are ones where there are no rules to govern how the magic works or the rules are loose enough that it might as well be random.
An example of this type of system at work can be seen in Harry Potter and various other stories more geared toward a younger audience. The magic present in that world is fanciful and varied but it doesn’t follow any sort of natural laws. Things just sort of happen for no rhyme or reason and none of the characters can explain why it works this way.
While this can be used well for tension or solving story elements, it generally tends to lead to a lot of questions about how exactly the magic works. Questions arise like, how come nobody has a spell to cure Harry’s eyesight when there exist spells that do things like regrow bones in someone’s arm or a magic potion that can regrow skin?
The essential problem with arbitrary magic systems can be summed up with the following: If the author can’t explain why magic works a certain way, the author also can’t explain why magic doesn’t work a certain way either.
This isn’t a blanket criticism of arbitrary magic systems. They’re often used to fantastic effect but it should be noted that in many of these cases, magic isn’t used systematically but more as plot devices. An example of this use of an arbitrary magic system can be found in The Lord of the Rings universe.
On the other hand, ruled magic systems are ones where the author does their best to explain how and why the magic system works the way it does.
An example of this type of system can be found in many of the books in the science fiction genre (Old Man’s War for example) or Mistborn. By explaining how the magic works and the strict rules that it follows, we open up the way to let the reader follow along and understand why things are happening. Once the heroes get to the end of the story and face up against the villain, the reader knows that the author isn’t going to suddenly whip out an all-purpose villain bashing spell that will lead to victory. Because of this, whatever the heroes do to overcome the threat at the end of the story, it’s a solution that will make sense within the confines of the world the reader has been shown.
These are both some fairly general looks at what can be a complex system, though.
If you take The Name of the Wind as an example, Rothfuss actually incorporates both types of magic systems into his story, sympathy and the fairy magic. He explained at a reading that while he wanted the system of sympathy to be the primary magic system in the story, he realized that there were certain things that he needed to happen that sympathy couldn’t account for, hence the balancing of the rigid and ruled sympathy with the more spontaneous and wild fairy magic.
At the end of the day, I think it’s important to emphasize the point that not all arbitrary systems are bad and not all ruled systems are good.
Something like The Fionavar Tapestry by Guy Gavriel Kay boasts a fairly arbitrary system of magic where the spells being cast do what the plot requires of them without any clear understanding of what the limits are. But it would take a pretty critical reader to claim that the system doesn’t work for the story.
More than anything else, it’s important to understand why your magic system works the way it does and to always be aware (and avoid!) the common pitfalls of each type of system.