How to remove mods from FTB

Its not as simple as going into the modpack, editing it, and then disabling it, turns out Forge Mod Loader still looks for it, and wont load the game further.
Trying to remove those god dam magic mods from direwolf20 modpack
I'm sure someone here is experienced with this stuff