It's still part of one's arsenal/power, regardless of who has the better control. You still tried to make excuse of exclusion of one's ability.
Should I take away the Red Dragon from Issei in Highschool DxD because he doesn't have good control over its power? No, same thing. The dragon's power is still part of Issei's Sacred Gear arsenal that is placed inside of him. This is from a different anime.