Getting out of a contract

[Rantanplan]

This is book not script, but hopefully advice will still apply. I signed a contract with the publisher of my e-novel giving him an "agency clause" on my next novel, which I now want to get out of so that I can pursue representation elsewhere.

Is it highly delicate to just write a note informing him of that or is it standard practice?

Anyone here ever receive legal threats or anything for breaking a contract?

I know many people around here have gone from rep to rep to rep, but I'm guessing that was mostly with no written contract in place.