Yep . It's like being caught in a trap.
When I was looking at buying woodland, I came across a couple of folks that just buy land with buildings already on there. They just live in a caravan , or in a barn etc , until the council find out , and try to move them , then they just buy the next chunk of land , and live there.
They reckon, don't even bother wasting your time trying to fight the council. Just sell up and move on .