It's happening in the Suffolk County hamlet of the appropriately named Rocky Point.
"If you go down on the beach and looking up the bluff, you wouldn't feel comfortable bringing your family here and having a relaxing day," said Jane Bonner, Town of Brookhaven.
Because just what if the ground loosened up?
That's what concerns people who frequent the private beach in Rocky Point.
"It's not going to slide down the cliff today. Would it survive another fierce storm where we lose another foot or two of sand? I think that's risky," said Desmond Butler, of the North Shore Beach Property Owners Association.
The house on Soundview Drive has been boarded up for almost a year now.
The homeowner went into foreclosure.
Bank of America says while they don't yet legally own the home, they are prepared to demolish it.
The problem is they have to track down the original owner to sign off on the demolition.
They've had a tough time finding him.
The town says the bank must also clean up asbestos in the home before the town will issue a demolition permit.
"I want it done as quickly and as safely as possible and if it's going to take them a couple of weeks to get their ducks in a row, I think the community can be patient for a couple weeks, but much more beyond that I think that's going to be a problem," Bonner said.
The other concern is that graduation season is just around the corner and a lot of kids like to come there to party.
People worry that this cliff may be just a little too tempting.
"There's real jeopardy of kids being hurt," Butler said.
They say it's an accident waiting to happen.