Your gripe is legitimate if they are over your property line; that line doesn't lean with the fence. I'm surprised the inspector has not already said anything yet.
If there is any upside to this situation is that your property value will most likely increase when this is all over.
