We have a lot of Amazon orders that we have to cancel from ship station. When we cancel not only is there a pop up box asking if we want to cancel, this other pop up box needs to be selected. It should disappear by default after a second or two.
The old software the pop up box went away.
Are you cancelling the orders because they're FBA orders and you don't actually have to ship them? If that's the case, you may want to look into setting up an automation rule to automatically cancel those orders.