Imagine someone who loses a phone and can't get into their own wallet. That seems scary, sure. But trusting an exchange means someone else can freeze it or lose it for you, and that feels worse to me. I don't really know, though.
When an exchange halts withdrawals, people can just stare at a balance they cannot move. That seems worse than someone losing a phone and struggling with their own wallet. I get the other side's point, but I still feel safer not letting a company hold it.