Allow pending contacts to be removed any time

Currently, we only allow the user to remove pending contacts when adding them remotely failed. However, the user might want to remove them even earlier, because they changed their mind about adding this contact for example.