Currently, we only allow the user to remove pending contacts when adding them remotely failed. However, the user might want to remove them even earlier, because they changed their mind about adding this contact for example.