Investigate whether equivalent public keys can damage the security of handshake mode

Some ECDH public keys are equivalent, in the sense that multiplying them by the same scalar produces the same point. If an attacker sends us a handshake public key that's equivalent to a contact's handshake public key, we'll fail to detect that it's the same key (see #1565 (closed)) and derive the same handshake mode transport keys. The attacker won't be able to derive the keys, but we'll use the same keys for handshaking with the contact and trying to handshake with the attacker.

This shouldn't affect confidentiality, integrity or authenticity, as we use a unique random nonce with the stream header key, but it could affect protocol obfuscation by using the same tags for sending streams to the contact and the attacker.

Work out whether this attack is possible, and if so, whether we can detect and prevent it.

Subtask of #1232 (closed).

changed milestone to %Android 1.2

added ~20 Sponsor 1 ~1 labels

For X25519, an equivalent public key can be created by adding a point with low order to a legitimate public key:

https://moderncrypto.org/mail-archive/curves/2017/000846.html

I don't know if it's possible to detect whether this has been done, but we could detect equivalent keys by deriving a value deterministically from the shared secret and using that as the unique key identifier. Equivalent keys will produce the same shared secret and thus the same key identifier.

Useful explanations of point equivalence in X25519 and Ed25519:

https://github.com/dalek-cryptography/ed25519-dalek/issues/20

https://crypto.stackexchange.com/questions/55632/libsodium-x25519-and-ed25519-small-order-check/55643#55643

Equivalent public keys should already produce different shared secrets, and thus different handshake mode transport keys, because we hash both public keys into the shared secret.

closed

mentioned in merge request !1093 (merged)