Only add transport properties and keys when the contact was added

This will be changed once we have a way to reset state for peers
that were contacts already at some point in the past.
One contact might have deleted the other, but not vice versa.
So they have mismatching state that needs to be reset.

See #2 for more information.