Bluetooth dialog needs to be dismissed multiple times after rotating screen

Steps to reproduce:

Start to add a nearby contact
When the Bluetooth permission dialog is visible, rotate the screen
Dismiss the dialog
Expected: The dialog only needs to be dismissed once
Actual: The dialog needs to be dismissed once, plus once for every time the screen was rotated

Test devices:

Sony Xperia Tipo (Android 4.0.4)
Motorola Moto E3 (Android 6.0)