Experimentation orchestration library: Mobly
Activity
marked this issue as related to #13
Mobly allows us to run remote procedures on testing devices initiated by a host. From the mobly-snippet-lib README:
The Mobly Snippet Lib allows you to write Java methods that run on Android devices, and trigger the methods from inside a Mobly test case. The Java methods invoked this way are called
snippets.While we can write such snippets ourselves, this is one of the more advanced topics. There's a project called Mobly Bundled Snippets (MBS), which is basically an Android app that can be installed on any device. With the mobly command line tools (a set of Python scripts) installed, one can then interact with the MBS from a Python REPL:
snippet_shell.py com.google.android.mobly.snippet.bundled >>> print(s.help()) ... RPCs provided by BluetoothAdapterSnippet: @Rpc btBecomeDiscoverable(Integer) returns void // Become discoverable in Bluetooth. @Rpc btCancelDiscovery() returns void // Cancel ongoing bluetooth discovery. @Rpc btDisable() returns void // Disable bluetooth with a 30s timeout. @Rpc btDiscoverAndGetResults() returns List // Start discovery, wait for discovery to complete, and return results, which is a list of serialized BluetoothDevice ob jects. @Rpc btEnable() returns void // Enable bluetooth with a 30s timeout. @Rpc btGetAddress() returns String // Returns the hardware address of the local Bluetooth adapter. @Rpc btGetCachedScanResults() returns ArrayList // Get bluetooth discovery results, which is a list of serialized BluetoothDevice objects. @Rpc btGetName() returns String // Get the friendly Bluetooth name of the local Bluetooth adapter. @Rpc btGetPairedDevices() returns List // Get the list of paired bluetooth devices. @Rpc btIsEnabled() returns boolean // Return true if Bluetooth is enabled, false otherwise. @Rpc btPairDevice(String) returns void // Pair with a bluetooth device. @Rpc btSetName(String) returns void // Set the friendly Bluetooth name of the local Bluetooth adapter. @Rpc btStopBeingDiscoverable() returns void // Stop being discoverable in Bluetooth. @Rpc btUnpairDevice(String) returns void // Un-pair a bluetooth device. ... RPCs provided by WifiManagerSnippet: @Rpc wifiClearConfiguredNetworks() returns void // Clears all configured networks. This will only work if all configured networks were added through this MBS instan ce @Rpc wifiConnect(JSONObject) returns void // Connects to a Wi-Fi network. @Rpc wifiConnectSimple(String, String) returns void // Connects to a Wi-Fi network. This covers the common network types like open and WPA2. @Rpc wifiDisable() returns void // Turns off Wi-Fi with a 30s timeout. @Rpc wifiDisableSoftAp() returns void // Disable Wi-Fi Soft AP (hotspot). @Rpc wifiEnable() returns void // Turns on Wi-Fi with a 30s timeout. @Rpc wifiEnableSoftAp(JSONObject) returns void // Enable Wi-Fi Soft AP (hotspot). @Rpc wifiGetCachedScanResults() returns JSONArray // Get Wi-Fi scan results, which is a list of serialized WifiScanResult objects. @Rpc wifiGetConfiguredNetworks() returns List // Get the list of configured Wi-Fi networks, each is a serialized WifiConfiguration object. @Rpc wifiGetConnectionInfo() returns JSONObject // Get the information about the active Wi-Fi connection, which is a serialized WifiInfo object. @Rpc wifiGetDhcpInfo() returns JSONObject // Get the info from last successful DHCP request, which is a serialized DhcpInfo object. @Rpc wifiIs5GHzBandSupported() returns boolean // Check whether this device supports 5 GHz band Wi-Fi. Turn on Wi-Fi before calling. @Rpc wifiIsApEnabled() returns boolean // Check whether Wi-Fi Soft AP (hotspot) is enabled. @Rpc wifiIsEnabled() returns boolean // Checks if Wi-Fi is enabled. @Rpc wifiRemoveNetwork(Integer) returns void // Forget a configured Wi-Fi network by its network ID, which is part of the WifiConfiguration. @Rpc wifiScanAndGetResults() returns JSONArray // Start scan, wait for scan to complete, and return results, which is a list of serialized WifiScanResult objects. @Rpc wifiSetVerboseLogging(boolean) returns void // Enable or disable wifi verbose logging. @Rpc wifiStartScan() returns void // Trigger Wi-Fi scan. ...the list of available snippets is rather long, and this seems to be the source, a bunch of
*Snippet.javafiles that define them.Instead of interacting with the snippets interactively, one usually writes a python test script, a testbed configuration and executes the test on the testbed:
Python code:
from mobly import base_test from mobly import test_runner from mobly.controllers import android_device class HelloWorldTest(base_test.BaseTestClass): def setup_class(self): # Registering android_device controller module declares the test's # dependency on Android device hardware. By default, we expect at least one # object is created from this. self.ads = self.register_controller(android_device) self.dut = self.ads[0] # Start Mobly Bundled Snippets (MBS). self.dut.load_snippet('mbs', android_device.MBS_PACKAGE) def test_hello(self): self.dut.mbs.makeToast('Hello World!') if __name__ == '__main__': test_runner.main()YAML configuration:
TestBeds: # A test bed where adb will find Android devices. - Name: SampleTestBed Controllers: AndroidDevice: '*'Run:
python3 hello_world_test.py -c sample_config.ymlThere are three tutorials:
- Getting to know basic usage as the sample above: Mobly 101
- Running Instrumentation tests
- Writing custom snippets, see examples at the bottom of the page.
-
Open question: will it be possible to use on API levels below 26?
The library mobly-snippet-lib exports a minSdk of 26 which forces consumers of the library to do the same. It seems possible however to compile that library on Sdk 14 as well. The bump from minSdk 15 to 26 happened in a pull request in October 2021 without a clear reason.
-
I opened a ticket upstream, asking whether the authors remember why they bumped the version this high up: https://github.com/google/mobly-snippet-lib/issues/119
-
-
This is what happens when I disconnect one of the devices from USB during the test (and nothing changes when I reconnect before the test finishes the
sleep(10)here:$ python3 public_mesh_test.py -c public_mesh.yml [PublicMeshTestBed] 11-25 15:00:26.414 INFO ==========> PublicMeshTest <========== [PublicMeshTestBed] 11-25 15:00:26.781 INFO [AndroidDevice|PT99651AA1AC1803610] Initializing the snippet package org.briarproject.publicmesh. [PublicMeshTestBed] 11-25 15:00:37.867 INFO [AndroidDevice|ZY32BSN89S] Initializing the snippet package org.briarproject.publicmesh. [PublicMeshTestBed] 11-25 15:00:39.376 INFO [Test] test_basic_discovery [PublicMeshTestBed] 11-25 15:00:49.618 INFO [Test] test_basic_discovery PASS tearing down [PublicMeshTestBed] 11-25 15:00:57.101 ERROR Failed to stop service "snippets".: <AndroidDevice|PT99651AA1AC1803610> Failed to stop existing apk. Unexpected output: . Traceback (most recent call last): File "~/.local/lib/python3.8/site-packages/mobly/expects.py", line 152, in expect_no_raises yield File "~/.local/lib/python3.8/site-packages/mobly/controllers/android_device_lib/service_manager.py", line 204, in stop_all service.stop() File "~/.local/lib/python3.8/site-packages/mobly/controllers/android_device_lib/services/snippet_management_service.py", line 116, in stop client.stop() File "~/.local/lib/python3.8/site-packages/mobly/controllers/android_device_lib/snippet_client_v2.py", line 575, in stop self._stop_server() File "~/.local/lib/python3.8/site-packages/mobly/controllers/android_device_lib/snippet_client_v2.py", line 623, in _stop_server raise android_device_lib_errors.DeviceError( mobly.controllers.android_device_lib.errors.DeviceError: <AndroidDevice|PT99651AA1AC1803610> Failed to stop existing apk. Unexpected output: . [PublicMeshTestBed] 11-25 15:00:59.141 INFO Summary for test class PublicMeshTest: Error 1, Executed 1, Failed 0, Passed 1, Requested 1, Skipped 0 [PublicMeshTestBed] 11-25 15:00:59.142 INFO Summary for test run PublicMeshTestBed@11-25-2022_15-00-26-412: Total time elapsed 32.72900233200926s Artifacts are saved in "~/gitlab/briar/public-mesh-testbed/mobly/mesh-run/logs/PublicMeshTestBed/11-25-2022_15-00-26-412" Test results: Error 1, Executed 1, Failed 0, Passed 1, Requested 1, Skipped 0Edited by Sebastian -
The mobly library clearly has some utilities available that we can use such as this. When I add some
wait_for_device()calls towards the end of the test case like this, then the test will only finish after I reconnect the previously disconnected devices:def test_basic_discovery(self): self.droid1.pm.startActivity() self.droid2.pm.startActivity() self.droid1.pm.startAdvertising() self.droid2.pm.startDiscovery() time.sleep(10) print("wait for device 1") self.droid1.adb.wait_for_device() print("wait for device 2") self.droid2.adb.wait_for_device() -
The third Mobly tutorial is about Android Device Services:
Mobly's AndroidDevice controller is a Python module that lets users interact with Android devices with Python code.
Often times, we need long-running services associated with AndroidDevice objects that persist during a test, e.g. adb logcat collection, screen recording. Meanwhile, we may want to alter the device's state during the test, e.g. reboot.
AndroidDevice services makes it easier to implement long-running services.
The docs for BaseService's pause and resume methods seem to suggest that the service won't interact with the device while it's disconnected from USB - the interface is designed to tolerate the disconnection, but not to continue triggering actions on the device while disconnected.
I think it's important to test the behaviour of the devices while disconnected from power, so we would need to find some way to trigger actions on the devices (for example, device A starts discovery) while disconnected from USB.
I wonder if we might be able to achieve this through a combination of host-side code (Mobly) and device-side code:
- The script telling the device which actions to take would be written as a Java class in the test app, rather than as a Python script running on the host
- For tests where different devices should perform different actions (for example, device A starts discovery and device B starts advertising), the Java test script would accept a "role" parameter telling it which role to perform
- Mobly would be used to set up the devices and trigger the test script on each device, passing it the appropriate role parameter
- The devices would then be disconnected from USB
- The Java test scripts would continue executing, so the devices would perform the actions appropriate to their roles while disconnected from USB
- At the end of the test, the devices would be reconnected to USB
- Mobly would resume its connection to the devices and collect the logs
Unlike purely host-side scripting, this would not allow us to reboot the devices mid-test. So if that was needed for a particular test, we might have to use host-side scripting and keep the devices connected to USB for that test.
Another possibility would be to fake the charging state via adb commands, to make the device behave like it was unplugged from USB. But I'm skeptical about whether this would trigger real-world power management behaviour, especially when it comes to manufacturer-specific behaviour (https://stackoverflow.com/a/45267088).
-
I think this sounds very sensible. I think there are two things we still need to try out or find a solution for:
- pass the role parameter from the Mobly script to each test device. I think it should be possible to just pass a parameter for example to
startActitivity()RPC method. - collect the logs after reconnecting the devices. Not sure yet how to do that.
- pass the role parameter from the Mobly script to each test device. I think it should be possible to just pass a parameter for example to
-
pass the role parameter from the Mobly script to each test device. I think it should be possible to just pass a parameter for example to
startActitivity()RPC method.Yes hopefully, and if not I guess we could declare an RPC method for each role - startAsAlice(), startAsBob(), etc?
collect the logs after reconnecting the devices. Not sure yet how to do that.
Can we do the equivalent of
adb pullfrom the Mobly script? The testbed should be writing its logs to a file already, so we don't need to depend on logcat. We could configure the testbed to write the log file to shared storage if permissions are an issue. -
Yes hopefully, and if not I guess we could declare an RPC method for each role - startAsAlice(), startAsBob(), etc?
indeed, yes.
Can we do the equivalent of
adb pullfrom the Mobly script? The testbed should be writing its logs to a file already, so we don't need to depend on logcat. We could configure the testbed to write the log file to shared storage if permissions are an issue.I think we can, but the test usually already does this and I don't think circumventing the basic test flow is a good strategy, I'd rather find the way to make the test continue after reconnecting a device to USB. Anyway, as discussed on Monday, I don't known what else to do with this and will look into other mechanisms for orchestrating or doing the things we need with lower-level tools (or maybe some building blocks of Mobly). Anyway, I summed this up as an issue on the Mobly repo, in the hope that one of the authors there has an answer and maybe there is a simple solution to this which would allow us to pick this up. Here's that issue:
-
Great, I've received an answer from the main author and was now able to implement something that actually works:
def test_basic_discovery(self): self.droid1.pm.startActivity() self.droid2.pm.startActivity() self.droid1.pm.startAdvertising() self.droid2.pm.startDiscovery() with self.droid1.handle_usb_disconnect(): with self.droid2.handle_usb_disconnect(): maxWaitDisconnect = 10 print("waiting for disconnect for", maxWaitDisconnect, "seconds") time.sleep(maxWaitDisconnect) print("wait for device 1") self.droid1.adb.wait_for_device() print("wait for device 2") self.droid2.adb.wait_for_device() -
The mechanism currently is like this then:
- Mobly starts the application on both devices,
- it launches the activity on both,
- on one devices it starts advertising, on the other starts discovery,
- then it waits for 10 seconds in which it is expected that the user disconnects at least one of the devices from USB,
- the test then continues running until all devices are plugged back in,
- then the test is terminated, logs are collected and the activities are shut down.
changed milestone to %Public mesh research
-
Can we do the equivalent of
adb pullfrom the Mobly script? The testbed should be writing its logs to a file already, so we don't need to depend on logcat. We could configure the testbed to write the log file to shared storage if permissions are an issue.I'm looking into this now. When we use logcat gathered by Mobly, I think the amount of logs we can get will be limited by the internal log buffering of the device. If a test runs for a longer period of time, it's likely this will not allow us to get the logs for the whole test run.
Our
LogManagerImplis creating a log file already as you mentioned, however it is using an application-private directory to store that, so I think there's no way to pull it using adb.We need to make this file accessible from the outside. There used to be mode
MODE_WORLD_READABLEin addition toMODE_PRIVATE, however it has been deprecated with API 17 and using it throws a SecurityException on API 24+. So instead, I think we need to store the log on external memory. I think in order for that to work I need to request the runtime permission for writing external storage. -
I'm getting exceptions when trying to open any file in
Environment.getExternalStorageDirectory();for writing (which resolves to/storage/emulated/0) even though I have the permission for writing to external storage granted. I was thinking the behavior changed for this kind of access after some API level, but get confusing results. On an API 27 hardware device I can write files there, while on an API 30 hardware device I cannot. Tried to narrow the API level that introduced the change down using emulators: cannot write on 30, 29 and 28, but also cannot write on the 27 emulator😕 I can write on the emulator with levels 17, 21, 22, but not with 23. -
Strange. According to the docs, you shouldn't need any permissions to access the dir returned by Environment.getExternalStorageDirectory() on API 19 and higher. The docs suggest checking Environment.getExternalStorageState() to check whether the storage is mounted, although it seems unlikely that it wouldn't be mounted in an emulator?
https://developer.android.com/training/data-storage/app-specific#external
-
I wonder if another possibility would be for the app to read the log file itself and return it from a Mobly method call as a string or byte array?
I wonder how the method call arguments and return values are marshalled... passing a megabyte or two of log data might test the limits of the system :-)
-
If you use getExternalFilesDir() are the files accessible via adb?
yes, looks like they are accessible. Great, we can use that.
The docs suggest checking Environment.getExternalStorageState() to check whether the storage is mounted, although it seems unlikely that it wouldn't be mounted in an emulator?
yeah, I checked that I can read the contents of the directories which seems to indicate it's mounted.
I wonder if another possibility would be for the app to read the log file itself and return it from a Mobly method call as a string or byte array?
I wonder how the method call arguments and return values are marshalled... passing a megabyte or two of log data might test the limits of the system :-)
Good alternative idea and still interesting what the limits are, but it looks like fortunately we won't need it then at the moment.
-
with public-mesh-testbed@d71e3219 and public-mesh-testbed@b08c71e6 I'm now able to pull logs from the app's log file and store it along with the other output of the test run:
├── logs │ └── PublicMeshTestBed │ ├── 02-07-2023_16-11-29-758 │ │ ├── PublicMeshTest │ │ │ ├── AndroidDevicePT99651AA1AC1803610 │ │ │ │ ├── app.log │ │ │ │ └── logcat,PT99651AA1AC1803610,wolverine_00eea,02-07-2023_16-11-30-005.txt │ │ │ └── AndroidDeviceZY32BSN89S │ │ │ ├── app.log │ │ │ └── logcat,ZY32BSN89S,capri_retailen,02-07-2023_16-11-30-103.txt │ │ ├── test_log.DEBUG │ │ ├── test_log.INFO │ │ └── test_summary.yaml │ └── latest -> …
-
Thinking longer-running tests through a bit more, I noticed that currently it's required to keep the python shell open for as long as the tests run. If a test spans multiple days, we need to be sure to keep the host machine running. Putting the host machine to sleep doesn't seem to cause any problems so far, I was able to reconnect the devices and finish a running test after putting the host to sleep and waking it up again.
-
OK, seems to work just fine. I just had a two device test running, disconnected those devices and kept the test running. Then connected two different devices, used AS to upload the APK onto both of them, started the same test on those two devices too. Then I disconnected those two devices as well and reconnected the first two devices. The first test then successfully finished. After reconnecting the other two devices, the second test finished successfully, too.
-
While writing up the report, I noticed that our log collection has one flaw.
I'm writing down the reasons we did come up with the solution of logging to a file and retrieving that log file after the experiment is done. The reason mainly is that the log is usually captured by Mobly from the device's internal log buffer. That buffer has limited size and for a long experiment there's a risk we don't get access to the full log. There's an option in the developer settings of the device to increase that limit, however it doesn't really solve the problem. It's still a fixed size. Also it seems error-prone as the tester needs to be aware of that settings and manually set it to a high value. Forgetting to do so can result in a wasted experiment with almost no data collected.
So far so good. However, I can see one drawback: we're only collecting our own applications log into the log file. So we're basically throwing away all the unrelated logs. First I was even eager to report this as a plus of this method as the resulting log file is already filtered, hence smaller and easier to process. However, it's possible that sometimes the "unrelated" logs are actually related and would be useful to have access to. I'm thinking of system error messages that are sometimes interleaved with app logs and report on system failures that can be the cause of other issues we're seeing on the app level. Also, other apps might be interfering with whatever it is were doing in our app and without the log of the other apps, it's going to be difficult to recognize that.
-
Hmm, good point. From what I remember, newer versions of Android don't let the application run
logcat(to prevent the app from seeing other apps' logs). But I guess the shell user is allowed to run it. Maybe while the device is connected at the start of the experiment, we could useadb shellto start logcat and redirect its output to a file?I found this snippet from a previous round of experiments that allows us to run a script on the device (as the shell user) that will survive adb disconnection:
adb shell 'nohup sh /sdcard/foo.sh 2>&1 >/sdcard/foo.out'
removed Sponsor 6 label