@@ -79,7 +79,95 @@ All of the source code for this project can be found in the [Public Mesh Testbed
...
@@ -79,7 +79,95 @@ All of the source code for this project can be found in the [Public Mesh Testbed
### Automation Suite
### Automation Suite
Testing mesh networking software is often challenging from a developer's perspective because it usually entails juggling handfuls of hardware devices. In order to make it easier to create repeatable testing environments for our software, we started experimenting with Google's Mobly framework, which allows us to run repeatable p2p mesh test suites on sets of hardware devices. This is still a work-in-progress, you can track updates here: <https://code.briarproject.org/briar/public-mesh-research/-/issues/18>
Testing mesh networking software is often challenging from a developer's
perspective because it usually entails juggling handfuls of hardware devices.
In order to make it easier to create repeatable testing environments for our
software, we started experimenting with the Mobly framework,
which allows us to run repeatable p2p mesh test suites on sets of hardware
devices.
Mobly is a Python framework for writing test cases that require multiple
devices. It is developed and maintained by Googlers but is not an official
Google product.
The basic setup is that Mobly is installed on a development machine running
a desktop OS such as Linux. A test, or rather an experiment in our case, is
defined in a Python script that uses the Mobly module to interact with a
number of Android devices. It can launch apps on each device and call remote
procedures that are defined in Java classes of the app under test. It is
also capable of pulling files from the device, which we use to retrieve log
files from the test devices once the experiment has finished. Mobly can
also recognize when devices are disconnected from and later
reconnected to the development machine.
Under the hood, Mobly is using the Android Debug Bridge (adb) to interact
with devices and employs Android's instrumentation framework to make
remote procedure calls (RPC) possible.
We designed our experiments to work like this:
Our testbed application is installed on a number of Android devices
connected to the host machine via USB.
The app has all the logic for running the experiment on device and typically
has different code to run depending on the role the device has in the
experiment.
The Python script launches the testbed app on each device and calls
the same or different remote procedures on each of them in order to assign
a role to the device and trigger the code it needs to run for that role.
The script then expects all devices to be manually disconnected from the
host machine by the tester by removing the USB cable within a fixed time frame.
From that point on, the experiment is running for as long as necessary to
collect enough data.
The Python script will wait until all devices are manually reconnected to
the host machine via USB to conclude the experiment.
Once it recognizes all devices to be reconnected, it fetches the logs
collected on the Android devices and gathers them into a directory on the
host machine.
Consider this example: the testbed app is configured to run a bluetooth
advertising and scanning experiment. The Mobly script would start the app
on three devices and instruct one of them to start advertising, another one
to start scanning for nearby devices and the third one to do both things
at the same time.
All three devices write a log file reporting various metrics relevant to
the experiment such as the own bluetooth identifier as well as a record
for each time it is able to sense one of the other devices. The devices get
disconnected from the host and are physically moved around in the testing
environment for a while. Then they are reconnected to the host to end the
experiment. The Mobly script recognizes this, stops the app and fetches
the log files from all devices for data analysis.
Mobly is well suited to run these kind of experiments.
We did however come across a few limitations and challenges that required
us to extend its functionality with custom solutions.
One issue we found is that the way Mobly captures the device logs is not
well suited for experiments running for a long time. Usually, it would
grab the logs from the internal log buffer of the device. That buffer is
limited in size however. While that buffer size can be increased using the
developer options in the device's system settings, it only partially solves
the problem.
On one hand the issue remains as the buffer will still have a fixed size
even if made very large. On the other hand, it requires the tester to be
aware of that setting and not forget to set it on every test device, which
seems error-prone.
To work around this limitation, we changed our logger implementation to
log to logcat as usual but at the same time also log our output to a file.
This file is the one that gets retrieved by Mobly later on and is placed
in the directory with the summarized results of the experiment on the host.
Another issue we found is that processing of the experiment logs can be
tedious as it requires parsing of log output. Extracting data from logs
that is embedded in human-readable log lines requires repetitive and
error-prone code.
To avoid this, we implemented a structured logging method that logs
a JSON object for each line of logging. Each log object contains
a "message" property with the human-readable log message and it also
contains a "date", "time" and "tag" property so that those do not need to
be extracted from the message. If there is an exception associated with
the log call, it is stored in a "throwable" property.
In addition to that, a dictionary of arbitrary key value pairs can be passed
to the logging call that will get added to the JSON log object as additional
properties. That way, additional data points such as a recognized device's
identifier or a signal strength can be logged structurally and later on be
easily obtained from the logs during data analysis.
