Public-Mesh-Research-Report.md

@@ -79,7 +79,95 @@ All of the source code for this project can be found in the [Public Mesh Testbed
 
 ### Automation Suite
 
-Testing mesh networking software is often challenging from a developer's perspective because it usually entails juggling handfuls of hardware devices. In order to make it easier to create repeatable testing environments for our software, we started experimenting with Google's Mobly framework, which allows us to run repeatable p2p mesh test suites on sets of hardware devices. This is still a work-in-progress, you can track updates here: <https://code.briarproject.org/briar/public-mesh-research/-/issues/18>
+Testing mesh networking software is often challenging from a developer's
+perspective because it usually entails juggling handfuls of hardware devices.
+In order to make it easier to create repeatable testing environments for our
+software, we started experimenting with the Mobly framework,
+which allows us to run repeatable p2p mesh test suites on sets of hardware
+devices.
+
+Mobly is a Python framework for writing test cases that require multiple
+devices. It is developed and maintained by Googlers but is not an official
+Google product.
+The basic setup is that Mobly is installed on a development machine running
+a desktop OS such as Linux. A test, or rather an experiment in our case, is
+defined in a Python script that uses the Mobly module to interact with a
+number of Android devices. It can launch apps on each device and call remote
+procedures that are defined in Java classes of the app under test. It is
+also capable of pulling files from the device, which we use to retrieve log
+files from the test devices once the experiment has finished. Mobly can
+also recognize when devices are disconnected from and later
+reconnected to the development machine.
+Under the hood, Mobly is using the Android Debug Bridge (adb) to interact
+with devices and employs Android's instrumentation framework to make
+remote procedure calls (RPC) possible.
+
+We designed our experiments to work like this:
+Our testbed application is installed on a number of Android devices
+connected to the host machine via USB.
+The app has all the logic for running the experiment on device and typically
+has different code to run depending on the role the device has in the
+experiment.
+The Python script launches the testbed app on each device and calls
+the same or different remote procedures on each of them in order to assign
+a role to the device and trigger the code it needs to run for that role.
+The script then expects all devices to be manually disconnected from the
+host machine by the tester by removing the USB cable within a fixed time frame.
+From that point on, the experiment is running for as long as necessary to
+collect enough data.
+The Python script will wait until all devices are manually reconnected to
+the host machine via USB to conclude the experiment.
+Once it recognizes all devices to be reconnected, it fetches the logs
+collected on the Android devices and gathers them into a directory on the
+host machine.
+
+Consider this example: the testbed app is configured to run a bluetooth
+advertising and scanning experiment. The Mobly script would start the app
+on three devices and instruct one of them to start advertising, another one
+to start scanning for nearby devices and the third one to do both things
+at the same time.
+All three devices write a log file reporting various metrics relevant to
+the experiment such as the own bluetooth identifier as well as a record
+for each time it is able to sense one of the other devices. The devices get
+disconnected from the host and are physically moved around in the testing
+environment for a while. Then they are reconnected to the host to end the
+experiment. The Mobly script recognizes this, stops the app and fetches
+the log files from all devices for data analysis.
+
+Mobly is well suited to run these kind of experiments.
+We did however come across a few limitations and challenges that required
+us to extend its functionality with custom solutions.
+
+One issue we found is that the way Mobly captures the device logs is not
+well suited for experiments running for a long time. Usually, it would
+grab the logs from the internal log buffer of the device. That buffer is
+limited in size however. While that buffer size can be increased using the
+developer options in the device's system settings, it only partially solves
+the problem.
+On one hand the issue remains as the buffer will still have a fixed size
+even if made very large. On the other hand, it requires the tester to be
+aware of that setting and not forget to set it on every test device, which
+seems error-prone.
+To work around this limitation, we changed our logger implementation to
+log to logcat as usual but at the same time also log our output to a file.
+This file is the one that gets retrieved by Mobly later on and is placed
+in the directory with the summarized results of the experiment on the host.
+
+Another issue we found is that processing of the experiment logs can be
+tedious as it requires parsing of log output. Extracting data from logs
+that is embedded in human-readable log lines requires repetitive and
+error-prone code.
+To avoid this, we implemented a structured logging method that logs
+a JSON object for each line of logging. Each log object contains
+a "message" property with the human-readable log message and it also
+contains a "date", "time" and "tag" property so that those do not need to
+be extracted from the message. If there is an exception associated with
+the log call, it is stored in a "throwable" property.
+In addition to that, a dictionary of arbitrary key value pairs can be passed
+to the logging call that will get added to the JSON log object as additional
+properties. That way, additional data points such as a recognized device's
+identifier or a signal strength can be logged structurally and later on be
+easily obtained from the logs during data analysis.
 
 ## Bluetooth