1# Gabeldorsche Certification Tests
2
3[TOC]
4
5## What is GD cert test
6
7A core problem behind Bluetooth interoperability testing is testers' inability to
8automate test operations on certain peripherals. For example, although an user
9can automate API calls on Android with tools like
10[SL4A](https://android.googlesource.com/platform/external/sl4a/) or ADB shell
11commands, they often struggle with automating corresponding operations on
12carkits, headsets, and smart watches. Even if they find a way to automatically
13control one model of such peripheral device using device maker's tools or
14self-developed after-market tools such as relays and robotic arms, such kind of
15tool usually does not scale to other models of peripherals due to their
16differences in physical user interfaces.
17
18GD certification test framework comes into rescue. In
19the framework, a common API is defined using a [gRPC](https://grpc.io/) protobuf
20for profiles such as HFP, AVRCP and function groups like pairing, scanning, and
21advertising. Everyone who wants to test their Bluetooth device implements an
22**gRPC Facade Server** using this protobuf definition. This server executable is
23responsible in gluing various gRPC based APIs with their device specific
24automation hook through a **Device Specific Driver**. The server will then
25expose an IP address and a port number to the test runner. A **Test Runner**
26process will load and execute the actual test case and use the auto-generated
27gRPC facade client APIs to interact with the device being tested. The following
28diagram shows how the system could be configured:
29
30![cert_test_architecture_drawing](./cert_test_architecture_drawing.png)
31
32## Terminology
33
34**gRPC Facade**
35:   A set of automation APIs of a specific profile (HFP, AVRCP, etc) or function
36    group (pairing, scanning, advertising, etc) defined in gRPC protobuf format.
37
38**Test Runner**
39:   A process that loads and runs the actual test cases and use auto-generated
40    gRPC facade client library to interact with test devices. Currently, the
41    preferred test runner is the [Android ACTS](https://android.googlesource.com/platform/tools/test/connectivity/+/refs/heads/master/acts/)
42    framework and test cases are written in Python for fast iteration and easy debugging
43
44**gRPC Facade Server**
45:   A concrete implementation of the gRPC automation APIs, which will convert
46    RPC requests into invocations of device specific automation hook using a
47    **Device Specific Driver**. This server can be written in any language and
48    on any platform. The server need to expose an IP address and 3 ports
49
50**Tester Signal Port**
51:   A port to indicate that the server is ready for operation to the **Test
52    Runner**
53
54**gRPC Root Server Port**
55:   A port that allows **Test Runner** to start and stop various functional
56    facade services
57
58**gRPC Facade Server Port**
59:   A port that allows **Test Runner** to interact with actual profile
60    implementations on the test device
61
62**Device Specific Driver**
63:   A library or some mechanism that allows the **gRPC Facade Server** to
64    control the test device. This library is opaque to the **Test Runner** that
65    should have no knowledge of how this component works * **Root-Canal**: A
66    Bluetooth PHY emulator that takes either LMP or HCI packets. The goal is for
67    this emulator to simulate a Physical RF environment without using any real
68    Bluetooth adatper.
69
70## How to run GD cert test in Android tree
71
72Assume user has an Android checkout and finished `source build/envsetup.sh` and
73`lunch` to a preferred target
74
75### Run GD cert tests on host machine
76
77```shell
78$ANDROID_BUILD_TOP/system/bt/gd/cert/run --host
79```
80
81#### Python 3.8+
82The cert tests require >python3.8 to operate and the associated python
83virtualenv package.  The script may help properly install these requisites.
84
85```shell
86source $ANDROID_BUILD_TOP/system/bt/gd/cert/set_up_virtualenv.sh
87```
88
89### Run GD cert tests on devices for the first time
90
91Connect at least two Android devices and follow on-screen instructions after
92running the following command
93
94```shell
95$ANDROID_BUILD_TOP/system/bt/gd/cert/set_up_and_run_device_cert.sh
96```
97
98### Run GD cert tests on devices for the second time and after
99
100Keeping the same set of devices connected
101
102```shell
103$ANDROID_BUILD_TOP/system/bt/gd/cert/run
104```
105
106### `system/bt/gd/cert/run` command reference
107
108*   `--host`: Run tests on host only using `root-canal`
109*   `--clean`: Remove any test setup files and do a clean test run
110*   `--repeat=N`: Repeat running the same set of tests N times without redoing
111    test setup
112*   `--test_file=<file_name>`: Running only tests listed in `<file_name>`
113*   `--test_filter=<test_filter>`: Test case filter in the format of
114    "TestClass:test_case_name", for multiple test cases, quote them using " and
115    separate each filter by space such as "TestClass1:test_case_1
116    TestClass2:test_case_2"
117
118### Run GD cert tests on devices over SSH
119
120The following assumptions assume the following configuration
121
122*   Local setup: Linux or MAC laptop with two Android phones connected
123*   Remote setup: Linux or MAC workstation
124
1251.  Check if your ADB version is up to date on both host machine and remote
126    workstation. Both versions should be 29.0.3 or above. If not, uninstall and
127    reinstall your adb.
128
1291.  Enable SSH port forwarding for both ADB port and various ports used by our
130    tests, run the following command on your Mac, assuming the following
131    configuration {value=2}
132
133    *   ADB Port: 5037
134    *   Cert Device:
135        *   gRPC Root Server Port: 8896
136        *   gRPC Facade Port: 8898
137        *   Signal Port: 8894
138    *   Device Under Test:
139        *   gRPC Root Server Port: 8897
140        *   gRPC Facade Port: 8899
141        *   Signal Port: 8895
142
143    Among these ports, ADB Port needs to be forwarded from local machine to
144    listen on remote workstation so that its adb client can connect to local
145    machine's adb server (`ssh -R`). Signal Port needs to be forwarded from
146    remote workstation to listen on local machine so that local phone can
147    connect to Test Runner listening on this port during Facade Server bring up
148    (`ssh -L`). Both gRPC Root Server Port and gRPC Facade Port need to be
149    forwarded from local machine to listen on remote workstation so that Test
150    Runner can connect to these ports on the Facade Server (`ssh -R`).
151
152    Hence, the resulting `ssh` command is:
153
154    ```shell
155    ssh -R 5037:127.0.0.1:5037 -R 6401:127.0.0.1:6401 -R 6402:127.0.0.1:6402 \
156    -R 6403:127.0.0.1:6403 -L 8894:127.0.0.1:8894 -L 8895:127.0.0.1:8895 \
157    -R 8896:127.0.0.1:8896 -R 8897:127.0.0.1:8897 -R 8898:127.0.0.1:8898 \
158    -R 8899:127.0.0.1:8899 <host_name>
159    ```
160
1611.  Connect all your devices, open a different terminal on your Mac and run
162
163    ```shell
164    adb kill-server && adb devices
165    ```
166
167    You should see devices on your local machine shows up
168
1691.  SSH into your remote workstation and run
170
171    ```shell
172    adb kill-server && adb devices
173    ```
174
175    You should see same set of devices connected to your local machine
176
1771.  Continue with device setup
178
179    ```shell
180    $ANDROID_BUILD_TOP/system/bt/gd/cert/set_up_and_run_device_cert.sh
181    ```
182
1831.  In subsequent runs
184
185    ```shell
186    $ANDROID_BUILD_TOP/system/bt/gd/cert/run
187    ```
188
189## How to debug GD cert tests
190
191Logs are produced and saved whenever GD cert tests runs. Depending on how the
192tests were run, the log root is different
193
194When running tests on local Android checkout, logs or most recent run are stored
195at
196
197*   /tmp/logs/HostOnlyCert/latest
198
199Navigate test logs
200
201In test root, the following logs are available:
202
203*   In every directory layer:
204    *   test_run_debug.txt: DEBUG level output from Python logging API
205        scoped at their respective layer based on the directory
206*   In test root directory:
207    *   test_summary.json: A summary test results, including stack traces
208        for any failures and metadata
209    *   test_configs.json: The ACTs config used to run this test
210    *   GdDevice_cert_stack_backing_process_coverage_summary.txt: code
211        coverage summary from llvm-cov
212*   In test class directory:
213    *   rootcanal_logs.txt: Root-Canal stdout and stderrr, host test only
214    *   Cert stack logs:
215        *   GdDevice_cert_stack_backing_logs.txt: facade server process log
216        *   cert_stack_btsnoop_hci.log: HCI packet log on cert device
217        *   cert_stack_system_log: Android phone logcat output, device based
218            test only
219    *   Device under test logs:
220        *   GdDevice_stack_under_test_backing_logs.txt: facade server
221            process log
222        *   stack_under_test_btsnoop_hci.log: HCI packet log on cert device
223        *   stack_under_test_system_log: Android phone logcat output, device
224            based test only
225    *   Individual test directories: logs for individual test cases
226
227## PTS test case coverage
228
229A Python decorator is used to indicate a test's association with a Bluetooth SIG
230Profile Tuning Suite (PTS) Qualification test. For example
231
232```python
233@metadata(
234    pts_test_id="L2CAP/EXF/BV-01-C",
235    pts_test_name="Extended Features Information Response for "
236    "Enhanced Retransmission Mode")
237```
238
239These information can be found at the Test Case Reference List (TCRL) document
240in
241[Qualification Test Requirements](https://www.bluetooth.com/specifications/qualification-test-requirements/)
242page at the Bluetooth SIG website.
243
244## Code coverage
245
246If the facade server binary is compiled using [`clang`](https://clang.llvm.org/)
247and with
248[`-fprofile-instr-generate -fcoverage-mapping`](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html)
249flags. A `.profraw` file will be generated in ech test class log directory after
250the test run. The Test Runner will then try to index these profiling data using
251[`llvm-profdata`](https://llvm.org/docs/CommandGuide/llvm-profdata.html),
252iteratively merge them from each test class, and generate a line-by-line test
253coverage report using
254[`llvm-cov`](https://llvm.org/docs/CommandGuide/llvm-cov.html).
255