README.md
1# Flicker Test Library
2
3## Motivation
4Detect *flicker* — any discontinuous, or unpredictable behavior seen during UI transitions that is not due to performance. This is often the result of a logic error in the code and difficult to identify because the issue is transient and at times difficult to reproduce. This library helps create integration tests between `SurfaceFlinger`, `WindowManager` and `SystemUI` to identify flicker.
5
6## Adding a Test
7The library builds and runs UI transitions, captures Winscope traces and exposes common assertions that can be tested against each trace.
8
9### Building Transitions
10Start by defining common or error prone transitions using `TransitionRunner`.
11```java
12// Example: Build a transition that cold launches an app from launcher
13TransitionRunner transition = TransitionRunner.newBuilder()
14 // Specify a tag to identify the transition (optional)
15 .withTag("OpenAppCold_" + testApp.getLauncherName())
16
17 // Specify preconditions to setup the device
18 // Wake up device and go to home screen
19 .runBeforeAll(AutomationUtils::wakeUpAndGoToHomeScreen)
20
21 // Setup transition under test
22 // Press the home button and close the app to test a cold start
23 .runBefore(device::pressHome)
24 .runBefore(testApp::exit)
25
26 // Run the transition under test
27 // Open the app and wait for UI to be idle
28 // This is the part of the transition that will be tested.
29 .run(testApp::open)
30 .run(device::waitForIdle)
31
32 // Perform any tear downs
33 // Close the app
34 .runAfterAll(testApp::exit)
35
36 // Number of times to repeat the transition to catch any flaky issues
37 .repeat(5);
38```
39
40
41Run the transition to get a list of `TransitionResult` for each time the transition is repeated.
42```java
43 List<TransitionResult> results = transition.run();
44```
45`TransitionResult` contains paths to test artifacts such as Winscope traces and screen recordings.
46
47
48### Checking Assertions
49Each `TransitionResult` can be tested using an extension of the Google Truth library, `LayersTraceSubject` and `WmTraceSubject`. They try to balance test principles set out by Google Truth (not supporting nested assertions, keeping assertions simple) with providing support for common assertion use cases.
50
51Each trace can be represented as a ordered collection of trace entries, with an associated timestamp. Each trace entry has common assertion checks. The trace subjects expose methods to filter the range of entries and test for changing assertions.
52
53```java
54 TransitionResult result = results.get(0);
55 Rect displayBounds = getDisplayBounds();
56
57 // check all trace entries
58 assertThat(result).coversRegion(displayBounds).forAllEntries();
59
60 // check a range of entries
61 assertThat(result).coversRegion(displayBounds).forRange(startTime, endTime);
62
63 // check first entry
64 assertThat(result).coversRegion(displayBounds).inTheBeginning();
65
66 // check last entry
67 assertThat(result).coversRegion(displayBounds).atTheEnd();
68
69 // check a change in assertions, e.g. wallpaper window is visible,
70 // then wallpaper window becomes and stays invisible
71 assertThat(result)
72 .showsBelowAppWindow("wallpaper")
73 .then()
74 .hidesBelowAppWindow("wallpaper")
75 .forAllEntries();
76```
77
78All assertions return `Result` which contains a `success` flag, `assertionName` string identifier, and `reason` string to provide actionable details to the user. The `reason` string is build along the way with all the details as to why the assertions failed and any hints which might help the user determine the root cause. Failed assertion message will also contain a path to the trace that was tested. Example of a failed test:
79
80```
81 java.lang.AssertionError: Not true that <com.android.server.wm.flicker.LayersTrace@65da4cc>
82 Layers Trace can be found in: /layers_trace_emptyregion.pb
83 Timestamp: 2308008331271
84 Assertion: coversRegion
85 Reason: Region to test: Rect(0, 0 - 1440, 2880)
86 first empty point: 0, 99
87 visible regions:
88 StatusBar#0Rect(0, 0 - 1440, 98)
89 NavigationBar#0Rect(0, 2712 - 1440, 2880)
90 ScreenDecorOverlay#0Rect(0, 0 - 1440, 91)
91 ...
92 at com.google.common.truth.FailureStrategy.fail(FailureStrategy.java:24)
93 ...
94```
95
96---
97
98## Running Tests
99
100The tests can be run as any other Android JUnit tests. `platform_testing/tests/flicker` uses the library to test common UI transitions. Run `atest FlickerTest` to execute these tests.
101
102---
103
104## Other Topics
105### Monitors
106Monitors capture test artifacts for each transition run. They are started before each iteration of the test transition (after the `runBefore` calls) and stopped after the transition is completed. Each iteration will produce a new test artifact. The following monitors are available:
107
108#### LayersTraceMonitor
109Captures Layers trace. This monitor is started by default. Build a transition with `skipLayersTrace()` to disable this monitor.
110#### WindowManagerTraceMonitor
111Captures Window Manager trace. This monitor is started by default. Build a transition with `skipWindowManagerTrace()` to disable this monitor.
112#### WindowAnimationFrameStatsMonitor
113Captures WindowAnimationFrameStats for the transition. This monitor is started by default and is used to eliminate *janky* runs. If an iteration has skipped frames, as determined by WindowAnimationFrameStats, the results for the iteration is skipped. If the list of results is empty after all iterations are completed, then the test should fail. Build a transition with `includeJankyRuns()` to disable this monitor.
114#### ScreenRecorder
115Captures screen to a video file. This monitor is disabled by default. Build a transition with `recordEachRun()` to capture each transition or build with `recordAllRuns()` to capture every transition including setup and teardown.
116
117---
118
119### Extending Assertions
120
121To add a new assertion, add a function to one of the trace entry classes, `LayersTrace.Entry` or `WindowManagerTrace.Entry`.
122
123```java
124 // Example adds an assertion to the check if layer is hidden by parent.
125 Result isHiddenByParent(String layerName) {
126 // Result should contain a details if assertion fails for any reason
127 // such as if layer is not found or layer is not hidden by parent
128 // or layer has no parent.
129 // ...
130 }
131```
132Then add a function to the trace subject `LayersTraceSubject` or `WmTraceSubject` which will add the assertion for testing. When the assertion is evaluated, the trace will first be filtered then the assertion will be applied to the remaining entries.
133
134```java
135 public LayersTraceSubject isHiddenByParent(String layerName) {
136 mChecker.add(entry -> entry.isHiddenByParent(layerName),
137 "isHiddenByParent(" + layerName + ")");
138 return this;
139 }
140```
141
142To use the new assertion:
143```java
144 // Check if "Chrome" layer is hidden by parent in the first trace entry.
145 assertThat(result).isHiddenByParent("Chrome").inTheBeginning();
146```