Initially, when my team at work was formed, we decided to use JVM stack to implement UI autotests, namely:
Kotlin — a programming language;
JUnit5 — a core of autotest design;
Selenide — the basis for interaction with the Document Object Model of the browser in autotests;
Allure for JVM — a very convenient reporting tool for autotests.
In addition, we also have the following tools to improve UI autotesting processes:
Allure TestOPS — a tool for storing autotest artefacts and test documentation;
Selenoid — a tool for running tests remotely, cross-browser testing, and optional video recording of running autotests and their further storage
All of these tools almost always meet our requirements, each tool does its job, and does it well. But if everything was perfect, I wouldn’t be writing this post.
Very often the default use of these tools, when properly configured, caused several problems:
There used to be fewer tests, but their number has grown.
There are more autotests now. Video files in Selenoid with the recorded autotest execution have become longer. Also, they are not really linked to Allure reports.
Video files have become longer, making it increasingly difficult to understand which test is running at a given point in time.
Increased difficulty in understanding resulted in the need for more transparency and flexibility in the generation of UI autotest execution reports.
If there is a problem, it has to be solved. In general, similar nontrivial tasks have been solved before, only in the API autotesting environment. The selected stack (JUnit5, Selenide) provides quite deep configurations and allows the delivery of seriously improved video artefacts of the autotest running with support of timestamps to an Allure report.
Finding a Solution
When solving a problem, I relied on the following supporting facts:
Allure allows the usage of “attachments” in the resulting report, and this attachment can be applied to both a specific step in the test and the entire test;
Selenide starts a browser session in the Selenoid container when you first call the open() method — you can also use it when implementing the solution.
Selenide has a secure contract for accessing the webdriver using a WebdriverRunner wrapper.
The Java implementation of Selenium (which Selenide is based on) has a contract to work with events implemented by WebdriverListener;
In the default mode, Selenide takes responsibility when its webdrivers close after all tests are completed.
JUnit5 is packed with rich features to tap into the lifecycle test using JUnit5 lifecycle extensions.
The basic idea of implementing an improved UI autotest reporting process can be broken down into the following steps:
Rather than including the video recording of all tests in one file, split up the video.
Form a time line for autotest execution from the start of video recording in Selenoid.
Create timestamps for failed autotests relative to the time line.
Upgrade the video according to the timestamps.
Add an “attachment” report with the upgraded video to Allure.
I implemented some of the ideas as-is, but I had to work a little harder on others in order to implement an algorithm that was reasonable in terms of the time spent for its development. The details are below.
How to Improve the Existing Video Recording Process in Selenoid
The following conditions must be taken into account when working on a project:
Selenoid records actions in a container with a browser only when the following capabilities are set for the webdriver (I will represent this as json for convenience):
{
"selenoid:options": {
"enableVideo": true
}
}Selenoid begins video recording when starting a new session (opens session → starts browser → clicks the url link for UI tests)
Selenoid stops video recording after closing the session (or the webdriver itself) in the autotest code
After discussing various ideas with the team and potential customers of this solution, we came up with the following approach — to record a separate video for each test suite or test class, to put it simply.
This can be done by implementing a JUnit5 Extension. This is a class that must implement one or more interfaces contained in the JUnit5 library. Here, we are interested in the following interfaces:
BeforeAllCallback is an interface that adds some functionality before running all tests and fixtures in a test container (test class). Here we shall add a Selenide pre-configuration to set up video recording options in Selenoid;
AfterAllCallback is an interface that adds some functionality after executing all tests and fixtures in a test container (test class). Here, we implement the process of stopping the video recording.
In addition, when using this extension, it is important to understand the following:
Selenide tests are run to set up the capabilities we want;
the user wants to run tests on Selenoid;
the user wants to use the video recording option on Selenoid.
I couldn’t think up anything better than to
classify tests implemented with the use of the Selenide library and annotations. An autotest developer must annotate a test class in which their Selenide tests are stored.
@Target(AnnotationTarget.CLASS)
annotation class SelenideTestI was thinking of a situation in which UI, API, and other tests could be stored in the same autotest repository, but this extension shouldn’t affect anything other than Selenide tests.
The 2nd and the 3rd points shall be solved using the flags. This can be easily done by using environment variables that hold a boolean value:
SELENOID_SUPPORT — for an option of running tests through Selenoid;
SELENOID_VIDEO — for a video recording option.
The MVP of the solution can be represented in the code as follows (some additional methods necessary to run the extension are summarized and given without explanation, and may vary depending on your specific project):
Extension with a video recording function by test classes
class SelenideCoolExtension : BeforeAllCallback, AfterAllCallback {
override fun beforeAll(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Настраиваем Selenide конфигурацию только в случае запуска Selenide тестов
matchAnnotation?.let {
// Рекомендация от разработчиков Selenide — использовать однопоточный режим исполнения тестов
checkSingleThreadExecution(context)
val capabilities = DesiredCapabilities()
// Здесь могут идти дополнительные настройки самого браузера
// Настраиваем, если нужно запуск через Selenoid
if (checkSelenoidSupport()) {
Configuration.remote = "$selenoidUrl/wd/hub"
Configuration.driverManagerEnabled = false
// Настраиваем, если нужно, и опцию видеозаписи с уникальным именем для видео
// на основе метки имени тестового класса, данных CI/Local режима запуска,
// и UTC времени для удобства поиска в Selenoid capabilities.setCapability("selenoid:options", prepareSelenoidOptions(context))
}
// Полностью готовые capabilities пробрасываются в конфигурацию
Configuration.browserCapabilities = capabilities
}
}
override fun afterAll(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Закрываем сессию только для Selenide тестов
matchAnnotation?.let {
WebDriverRunner.closeWebDriver()
}
}
private fun parseSelenideTest(ctx: ExtensionContext): SelenideTest? {
return ctx
.testClass
.orElseThrow { -> IllegalStateException("Something wrong with context. Test must contain ancestor (test class)") }
.let { AnnotationUtils.findAnnotation(it, SelenideTest::class.java) }
.orElse(null)
}
private fun checkSingleThreadExecution(context: ExtensionContext) {
context.getConfigurationParameter("junit.jupiter.execution.parallel.enabled")?.ifPresent {
if (it.toBoolean()) throw IllegalStateException("This extension works only with single thread mode")
}
}
private fun checkSelenoidSupport(): Boolean {
return when (System.getenv(SELENOID_SUPPORT_ENV_PROPERTY_NAME)) {
"1" -> true
else -> false
}
}
private fun checkVideoRecorderSupport(): Boolean {
return when (System.getenv(SELENOID_VIDEO_SUPPORT_ENV_PROPERTY_NAME)) {
"1" -> true
else -> false
}
}
private fun prepareSelenoidOptions(context: ExtensionContext): Map<String, Any> {
val selenoidOptions = mutableMapOf<String, Any>(
"enableVNC" to true,
)
if (checkVideoRecorderSupport()) {
selenoidOptions["enableVideo"] = true
selenoidOptions["videoName"] = generateVideoName(context)
}
return selenoidOptions
}
private fun generateVideoName(context: ExtensionContext): String {
val ciJobId = System.getenv(GITLAB_JOB_ID_ENV_PROPERTY_NAME) ?: "Local"
val currentTime = ZonedDateTime.now(Clock.systemUTC())
.format(DateTimeFormatter.ofPattern(SELENOID_VIDEO_DATETIME_PATTERN))
val suiteName = context
.testClass
.orElseThrow().let {
AnnotationUtils.findAnnotation(it, DisplayName::class.java)
}.orElseThrow {
IllegalStateException("Test suite must be annotated with @DisplayName")
}.value
.splitToSequence(" ")
.joinToString("-")
val resultName = "$ciJobId-$suiteName-$currentTime.mp4"
return resultName
}
private val selenoidUrl by lazy {
System.getenv(SELENOID_URL_ENV_PROPERTY_NAME) ?: SELENOID_DEFAULT_SERVICE_URL
}
}I deliberately try to write a simple and clear code for such solutions, so that the solution can be used by many people and teams.
A synthetic test helped verify this solution:
Run two test classes:
@SelenideTest
@DisplayName("A")
class ASuite {
@Test
fun aTest() {
open("https://www.google.com")
Selenide.sleep(5 * 1000)
}
}@SelenideTest
@DisplayName("B")
class BSuite {
@Test
fun bTest() {
Selenide.open("https://www.yandex.ru")
Selenide.sleep(5 * 1000)
}
}Make sure that exactly two videos with the right name are created in Selenoid, with each video containing execution of all tests of the test class.
I was more than happy with the result:
How to Create a Time Line and Timestamps for a Test Suite
The challenge here was that I wanted to make a simple and clear solution, while also creating all these timestamps as accurately as possible given the behaviour of the tests on Selenoid.
Inspired,I used the JUni5 official documentation and the algorithm came to my mind right away:
When you start the extension, create a field inside it that contains a stopwatch starting at zero.
Before running all the tests in the class, have a SELENOID_VIDEO flag checked, and register a custom WebdriverListener for a webdriver that has a link to this counter. This listener should start counting time only when the first interaction with the browser in the autotests happens. I chose to link it to triggering a webdriver navigate() method, as this method is called when the Selenide library open() method is called, which runs any autotest. This listener should also store the state of the first call to the navigate method inside itself to correctly handle and ignore subsequent navigate calls.
Before running a particular autotest, read the state of the stopwatch and register it in some variable. Store it in the log as well. We can do that by implementing a new interface inside our extension — BeforeEachCallback that allows the addition of a function to be executed before running an individual autotest.
If the test fails for any reason, you need to obtain the current state of the stopwatch and its state at the moment the test is run, and add it all to the log. We can do that by implementing a new interface inside our extension — TestWatcher that allows the addition of a function after a test has been completed successfully, has failed, or has been skipped.
After executing all of the autotests, reset the stopwatch and call back WebDriverListener.
Don’t forget that all these functions should be executed only for Selenide tests.
Let’s take a look at a new stage of MVP implementation of our extension:
Extension with a timestamp support
private val logger = KotlinLogging.logger {}
class SelenideCoolExtension : BeforeAllCallback, AfterAllCallback, TestWatcher, BeforeEachCallback {
// Инциализируем переменную для регистрации меток времени
private val timeSnapshot = AtomicLong(0)
// Инициализируем счетчик времени
private val stopwatch: Stopwatch = Stopwatch.createUnstarted()
private lateinit var extensionListener: WebDriverListener
override fun beforeAll(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Настраиваем Selenide конфигурацию только в случае запуска Selenide тестов
matchAnnotation?.let {
// Рекомендация от разработчиков Selenide - использовать однопоточный режим исполнения тестов
checkSingleThreadExecution(context)
val capabilities = DesiredCapabilities()
// Здесь могут идти дополнительные настройки самого браузера
// Настраиваем, если нужно запуск через Selenoid
if (checkSelenoidSupport()) {
Configuration.remote = "$selenoidUrl/wd/hub"
Configuration.driverManagerEnabled = false
// Настраиваем, если нужно, и опцию видеозаписи с уникальным именем для видео
// на основе метки имени тествого класса, данных CI/Local режима запуска,
// и UTC времени для удобства поиска в Selenoid
capabilities.setCapability("selenoid:options", prepareSelenoidOptions(context))
}
// Регистрируем listener только при включенной опции записи видео
if (checkVideoRecorderSupport()) {
extensionListener = CounterListener(stopwatch)
WebDriverRunner.addListener(extensionListener)
}
// Полностью готовые capabilities пробрасываются в конфигурацию
Configuration.browserCapabilities = capabilities
}
}
override fun beforeEach(context: ExtensionContext) {
val matchAnnotation = parseSelenideTest(context)
matchAnnotation?.let {
// Фиксируем время старта теста только при включенной опции записи видео
if (checkVideoRecorderSupport()) {
stopwatch.elapsed(TimeUnit.SECONDS).let { time ->
timeSnapshot.set(time)
}
}
}
}
override fun testFailed(context: ExtensionContext, cause: Throwable?) {
val matchAnnotation = parseSelenideTest(context)
// Логируем время старта теста и время падения по отношению к таймлайну видео
// только при включенной опции записи видео
matchAnnotation?.let {
if (checkVideoRecorderSupport()) {
logger.error { "Test is failed. Test start time in video - ${timeSnapshot.get()} sec" }
stopwatch.elapsed(TimeUnit.SECONDS).let {
logger.error { "Test failure is on $it sec" }
}
}
}
}
override fun afterAll(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Закрываем сессию только для Selenide тестов
// Отзываем listener
// А также логируем общее время видеозаписи
matchAnnotation?.let {
if (checkVideoRecorderSupport()) {
logger.warn { "Test run seconds: ${stopwatch.elapsed(TimeUnit.SECONDS)} seconds" }
stopwatch.reset()
}
WebDriverRunner.closeWebDriver()
if (this::extensionListener.isInitialized) {
WebDriverRunner.removeListener(extensionListener)
}
}
}
private fun parseSelenideTest(ctx: ExtensionContext): SelenideTest? {
return ctx
.testClass
.orElseThrow { -> IllegalStateException("Something wrong with context. Test must contain
ancestor (test class)") }
.let { AnnotationUtils.findAnnotation(it, SelenideTest::class.java) }
.orElse(null)
}
private fun checkSingleThreadExecution(context: ExtensionContext) {
context.getConfigurationParameter("junit.jupiter.execution.parallel.enabled")?.ifPresent {
if (it.toBoolean()) throw IllegalStateException("This extension works only with single thread mode")
}
}
private fun checkSelenoidSupport(): Boolean {
return when (System.getenv(SELENOID_SUPPORT_ENV_PROPERTY_NAME)) {
"1" -> true
else -> false
}
}
private fun checkVideoRecorderSupport(): Boolean {
return when (System.getenv(SELENOID_VIDEO_SUPPORT_ENV_PROPERTY_NAME)) {
"1" -> true
else -> false
}
}
private fun prepareSelenoidOptions(context: ExtensionContext): Map<String, Any> {
val selenoidOptions = mutableMapOf<String, Any>(
"enableVNC" to true,
)
if (checkVideoRecorderSupport()) {
selenoidOptions["enableVideo"] = true
selenoidOptions["videoName"] = generateVideoName(context)
}
return selenoidOptions
}
private fun generateVideoName(context: ExtensionContext): String {
val ciJobId = System.getenv(GITLAB_JOB_ID_ENV_PROPERTY_NAME) ?: "Local"
val currentTime = ZonedDateTime.now(Clock.systemUTC())
.format(DateTimeFormatter.ofPattern(SELENOID_VIDEO_DATETIME_PATTERN))
val suiteName = context
.testClass
.orElseThrow().let {
AnnotationUtils.findAnnotation(it, DisplayName::class.java)
}.orElseThrow {
IllegalStateException("Test suite must be annotated with @DisplayName")
}.value
.splitToSequence(" ")
.joinToString("-")
val resultName = "$ciJobId-$suiteName-$currentTime.mp4"
return resultName
}
private val selenoidUrl by lazy {
System.getenv(SELENOID_URL_ENV_PROPERTY_NAME) ?: SELENOID_DEFAULT_SERVICE_URL
}
}MVP implementation of our own WebDriverListener looks like this:
private val logger = KotlinLogging.logger {}
class CounterListener(private val timer: Stopwatch) : WebDriverListener {
private val sessionCreated = AtomicBoolean(false)
override fun afterAnyCall(target: Any?, method: Method?, args: Array<out Any>?, result: Any?) {
if (method?.name == LISTENER_TARGET_METHOD_FOR_TRACKING) {
if (!sessionCreated.get()) {
logger.warn { "Session start was intercepted, starting counting time" }
if (!timer.isRunning) {
timer.start()
}
sessionCreated.set(true)
}
}
}
override fun afterQuit(driver: WebDriver?) {
logger.warn { "Resetting time counter due to driver close" }
sessionCreated.set(false)
}
}I have chosen a Guava stopwatch with proven quality. We will count in milliseconds, though that is inexact, but for the first implementation that will do.
To check the improvements, I wrote the following synthetic test:
Run a test class in which all test methods are implemented as failing.
@SelenideTest
@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
@DisplayName("A")
class ASuite {
private val EXPECTED_TITLE = "совсем случайный текст"
@Test
@Order(1)
fun aTest() {
open("https://www.google.com")
element(By.xpath("//input[@title='Поиск']")).setValue("Проверка синтетического теста1").submit()
Assertions.assertEquals(title(), EXPECTED_TITLE)
}
@Test
@Order(2)
fun bSyntheticFailedTest() {
open("https://www.google.com")
element(By.xpath("//input[@title='Поиск']")).setValue("Проверка синтетического теста2").submit()
Assertions.assertEquals(title(), EXPECTED_TITLE)
}
@Test
@Order(3)
fun cTest() {
open("https://www.google.com")
element(By.xpath("//input[@title='Поиск']")).setValue("Проверка синтетического теста3").submit()
Assertions.assertEquals(title(), EXPECTED_TITLE)
}
}Use the logs to see the timestamps.
In the generated video, go to the second of failure from log 3.
After running these tests, the following is displayed in the logs:
[Test worker] INFO com.codeborne.selenide.drivercommands.CreateDriverCommand - Created webdriver in thread 1: RemoteWebDriver -> RemoteWebDriver: chrome on LINUX (a7d95bf7873bfc86345b131ee7e8a3ae)
[Test worker] INFO com.codeborne.selenide.drivercommands.CreateDriverCommand - Add listeners to webdriver: [tech.inno.qa.core.ui.extension.CounterListener@1cdd31a4]
[Test worker] WARN tech.inno.qa.core.ui.extension.CounterListener - Session start was intercepted, starting counting time
[Test worker] ERROR tech.inno.qa.core.ui.extension.SelenideExtension - Test is failed. Test start time in video - 0 ms
[Test worker] ERROR tech.inno.qa.core.ui.extension.SelenideExtension - Test failure is on 1748 ms
expected: <совсем случайный текст> but was: <Проверка синтетического теста1 - Поиск в Google>
Expected :совсем случайный текст
Actual :Проверка синтетического теста1 - Поиск в Google
[Test worker] ERROR tech.inno.qa.core.ui.extension.SelenideExtension - Test is failed. Test start time in video - 2204 ms
[Test worker] ERROR tech.inno.qa.core.ui.extension.SelenideExtension - Test failure is on 5044 ms
expected: <совсем случайный текст> but was: <Проверка синтетического теста2 - Поиск в Google>
Expected :совсем случайный текст
Actual :Проверка синтетического теста2 - Поиск в Google
[Test worker] ERROR tech.inno.qa.core.ui.extension.SelenideExtension - Test is failed. Test start time in video - 5411 ms
[Test worker] ERROR tech.inno.qa.core.ui.extension.SelenideExtension - Test failure is on 7773 ms
expected: <совсем случайный текст> but was: <Проверка синтетического теста3 - Поиск в Google>
Expected :совсем случайный текст
[Test worker] WARN tech.inno.qa.core.ui.extension.SelenideExtension - Test run: 7779 ms
[Test worker] INFO com.codeborne.selenide.drivercommands.CloseDriverCommand - Close webdriver: 1 -> Decorated {RemoteWebDriver: chrome on LINUX (a7d95bf7873bfc86345b131ee7e8a3ae)}...
[Test worker] WARN tech.inno.qa.core.ui.extension.CounterListener - Resetting counter state due to driver close
[Test worker] INFO com.codeborne.selenide.drivercommands.CloseDriverCommand - Closed webdriver 1 in 117 msAs an example, let’s open the video at the moment of an error of test N3 — 7,773 ms (at 7-8 seconds on a standard HTML5 video container) and verify that a failure happens there:
Some notes
Unfortunately, I didn’t manage to achieve precision in timestamps, because the entire process is built on wrappers around events. I didn’t manage to find a reasonable way to capture the event of the start of video recording itself, as this is handled by the Selenoid function. The stopwatch can miss by a tiny fraction of a second, depending on the networking speed.
In my version, it helped a lot that the CI runners which run autotests on have very fast access to Selenoid, since they are on the same infrastructure subnet.
Overall, I’m happy with the result. The timestamps are more or less accurate, so let’s move on.
How to Link Video Timestamps with Allure Report
Again, I had to be somewhat creative. I couldn’t find a predictable solution on the go, but I had the following ideas:
it might be easier to crop a piece of video from the main video of the test suite using timestamps, and attach it to the report as an “attachment” for a particular test. As it turned out, this was not easier, and I did not really want to use JVM video processing libraries.
as an alternative, don’t bother at all, and just send a link to the video of a particular test to the Allure report, as well as instructing users of the solution to look in the logs and go to the required seconds manually. As it turned out, end users wanted to see all the facts of test execution in one place, so this option also was discarded.
I soon realized that there was nothing wrong with using pieces of HTML markup as “attachments”. This means that it will be possible to:
control the behaviour of a particular HTML5 element (in this case, <video/>) at the javascript level;
writing whatever users want in this markup.
This is the algorithm that came to my mind:
Before running all the tests in the class, we save the generated name for a Selenoid video in the field of our extension.
After any of the tests fails, we generate an HTML5 code with self-writing is that modifies a video player, and opens a video only for the duration of the failed test running. For example, it shows only 30 seconds from a 10-minute video.
The generated code is saved as an “attachment” of the particular test using AllureLifecycle. This class lets you safely handle the data for the Allure report. We will develop a separate method for this.
So in the extension, you just need to add behaviour to the implementation of the TestWatcher interface, having previously implemented the function of generating an improved “attachment” for Allure.
To improve videos, I decided to wrap a standard video container in the Plyr player. It seemed to be a better wrapper over a standard video player, and it may come in handy if at some point users wish to have more features to interact with video reports.
So how do you make the player open the video only at the desired range of seconds? The answer lies in an outdated W3C specification for dealing with media fragments. This helps solve the problem of opening the same video on each Allure “attachment”, but at different timestamps.
Let’s look at the final code:
Final Extension version
class SelenideExtension1 : BeforeAllCallback, AfterAllCallback, BeforeEachCallback, TestWatcher {
// Инциализируем переменную для регистрации меток времени
private val testStartTimeSnapshot = AtomicLong(0)
// Инициализируем счетчик времени
private val stopwatch: Stopwatch = Stopwatch.createUnstarted()
private lateinit var extensionListener: WebDriverListener
// Инициализируем переменную для хранения названия видео-файла
// Так как все тесты в классе принадлежат одному видео
private val currentVideoLink = AtomicReference("")
override fun beforeAll(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Настраиваем Selenide конфигурацию только в случае запуска Selenide тестов
matchAnnotation?.let {
// Рекомендация от разработчиков Selenide - использовать однопоточный режим исполнения тестов
checkSingleThreadExecution(context)
val capabilities = DesiredCapabilities()
// Здесь могут идти дополнительные настройки самого браузера
// Настраиваем, если нужно запуск через Selenoid
if (checkSelenoidSupport()) {
Configuration.remote = "$selenoidUrl/wd/hub"
Configuration.driverManagerEnabled = false
// Настраиваем, если нужно, и опцию видеозаписи с уникальным именем для видео
// на основе метки имени тествого класса, данных CI/Local режима запуска,
// и UTC времени для удобства поиска в Selenoid
capabilities.setCapability("selenoid:options", prepareSelenoidOptions(context))
}
// Полностью готовые capabilities пробрасываются в конфигурацию
Configuration.browserCapabilities = capabilities
// Регистрируем listener только при поднятом флаге необходимости записи видео
if (checkSelenoidVideoSupport()) {
extensionListener = CounterListener(stopwatch)
WebDriverRunner.addListener(extensionListener)
}
}
}
override fun beforeEach(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
matchAnnotation?.let {
// Здесь, если нужно, чистим состояние браузера (cookies, localstorage, etc..)
// Фиксируем время старта теста только при включенной опции записи видео
if (checkSelenoidVideoSupport()) {
testStartTimeSnapshot.set(stopwatch.elapsed(TimeUnit.MILLISECONDS))
}
}
}
override fun testFailed(context: ExtensionContext, cause: Throwable?) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Логируем время старта теста и время падения по отношению к таймлайну видео
// только для Selenide тестов, и при включенной опции записи видео
// Готовим вложение с метками времени в случае падения теста
matchAnnotation?.let {
if (checkSelenoidVideoSupport()) {
val testStartTime = testStartTimeSnapshot.get()
logger.error { "Test is failed. Test start time in video - ${testStartTime} ms" }
stopwatch.elapsed(TimeUnit.MILLISECONDS).let {
logger.error { "Test failure is on $it ms" }
prepareAllureVideoAttachment(testStartTime, it)
}
}
}
}
override fun afterAll(context: ExtensionContext) {
// Проверяем, что тестовый класс содержит Selenide тесты
val matchAnnotation = parseSelenideTest(context)
// Закрываем сессию только для Selenide тестов,
// отзываем listener, и также логируем общее время видеозаписи
matchAnnotation?.let {
if (checkSelenoidVideoSupport()) {
logger.warn { "Test run: ${stopwatch.elapsed(TimeUnit.MILLISECONDS)} ms" }
stopwatch.reset()
}
if (WebDriverRunner.hasWebDriverStarted()) {
WebDriverRunner.closeWebDriver()
}
if (this::extensionListener.isInitialized) {
WebDriverRunner.removeListener(extensionListener)
}
}
}
private val selenoidUrl by lazy {
System.getenv(SELENOID_URL_ENV_PROPERTY_NAME) ?: SELENOID_DEFAULT_SERVICE_URL
}
private fun parseSelenideTest(ctx: ExtensionContext): SelenideTest? {
return ctx.testClass.orElseThrow { -> IllegalStateException("Something wrong with context. Test must contain ancestor (test class)") }
.let { AnnotationUtils.findAnnotation(it, SelenideTest::class.java) }.orElse(null)
}
private fun checkSingleThreadExecution(context: ExtensionContext) {
context.getConfigurationParameter("junit.jupiter.execution.parallel.enabled")?.ifPresent {
if (it.toBoolean()) throw IllegalStateException("This extension works only with single thread mode")
}
}
private fun checkSelenoidSupport(): Boolean {
return when (System.getenv(SELENOID_SUPPORT_ENV_PROPERTY_NAME)) {
"1" -> true
else -> false
}
}
private fun checkSelenoidVideoSupport(): Boolean {
return when (System.getenv(SELENOID_VIDEO_SUPPORT_ENV_PROPERTY_NAME)) {
"1" -> true
else -> false
}
}
private fun checkScreenResolution(): String {
val resolution = System.getenv(SELENOID_DEFAULT_SCREEN_RESOLUTION_ENV_PROPERTY_NAME) ?: "normal"
return try {
ScreenResolution.valueOf(resolution.uppercase()).value
} catch (e: IllegalArgumentException) {
ScreenResolution.NORMAL.value
}
}
private fun prepareSelenoidOptions(context: ExtensionContext): Map<String, Any> {
val selenoidOptions = mutableMapOf<String, Any>(
"enableVNC" to true, "enableLog" to true
)
if (checkSelenoidVideoSupport()) {
selenoidOptions["enableVideo"] = true
selenoidOptions["videoName"] = generateVideoName(context)
selenoidOptions["videoFrameRate"] to 24
selenoidOptions["screenResolution"] = checkScreenResolution()
}
return selenoidOptions
}
private fun generateVideoName(context: ExtensionContext): String {
val ciJobId = System.getenv(GITLAB_JOB_ID_ENV_PROPERTY_NAME) ?: "Local"
val currentTime = ZonedDateTime.now(Clock.systemUTC()).format(DateTimeFormatter.ofPattern(SELENOID_VIDEO_DATETIME_PATTERN))
val suiteName = context.testClass.orElseThrow().let {
AnnotationUtils.findAnnotation(it, DisplayName::class.java)
}.orElseThrow {
IllegalStateException("Test suite must be annotated with @DisplayName")
}.value.splitToSequence(" ").joinToString("-")
val resultName = "$ciJobId-$suiteName-$currentTime.mp4"
// Сохраняем имя файла
currentVideoLink.set(resultName)
return resultName
}
private fun generateVideoLink(): String {
return "$selenoidUrl/video/${currentVideoLink.get()}"
}
private fun prepareAllureVideoAttachment(startTime: Long, failTime: Long) {
val lifecycle = Allure.getLifecycle()
val startTimeFormatted = DurationFormatUtils.formatDurationHMS(startTime)
val failTimeFormatted = DurationFormatUtils.formatDurationHMS(failTime)
lifecycle.currentTestCase.ifPresent {
lifecycle.addAttachment(
"Recorded video", "text/html", ".html", """
<html>
<head>
<script src="https://cdn.plyr.io/3.7.2/plyr.js"></script>
<link rel="stylesheet" href="https://cdn.plyr.io/3.7.2/plyr.css" />
</head>
<body>
<p>Время прохождения теста: $startTimeFormatted - $failTimeFormatted</p>
<video controls playsinline id="player" onpause="this.load()">
<source src='${generateVideoLink()}#t=${startTimeFormatted},${failTimeFormatted}' type='video/mp4'>
</video>
<script>
const player = new Plyr(window.document.getElementById('player'), {
title: 'Selenoid Player',
invertTime: false
});
</script>
</body>
</html>
""".trimIndent().toByteArray()
)
}
}
}To test the functionality you will need to:
Run Selenide tests. What we said in the previous paragraph of the article also applies here;
Take a look at the generated timestamps;
Generate an Allure report, either locally or remotely in Allure TestOPS;
Make sure that timestamps in the logs match a fragment of the video with a failed test in the Allure report.
After running the tests, all we need to do is to look at the log snippet of any failed test:
It works!
The video opens on a particular timespan, and after it finishes, it returns to the start time of the failed test.
Voila!
Beauty Will Save Autotests
A functional and user-friendly solution improves the UI autotest reporting process. Yes, it’s made as “ad-hoc” solution, but it lets you solve the task efficiently. It once again reminds us that with due perseverance and motivation, it is possible to automate and customize processes.
I’m currently working on MVP improvement. There are ideas to make video reports more pleasant by implementing text description of a test in the video of a test suite using subtitles, but more on that another time.
Please leave your tips on optimization and questions on the implementation in the comments.