Testing Flash Applications or Java Applets Using Sikuli

When testing applications, sometimes it may be necessary to test the performance of a component that cannot be interacted with using Selenium. For example, when the application uses a map (such as Google or Yandex maps), or when there is a Flash application or Java applet. In such situations, a tool called Sikuli can be useful.

What is Sikuli? In short, it is a symbiosis of the Robot class and the OpenCV computer vision library. More specifically, it is a tool for automating actions on an application using screenshots of the elements of this application. That is, you think over the actions to perform a specific operation in the application, create screenshots of the elements that will be used for performing these actions, and write code using the Sikuli API, which will use the prepared images and perform the necessary actions. For example, if you want to check the operation of the page print button , first you need to take a screenshot of this button, then, using the Sikuli API, find the region where this button is located, hover the cursor over this region and click this button.

To install the latest version of Sikuli 1.0.1 and sikuli-api 1.0.2:

Download sikuli-api.jar https://code.google.com/archive/p/sikuli-api/downloads

Download sikuli-setup.jar https://launchpad.net/sikuli/+download

Run sikuli-setup. The following dialog box will appear:

Then go to the folder indicated in the dialog box and run runSetup.cmd

Click “OK” in the next dialog box.

For programming in Java, just select items 4 and 6.

Next, click “Setup Now”.

A confirmation window will appear. Click “YES”.

The installer will download the necessary files.

Then add two .jar files (sikuli-api and sikuli-java) to the classpath.

Let’s consider a simple example of using Sikuli. In this example, we will test changing the player resolution.

To do this, we will perform the following steps:

Since this article concentrates on getting started with this tool, we will not touch on the initialization and startup of Webdriver and navigation to the desired page. Instead, we’ll jump straight into the code that uses the Sikuli API.

First, we create global objects required for testing, namely Firefox, desktop region and mouse:

final App app = new App("Firefox");
final ScreenRegion s = new DesktopScreenRegion();
final Mouse mouse = new DesktopMouse();

Then we focus on the browser:

app.focus();

Declare the images that will be used in the test:

final Target targetSettings = new ImageTarget(new File("settings.PNG"));
final Target targetQuality = new ImageTarget(new File("quality.PNG"));
final Target target480p = new ImageTarget(new File("480p.PNG"));
final Target targetChecked480p = new ImageTarget(new File("checked_480p.PNG"));

Find the “settings” icon and click it:

findAndClick(s, mouse, targetSettings);

Check that there is the “quality” label:

Assert.assertNotNull(s.find(targetQuality));

Find and click the “480p” menu item:

findAndClick(s, mouse, target480p);

Find and click the “settings” icon:

findAndClick(s, mouse, targetSettings);

Check that there is the “quality” label:

Assert.assertNotNull(s.find(targetQuality));

Verify that the “480p” item is checked:

Assert.assertNotNull(s.find(targetChecked480p ));

Since there is a repetitive sequence of actions in the test (i.e. find and click), they were moved to a separate method:

private static void findAndClick(final ScreenRegion s, final Mouse
mouse, final Target target) {
final ScreenRegion targetRegion = s.find(target);
Assert.assertNotNull(targetRegion);
mouse.click(targetRegion.getCenter());
}

Sikuli can certainly be useful in situations where Selenium fails, but it also has disadvantages. For example, the fact that you have to take screenshots of the elements required for testing, and if the application interface changes, you will have to take screenshots again. The scale of the element in the screenshot must match the scale of the element in the application, otherwise Sikuli simply won’t find this image. This problem can be worked around by gradually rescaling the image and trying to find the region with that image every time the scale was changed. There were a few cases when the image search worked incorrectly, resulting in finding a completely different element.