The demo of this script can be seen on youtube.

The full script can be downloaded at here.

The logic takes information from the Sikuli scripts that read the screen for product information and then, after processing that information, uses Sikuli scripting to interact with the Magic Online interface to finish the transaction.

Read the README on his GITHUB for further details. There is a full demonstration of the bot in the youtube video below.

The script to download a bunch of files.

The goal of this script is to download all ZIP files whose file name starts with “Sikuli-IDE-osx” in a web page. To find all ZIP files is easy, because they have an identifiable icon. However, we can’t simply click on the icon. We need to click on the hyperlink next to the icon, instead. The really tricky thing is, how to know a hyperlink that starts with “Sikuli-IDE-osx”?

The simplest solution is capturing a screenshot of the text “Sikuli-IDE-osx”, and putting the screenshot as the parameter of click(). Unfortunately, this way is not really robust since the images of text only have tiny differences. Sikuli’s fuzzy image matching engine may treat “S1kul1-1DE-esx” as the same as what we want. So, a right way to do this is – extracting the text from the web page.

You may recall that there was an ancient technology named OCR (Optical Character Recognition), which recognizes text from scanned books. However, current OCR engines do not really work well. They are slow and inaccurate.

In fact, even without OCR, we still can extract text from the screen using Sikuli.

How?

Use the clipboard!

The following script tells you how to do that. At first, we use Sikuli to find all locations of the ZIP icon (line 7). On line 8, we sort the list of the matched locations by their y coordinate, it’s not necessary though. The key steps are line 10 and 11. We click on the space between the icon and the hyperlink next to it using the new target offset attribute of a pattern. (denoted by a small red cross in the ZIP icon on line 7. See the figure below for its setting window.)

And then we hold the mouse button and drag to the location 600-pixel right to the center of the icon. This step (line 2) actually simulates using a mouse to select a line of text with a dragDrop(). After the selection, we simply press Command+C (Ctrl-C on Windows) to copy the selected text into the system clipboard.

Copy (extract) text from a browser.

Aha! We got the precise text in the clipboard.

The rest of the script are trivial. Sikuli 0.10 provides a new API Env.getClipboard() to get the content of the system clipboard. We can use this function to get the text of the hyperlinks and then use Python’s string function (startswith) to filter out the files we need.

I also made a tiny video that shows how this script looks like when it is running.

Since the success of of Sikuli 0.9, we have heard lots of suggestions and feature requests from the world. Sikuli 0.9 was a research prototype for showing the possibilities of applying computer vision techniques to the desktop environment. It was cool, but not very solid and well designed. So, we decided to redesign and rewrite (almost) everything to make a faster, more robust,  more flexible Sikuli.

Sikuli 0.10 will be a more comprehensive visual automation library for both Java and Jython. Novices can still simply use the basic functions, such as click, type, and dragDrop, with screen shots without learning anything new. Advanced users can restrict the searching region on any screen more easily. You also can click on a position that is next to the found visual target. It is especially useful for scripting games that have an arrow to indicate what to do (Fig. 1).

Fig. 1. Clicks on the "hammer" that is below the found target "arrow".

Sikuli 0.10  provides a new programming model – visual event driven programming (Fig.2). You can run a Sikuli Script in background to monitor certain visual events (something appears, disappears, or changes), and the specified event handling function will be called once the monitoring events happen.

Fig. 2. The function "darker" is called when the "Charged" icon disappears.

Sikuli IDE 0.10 also becomes better. A command toolbar that lists the core Sikuli commands and their usage inserts a line of command with just one click. Now it is possible to write a Sikuli script without typing a single line of code!

Sikuli IDE 0.10

We also support internationalization(i18n) since Sikuli 0.10. We already have Traditional Chinese and German interfaces now. The whole interface can be easily translated to your languages. Come to the Translation page on our Launchpad page to help translation!

Sikuli IDE with a Traditional Chinese interface.

Low-level keyboard and mouse actions, such as keyDown(), keyUp(), mouseDown() and mouseUp() are also supported for advanced operations. In addition, multiple screen environments are also supported. It means you can run Sikuli on the second monitor without messing up the working desktop on the primary monitor.

Sikuli 0.10 also becomes more universal. It will support 64bit platforms, including 64bit Linux, 64bit Mac OS X, and 64bit Windows.

Finally, we know the importance of good documentation, so we will release a reference book: The Complete Guide to Sikuli Script, which contains the usage and examples of all classes and functions in Sikuli. We hope the guide can give you a better understanding of Sikuli and assist you in learning Sikuli without trial-and-error.

Can’t wait anymore?

In fact, you don’t need to wait too long. You can get it tomorrow!!

Sikuli is not only visual but musical. ram B has uncovered Sikuli’s musical potential with a creative script that plays a virtual piano. The script opens a browser and opens up an online virtual piano. Then it asks for an input to choose from a collection of two songs: Terminator’s theme and Godfather’s theme.

The whole script is shown below. You can also download the script to try it out on your computer. Enjoy!!

A while ago we posted several examples of automatic tweets on Twitter sent by Sikuli. Andrew Burton (a.k.a. jarod), from whom one of these tweets were originated, sent us a Twitter script he created that does much more than the simple example we showed last time. This script not only sends a tweet but also signs in one’s Twitter account. When the script is run, it prompts the user for signin id, password, and message. Then the script goes about looking for various buttons to click and text fields to type. In the process, it starts a web browser, types the address of Twitter, waits for the Twitter logo to appear, clicks on the sign-in button, types the login and password, clicks the message box, types the message, and finally clicks the update button to submit the tweet.

The whole script is shown below.

Many thanks to ram B for contributing another interesting script. The purpose of the script is to search for pictures on Picasa. The script first opens Picasa in Firefox browser. Then in this example it enters a keyword “cars” and clicks on the “search” button to submit the keyword. Unlike Flickr search, in Picasa, one has to click the “play” button even after clicking the slide show button. So this script will start the slide show increase the speed by 1 sec, hit the play button and will close the window after displaying around 10 pictures.

The full script can be seen below.

FFXI (i.e., Final Fantasy 11) is one of the most popular online RPG games, but notorious for its ridiculously long and tedious login process. firewing at the Bluegarttr created a Sikuli script to automate this process. By exporting the script to an executble (.skl) file and placing a shortcut on the desktop, all one needs to do to start playing FFXI is to double-click the shortcut. How easy!!

The full script annotated by the author is shown below.

Today’s showcase script is contributed by Ram B for tuning in to an online radio station on last.fm to listen to Beatles music. This script launches Firefox, types “www.last.fm” in the address bar, types “beatles” in the search box, clicks the “GO” button to confirm the search, and finally clicks on “Play The Beatles Radio” button to play. The entire script can be viewed below.

To display data, Sikuli currently supports only popup messages and debug messages, which can be very limiting. Garrett uses a very clever trick to get around this limitation. He uses Sikuli to open up another text editor and type the data in the editor to display.

It may not be clear to many Sikuli users that under the hood of Sikuli, there is a Java virtual machine providing a rich set of UI components such as tables and lists that can be called upon to display data. In the future release of Sikuli, we plan to introduce simple API calls to uses these components for output, while hiding the complexity of Java programming. Please stay tuned!