Virtuous Programmer

Test Automation with Robot Framework

The Plan

Robot Framework is a functional test framework that makes it possible for its users to:

• Write tests in easily understood language
• Organize what tests are run for specific purposes
• Generate both high level and detailed test results

Notice that what it does not do is any specific testing activity. Instead it acts as a front end for libraries like Selenium and AutoIT which perform the actions themselves. For the purpose of this tutorial I’ll be focusing on web automation using Selenium as the back-end.

What to Expect

When this article is complete:

• You will have:

  • An installation of Robot Framework with Selenium
  • Several tests for a basic login page

• You will know:

  • How to use Robot Framework(RF):
    • To drive Selenium
    • To write functional tests
    • To write data-driven tests
    • To select or exclude tests by their tags
  • How to read an RF results file
  • How to read an RF log file
  • To find identifiers on an HTML page

Installing Robot Framework and Selenium

Before installing Robot Framework, you need to install Python 2.5 or later. Robot Framework hasn’t been updated to work with Python 3, so currently 2.7 is your best bet. If you don’t already have Python installed you can download it from here.

Next you’ll need to download Robot Framework and Selenium. For each of the installs select the default values.

Finally, add your python installation and robot installation to your path. If you’ve installed Python 2.7 it will be: “c:\Python27\;c:\Python27\Scripts\;”.

Files Used in this Project

• Simple Test
• Login Test Directory

Executing Robot Framework Tests

In Robot Framework each file contains one or more tests. That file is treated as a test suite. Every directory that contains a test suite file or directory is also a test suite. When RF is executed on a directory it will recurse through all files and directories of the correct kind except those that start with an underscore character.

To run the tests in “SimpleTest.txt” navigate to the directory that contains it in the command line and type:

pybot SimpleTest.txt

There are arguments that will give you greater control over what tests are executed, but I’ll get to those later in this article.

Notice that while the test is running there are two windows. One is the window in which the test is being executed, the other is the controller window that Selenium is executing the JavaScript that controls the other window from.

When the test is finished executing four files will have been generated: report.html, log.html, output.xml and selenium_server_log.txt. Of these the only one’s you’ll generally need to be aware of are report.html and log.html.

Reading the Results

Your first stop after a test run is report.html. When you open it, if you’re lucky, you’ll see this happy image:

The background is green when all tests have passed, red if any have failed. If you need more detail than that the “Test Statistics” section gives how many tests have passed and failed sorted by criticality, tag and suite. The “Test Details” section gives how long the test took to run and, if it failed, what the fail message was, as can be seen here:

When a test fails, unless the message makes it immediately obvious, you’ll generally next need to read the log. If you click the failed test name, it will then open up the test logs:

The log gives a detailed view of the execution of each of the tests. While the information for all of the tests are available, the tests and steps that are failed are marked in red and already expanded.

The Code

Syntax and Formats

Robot Framework files are sorted into tables. The format of those tables is flexible, you may create them in tab-separated, html, ReStructured text, or just plain text. I’ve tried writing tests in each of these, HTML makes the prettiest files, and is also a pain to work with. In the end, plain text has been the best combination of nice to read for my customer and easy for me to maintain.

There are up to four tables in a test file, the “Testcases” table is mandatory, it contains all the tests you’re going to run, the optional tables are:

• Settings: Data needed across tests, like what libraries to use and how to setup and teardown tests.
• Variables: Variable names and values
• Keywords: These are function definitions, RF calls them keywords, there’s no difference.

The Test

Given a Login Page verify that when the correct username and password are supplied, it is possible to login.

A First Attempt

Settings

*** Settings ***
Library       Selenium Library

Each Table is marked out by *** <table name> *** and continues until the next table name is reached. The text version of the RF syntax is whitespace delimited. Each item under a table is flush to the left of the screen. If a keyword takes arguments then the follow it on the same line, separated by either a tab or two or more spaces. I generally space them out more than two for readability.

The Library setting imports the named library into the test suite. The Selenium Library allows its users to control a web browser automatically and ideally suited to the task at hand.

Identifying Elements on a Web Page

When automating any system, you have to have a way for your test software to find the elements that you’re interacting with. Selenium does this through one of several methods, depending on how automation friendly the software you’re testing is. If it’s very friendly then every element on the page has either a unique name or id attribute. If it’s more hostile then you can still identify any element through its xpath. I recommend using xpath only as a last resort. It’s a powerful tool, but makes your code hard to understand.

I wrote the login page, so it’s very test friendly. For simpler pages like this, I general just look at the sourcecode of the page to find out how to identify the elements, if you look at the Login Page, you’ll find that the identifiers are:

Test Cases

*** Testcases ***
Login Should Succeed When the Correct Username and Password are Entered
  Start Selenium Server
  Open Browser  http://zdiles.chaosnet.org/  ie
  Maximize Browser Window
  Input Text    uname  BUser
  Input Text    pwd    TestPass
  Click Button  login
  Page Should Contain  Welcome
  Close Browser
  Stop Selenium Server

I’ve named this file “SimpleTest.txt”. Here’s a line By line look at it, I’ve informally labeled the below steps with their purpose: Setup, Action, Verification, Teardown

1. A Testcases table contains one or more tests.
2. The first line will be the name of the test as used in the reports and logs.
3. Setup Task: To use Selenium a server has to be started that will receive commands and control a browser. Notice that keyword names can have single spaces in them.
4. Setup Task: A browser is opened to the page given, by default Firefox is used. If specified as it is here it can also use Internet Explorer or Chrome.
5. Setup Task: By default the browser window isn’t maximized. This fixes that.
6. Test Action: The Input Text keyword enters text into a field, it takes two arguments, the identifier for the element and the text to enter. This fills the username field.
7. Test Action: The next Input Text call fills the password field.
8. Test Action: One of the nice things about robot framework is that its keyword names tend to be pretty obvious. The argument passed to it identifies the Login Button.
9. Test Verification: Page Should Contain searches through all the text on the page and passes if it contains the given text and fails otherwise.
10. Teardown Task: Close Browser, this will shutdown the browser opened in line 4, if not run then it just leaves all the windows opened between tests, each test will ad more leading to clutter and confusion.
11. Teardown Task: Shuts down the Selenium Server, if this isn’t done then it can occasionally lead to trouble when you try to start the Selenium Server in a future test.

Suite Settings

The test as it’s written will faithfully perform the tests given, but has a couple of problems. When a test fails Robot Framework stops running the test to save time and avoid performing activities on a system that’s in a bad state. This means that if the verification step fails, then our teardown tasks will never be executed. To deal this problem, and make neater more readable test cases there are settings at both the test and suite level to setup and teardown tests.

Setup/Teardown

*** Settings ***
Library         Selenium Library
Suite Setup     Start Selenium Server
Suite Teardown  Stop Selenium Server
Test Setup      Open Browser  http://zdiles.chaosnet.org/  ie 
Test Teardown   Close All Browsers

Setup and Teardown can be set at both the test and the suite level. Each fires off at different points during a test run:

• Suite Setup: Fires once before any tests during the suite are run.
• Test Setup: Fires before each test is executed.
• Test Teardown: Fires at the end of each test execution, even when the test fails.
• Suite Teardown: Fires when all tests in the suite have completed, even when one of them has failed.

This slims our testcase down considerably:

*** Testcases ***
Login Should Succeed When the Correct Username and Password are Entered
  Maximize Browser Window
  Input Text    uname  BUser
  Input Text    pwd    TestPass
  Click Button  login
  Page Should Contain  Welcome

But we still have the Maximize Browser Window keyword, which really doesn’t improve the reader’s understanding of the intent of the test, and is likely to be used in the execution of any future testcase. Unfortunately the setup/teardown keywords can only take one keyword.

The Keywords Table

You can solve this by adding a keyword that calls both of these:

*** Keywords ***
Setup Test
  Open Browser  http://zdiles.chaosnet.org/  ie
  Maximize Browser Window

And change the Test Setup setting to use it:

*** Settings ***
Library         Selenium Library
Suite Setup     Start Selenium Server
Suite Teardown  Stop Selenium Server
Test Setup      Setup Test
Test Teardown   Close Browser

Making Your Tests More Readable

*** Testcases ***
Login Should Succeed When the Correct Username and Password are Entered
  Input Text    uname  BUser
  Input Text    pwd    TestPass
  Click Button  login
  Page Should Contain  Welcome

Now that I’ve gotten rid of the steps that obscure the intent of my test, it’s time to make the purpose of the tests themselves more clear by creating keywords with more meaningful names than those given, so we’ll add a couple more keywords to our Keywords table:

Enter Username  [Arguments]  ${username}
  Input Text  uname  ${username}

Enter Password  [Arguments]  ${password}
  Input Text  pwd  ${password}

Click the Login Button
  Click Button  login

Login Is Successful
  Page Should Contain  Welcome

Which introduces the use of settings in individual keywords. The syntax used for keyword and test settings(to be seen soon) is the same. A special element surrounded in square brackets, you can find a full listing of these in Robot Framework’s User Guide. Here I’m able to specify arguments that will be passed into the keywords from the test case that calls it. The syntax for variables is ${<variable name>} and can contain spaces.

Now my testcase is considerably more readable:

Login Should Succeed When the Correct Username and Password are Entered
  Enter Username  AUser
  Enter Password  TestPass
  Click the Login Button
  Login Is Successful

The Power of Tags

While it’s useful to be able to run every test in your test suite at once, often you only need to run a small subset of them to perform a smoke test, or to test only a specific feature. To this end you can tag tests and suites to indicate their function.

The code below is the same as what I’ve constructed with one more test, which should fail every time, and a couple of tags (Keywords left out because they don’t change):

*** Settings ***
Library         Selenium Library
Resource        LoginKeywords.txt
Suite Setup     Start Selenium Server
Suite Teardown  Stop Selenium Server
Test Setup      Login Test Setup
Test Teardown   Login Test Teardown
Force Tags      FunctionalTest
Default Tags    ValidTest

*** Testcases ***
Login Should Succeed When No Username Is Given
  [Tags]  InvalidTest
  Enter Username  ${empty}
  Enter Password  TestPass
  Click the Login Button
  Login Is Successful

Login Should Succeed When the Correct Username and Password are Entered
  Enter Username  AUser
  Enter Password  TestPass
  Click the Login Button
  Login Is Successful

One Test level setting has been added, tagging the first test with InvalidTest. At the suite level there are two more. Force Tags ensures that every test in the suite will be tagged as a FunctionalTest. Default Tags set’s a test’s tags to ValidTest unless the test has a tag setting of its own like the first test.

There are two benefits from adding descriptive tags to your tests. When you run your tests your test results will be sorted by them. This will frequently help determine where an individual problem is. Additionally you can determine which tests get executed. For example when you use:

pybot -i InvalidTest SimpleTest.txt

Will only include the tests with that tag, while:

pybot -e InvalidTest SimpleTest.txt

Will exclude any test with that tag.

Keyword Files

Keywords like Enter Username will be used in nearly every test you write for this page. It’s often helpful to break up your test suites into multiple files, or even multiple directories. To avoid copy pasting the same set of keywords into every file, you can break them out into a separate resource file. The easiest way in our current case is to just cut everything from *** Keywords *** on down and put it in a new file called “LoginKeywords.txt”

We’re getting to a point where this is well beyond a simple test, so while you’re at it rename “SimpleTest.txt” to “LoginTest.txt” and add one more line to the Settings section of LoginTest.txt, like so:

*** Settings ***
Library         Selenium Library
Resource        LoginKeywords.txt
Suite Setup     Start Selenium Server
Suite Teardown  Stop Selenium Server
Test Setup      Login Test Setup
Test Teardown   Login Test Teardown
Force Tags      FunctionalTest
Default Tags    ValidTest

Directories as Suites

I’m going to add more test files, so to organize them I’m going to create a directory “LoginTest” and put “LoginTest.txt” and “LoginKeywords.txt” in it. Now if I execute pybot from the directory above the “LoginTest” directory as:

pybot LoginTest

It will execute all of the test files in LoginTest.

Init Files

As stated above, both directories and files are treated as test suites by Robot Framework. You can create a Settings section for a directory by adding the file “__init__.txt” and putting a Settings table in it. Since I’m about to add more tests and they’re all going to need the same setup and teardowns, I’ll move a few more settings from “LoginTest.txt” to “__init__.txt” giving us:

*** Settings ***
Library         Selenium Library
Resource        LoginKeywords.txt
Suite Setup     Start Selenium Server
Suite Teardown  Stop Selenium Server
Test Setup      Login Test Setup
Test Teardown   Login Test Teardown

And now the Settings section of “LoginTest.txt” is:

*** Settings ***
Library         Selenium Library
Resource        LoginKeywords.txt
Force Tags      FunctionalTest
Default Tags    ValidTest

Data Driven Tests

Data driven tests are designed to repeatedly run the same test with changes in the data is being used for inputs. This allows the person writing the tests to create a large number of tests scenarios with minimal effort.

Robot Framework has a special setting called Test Template that lets you create files that are dedicated to Data Driven test suites:

*** Settings ***
Library         Selenium Library
Resource        LoginKeywords.txt
Test Template   Login Should Fail When
Force Tags      DataDriven  ValidTest

*** Testcases ***       Username  Password
The Password is Wrong   AUser     TestFail
The Password is Empty   AUser     ${empty} 
Both Fields are Empty   ${empty}  ${empty} 

*** Keywords ***
Login Should Fail When   [Arguments]  ${username}  ${password}
  Enter Username  ${username}
  Enter Password  ${password}
  Click the Login Button
  Login Is Unsuccessful

Using Login Should Fail When as the template means that each of the three testcases use it as their only keyword, and apply the arguments given to it. You can see on the first line of the Testcases table there are two columns set “Username” and “Password”. Robot Framework ignores anything in the first line of the table after the name, the column names are strictly for the reader’s benefit. The other new element is the ${empty} variable, this indicates an empty string.

For this suite you can see I’ve added one more new keyword to “LoginKeywords.txt” called “Login Is Unsuccessful.” It is the negation of “Login Is Successful”:

Login is Unsuccessful
  Run Keyword And Expect Error  *  Login Is Successful

Run Keyword and Expect Error does exactly what it says. The second argument is the error expected, * will allow for any error. The third argument is the keyword to run, then the follow arguments get passed to the keyword in the third argument. This is a useful way to define negative test cases.

Variables

When your tests get even moderately complicated there’s going to be data that needs to change from one run to the next or depending on the context you’re running it in. Use a variable table to minimize the pain of dealing with this. For my tests I have two variables that I’m concerned with, the URL under test and the browser used, pulling these out as variables in the keyword file gives me:

*** Variables ***
${login address}=  http://zdiles.chaosnet.org/
${browser}=  ie


*** Keywords ***
Enter Username  [Arguments]  ${username}
  Input Text  uname  ${username}

Enter Password  [Arguments]  ${password}
  Input Text  pwd  ${password}

Click the Login Button
  Click Button  login
 
Login Is Successful
  Page Should Contain  Welcome

Login is Unsuccessful
  Run Keyword And Expect Error  *  Login Is Successful

Login Test Setup
  Open Browser  ${login address}  ${browser}
  Maximize Browser Window 

Notice that when they’re assigned the variable names have an = appended to them, this is optional, but it makes the code a little more clear.

Final Summary

I’ve found Robot Framework to be powerful tool for designing and executing tests. By adding functional tests to your project you can improve its overall quality. Enjoy.

Coming Up

Next week I’ll start with Smalltalk, using the Pharo environment which is redesigned from Squeak to be a powerful professional development tool.

Resources

• Robot Framework
• Selenium Library Docs
• Robot Framework User Guide