Test Automation with Robot Framework
The Plan
Robot Framework is a functional test framework that makes it possible for its users to:
- Write tests in easily understood language
- Organize what tests are run for specific purposes
- Generate both high level and detailed test results
Notice that what it does not do is any specific testing activity. Instead it acts as a front end for libraries like Selenium and AutoIT which perform the actions themselves. For the purpose of this tutorial I’ll be focusing on web automation using Selenium as the back-end.
What to Expect
When this article is complete:
You will have:
- An installation of Robot Framework with Selenium
- Several tests for a basic login page
You will know:
- How to use Robot Framework(RF):
- To drive Selenium
- To write functional tests
- To write data-driven tests
- To select or exclude tests by their tags
- How to read an RF results file
- How to read an RF log file
- To find identifiers on an HTML page
- How to use Robot Framework(RF):
Installing Robot Framework and Selenium
Before installing Robot Framework, you need to install Python 2.5 or later. Robot Framework hasn’t been updated to work with Python 3, so currently 2.7 is your best bet. If you don’t already have Python installed you can download it from here.
Next you’ll need to download Robot Framework and Selenium. For each of the installs select the default values.
Finally, add your python installation and robot installation to your path. If you’ve installed Python 2.7 it will be: “c:\Python27\;c:\Python27\Scripts\;”.
Files Used in this Project
Executing Robot Framework Tests
In Robot Framework each file contains one or more tests. That file is treated as a test suite. Every directory that contains a test suite file or directory is also a test suite. When RF is executed on a directory it will recurse through all files and directories of the correct kind except those that start with an underscore character.
To run the tests in “SimpleTest.txt” navigate to the directory that contains it in the command line and type:
pybot SimpleTest.txt
There are arguments that will give you greater control over what tests are executed, but I’ll get to those later in this article.
Notice that while the test is running there are two windows. One is the window in which the test is being executed, the other is the controller window that Selenium is executing the JavaScript that controls the other window from.
When the test is finished executing four files will have been generated: report.html, log.html, output.xml and selenium_server_log.txt. Of these the only one’s you’ll generally need to be aware of are report.html and log.html.
Reading the Results
Your first stop after a test run is report.html. When you open it, if you’re lucky, you’ll see this happy image:
The background is green when all tests have passed, red if any have failed. If you need more detail than that the “Test Statistics” section gives how many tests have passed and failed sorted by criticality, tag and suite. The “Test Details” section gives how long the test took to run and, if it failed, what the fail message was, as can be seen here:
When a test fails, unless the message makes it immediately obvious, you’ll generally next need to read the log. If you click the failed test name, it will then open up the test logs:
The log gives a detailed view of the execution of each of the tests. While the information for all of the tests are available, the tests and steps that are failed are marked in red and already expanded.
The Code
Syntax and Formats
Robot Framework files are sorted into tables. The format of those tables is flexible, you may create them in tab-separated, html, ReStructured text, or just plain text. I’ve tried writing tests in each of these, HTML makes the prettiest files, and is also a pain to work with. In the end, plain text has been the best combination of nice to read for my customer and easy for me to maintain.
There are up to four tables in a test file, the “Testcases” table is mandatory, it contains all the tests you’re going to run, the optional tables are:
- Settings: Data needed across tests, like what libraries to use and how to setup and teardown tests.
- Variables: Variable names and values
- Keywords: These are function definitions, RF calls them keywords, there’s no difference.
The Test
Given a Login Page verify that when the correct username and password are supplied, it is possible to login.
A First Attempt
Settings
*** Settings ***
Library Selenium Library
Each Table is marked out by *** <table name> ***
and continues until the next table name is reached. The text version of the RF syntax is whitespace delimited. Each item under a table is flush to the left of the screen. If a keyword takes arguments then the follow it on the same line, separated by either a tab or two or more spaces. I generally space them out more than two for readability.
The Library
setting imports the named library into the test suite. The Selenium Library allows its users to control a web browser automatically and ideally suited to the task at hand.
Identifying Elements on a Web Page
When automating any system, you have to have a way for your test software to find the elements that you’re interacting with. Selenium does this through one of several methods, depending on how automation friendly the software you’re testing is. If it’s very friendly then every element on the page has either a unique name
or id
attribute. If it’s more hostile then you can still identify any element through its xpath. I recommend using xpath only as a last resort. It’s a powerful tool, but makes your code hard to understand.
I wrote the login page, so it’s very test friendly. For simpler pages like this, I general just look at the sourcecode of the page to find out how to identify the elements, if you look at the Login Page, you’ll find that the identifiers are:
Test Cases
*** Testcases ***
Login Should Succeed When the Correct Username and Password are Entered
Start Selenium Server
Open Browser http://zdiles.chaosnet.org/ ie
Maximize Browser Window
Input Text uname BUser
Input Text pwd TestPass
Click Button login
Page Should Contain Welcome
Close Browser
Stop Selenium Server
I’ve named this file “SimpleTest.txt”. Here’s a line By line look at it, I’ve informally labeled the below steps with their purpose: Setup, Action, Verification, Teardown
- A Testcases table contains one or more tests.
- The first line will be the name of the test as used in the reports and logs.
- Setup Task: To use Selenium a server has to be started that will receive commands and control a browser. Notice that keyword names can have single spaces in them.
- Setup Task: A browser is opened to the page given, by default Firefox is used. If specified as it is here it can also use Internet Explorer or Chrome.
- Setup Task: By default the browser window isn’t maximized. This fixes that.
- Test Action: The
Input Text
keyword enters text into a field, it takes two arguments, the identifier for the element and the text to enter. This fills the username field. - Test Action: The next
Input Text
call fills the password field. - Test Action: One of the nice things about robot framework is that its keyword names tend to be pretty obvious. The argument passed to it identifies the Login Button.
- Test Verification:
Page Should Contain
searches through all the text on the page and passes if it contains the given text and fails otherwise. - Teardown Task: Close Browser, this will shutdown the browser opened in line 4, if not run then it just leaves all the windows opened between tests, each test will ad more leading to clutter and confusion.
- Teardown Task: Shuts down the Selenium Server, if this isn’t done then it can occasionally lead to trouble when you try to start the Selenium Server in a future test.
Suite Settings
The test as it’s written will faithfully perform the tests given, but has a couple of problems. When a test fails Robot Framework stops running the test to save time and avoid performing activities on a system that’s in a bad state. This means that if the verification step fails, then our teardown tasks will never be executed. To deal this problem, and make neater more readable test cases there are settings at both the test and suite level to setup and teardown tests.
Setup/Teardown
*** Settings ***
Library Selenium Library
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Open Browser http://zdiles.chaosnet.org/ ie
Test Teardown Close All Browsers
Setup and Teardown can be set at both the test and the suite level. Each fires off at different points during a test run:
- Suite Setup: Fires once before any tests during the suite are run.
- Test Setup: Fires before each test is executed.
- Test Teardown: Fires at the end of each test execution, even when the test fails.
- Suite Teardown: Fires when all tests in the suite have completed, even when one of them has failed.
This slims our testcase down considerably:
*** Testcases ***
Login Should Succeed When the Correct Username and Password are Entered
Maximize Browser Window
Input Text uname BUser
Input Text pwd TestPass
Click Button login
Page Should Contain Welcome
But we still have the Maximize Browser Window
keyword, which really doesn’t improve the reader’s understanding of the intent of the test, and is likely to be used in the execution of any future testcase. Unfortunately the setup/teardown keywords can only take one keyword.
The Keywords Table
You can solve this by adding a keyword that calls both of these:
*** Keywords ***
Setup Test
Open Browser http://zdiles.chaosnet.org/ ie
Maximize Browser Window
And change the Test Setup
setting to use it:
*** Settings ***
Library Selenium Library
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Setup Test
Test Teardown Close Browser
Making Your Tests More Readable
*** Testcases ***
Login Should Succeed When the Correct Username and Password are Entered
Input Text uname BUser
Input Text pwd TestPass
Click Button login
Page Should Contain Welcome
Now that I’ve gotten rid of the steps that obscure the intent of my test, it’s time to make the purpose of the tests themselves more clear by creating keywords with more meaningful names than those given, so we’ll add a couple more keywords to our Keywords
table:
Enter Username [Arguments] ${username}
Input Text uname ${username}
Enter Password [Arguments] ${password}
Input Text pwd ${password}
Click the Login Button
Click Button login
Login Is Successful
Page Should Contain Welcome
Which introduces the use of settings in individual keywords. The syntax used for keyword and test settings(to be seen soon) is the same. A special element surrounded in square brackets, you can find a full listing of these in Robot Framework’s User Guide. Here I’m able to specify arguments that will be passed into the keywords from the test case that calls it. The syntax for variables is ${<variable name>}
and can contain spaces.
Now my testcase is considerably more readable:
Login Should Succeed When the Correct Username and Password are Entered
Enter Username AUser
Enter Password TestPass
Click the Login Button
Login Is Successful
The Power of Tags
While it’s useful to be able to run every test in your test suite at once, often you only need to run a small subset of them to perform a smoke test, or to test only a specific feature. To this end you can tag tests and suites to indicate their function.
The code below is the same as what I’ve constructed with one more test, which should fail every time, and a couple of tags (Keywords left out because they don’t change):
*** Settings ***
Library Selenium Library
Resource LoginKeywords.txt
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Login Test Setup
Test Teardown Login Test Teardown
Force Tags FunctionalTest
Default Tags ValidTest
*** Testcases ***
Login Should Succeed When No Username Is Given
[Tags] InvalidTest
Enter Username ${empty}
Enter Password TestPass
Click the Login Button
Login Is Successful
Login Should Succeed When the Correct Username and Password are Entered
Enter Username AUser
Enter Password TestPass
Click the Login Button
Login Is Successful
One Test level setting has been added, tagging the first test with InvalidTest
. At the suite level there are two more. Force Tags
ensures that every test in the suite will be tagged as a FunctionalTest
. Default Tags
set’s a test’s tags to ValidTest
unless the test has a tag setting of its own like the first test.
There are two benefits from adding descriptive tags to your tests. When you run your tests your test results will be sorted by them. This will frequently help determine where an individual problem is. Additionally you can determine which tests get executed. For example when you use:
pybot -i InvalidTest SimpleTest.txt
Will only include the tests with that tag, while:
pybot -e InvalidTest SimpleTest.txt
Will exclude any test with that tag.
Keyword Files
Keywords like Enter Username
will be used in nearly every test you write for this page. It’s often helpful to break up your test suites into multiple files, or even multiple directories. To avoid copy pasting the same set of keywords into every file, you can break them out into a separate resource file. The easiest way in our current case is to just cut everything from *** Keywords ***
on down and put it in a new file called “LoginKeywords.txt”
We’re getting to a point where this is well beyond a simple test, so while you’re at it rename “SimpleTest.txt” to “LoginTest.txt” and add one more line to the Settings section of LoginTest.txt, like so:
*** Settings ***
Library Selenium Library
Resource LoginKeywords.txt
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Login Test Setup
Test Teardown Login Test Teardown
Force Tags FunctionalTest
Default Tags ValidTest
Directories as Suites
I’m going to add more test files, so to organize them I’m going to create a directory “LoginTest” and put “LoginTest.txt” and “LoginKeywords.txt” in it. Now if I execute pybot from the directory above the “LoginTest” directory as:
pybot LoginTest
It will execute all of the test files in LoginTest.
Init Files
As stated above, both directories and files are treated as test suites by Robot Framework. You can create a Settings section for a directory by adding the file “__init__.txt” and putting a Settings table in it. Since I’m about to add more tests and they’re all going to need the same setup and teardowns, I’ll move a few more settings from “LoginTest.txt” to “__init__.txt” giving us:
*** Settings ***
Library Selenium Library
Resource LoginKeywords.txt
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Login Test Setup
Test Teardown Login Test Teardown
And now the Settings section of “LoginTest.txt” is:
*** Settings ***
Library Selenium Library
Resource LoginKeywords.txt
Force Tags FunctionalTest
Default Tags ValidTest
Data Driven Tests
Data driven tests are designed to repeatedly run the same test with changes in the data is being used for inputs. This allows the person writing the tests to create a large number of tests scenarios with minimal effort.
Robot Framework has a special setting called Test Template
that lets you create files that are dedicated to Data Driven test suites:
*** Settings ***
Library Selenium Library
Resource LoginKeywords.txt
Test Template Login Should Fail When
Force Tags DataDriven ValidTest
*** Testcases *** Username Password
The Password is Wrong AUser TestFail
The Password is Empty AUser ${empty}
Both Fields are Empty ${empty} ${empty}
*** Keywords ***
Login Should Fail When [Arguments] ${username} ${password}
Enter Username ${username}
Enter Password ${password}
Click the Login Button
Login Is Unsuccessful
Using Login Should Fail When
as the template means that each of the three testcases use it as their only keyword, and apply the arguments given to it. You can see on the first line of the Testcases table there are two columns set “Username” and “Password”. Robot Framework ignores anything in the first line of the table after the name, the column names are strictly for the reader’s benefit. The other new element is the ${empty}
variable, this indicates an empty string.
For this suite you can see I’ve added one more new keyword to “LoginKeywords.txt” called “Login Is Unsuccessful.” It is the negation of “Login Is Successful”:
Login is Unsuccessful
Run Keyword And Expect Error * Login Is Successful
Run Keyword and Expect Error
does exactly what it says. The second argument is the error expected, *
will allow for any error. The third argument is the keyword to run, then the follow arguments get passed to the keyword in the third argument. This is a useful way to define negative test cases.
Variables
When your tests get even moderately complicated there’s going to be data that needs to change from one run to the next or depending on the context you’re running it in. Use a variable table to minimize the pain of dealing with this. For my tests I have two variables that I’m concerned with, the URL under test and the browser used, pulling these out as variables in the keyword file gives me:
*** Variables ***
${login address}= http://zdiles.chaosnet.org/
${browser}= ie
*** Keywords ***
Enter Username [Arguments] ${username}
Input Text uname ${username}
Enter Password [Arguments] ${password}
Input Text pwd ${password}
Click the Login Button
Click Button login
Login Is Successful
Page Should Contain Welcome
Login is Unsuccessful
Run Keyword And Expect Error * Login Is Successful
Login Test Setup
Open Browser ${login address} ${browser}
Maximize Browser Window
Notice that when they’re assigned the variable names have an =
appended to them, this is optional, but it makes the code a little more clear.
Final Summary
I’ve found Robot Framework to be powerful tool for designing and executing tests. By adding functional tests to your project you can improve its overall quality. Enjoy.
Coming Up
Next week I’ll start with Smalltalk, using the Pharo environment which is redesigned from Squeak to be a powerful professional development tool.
Very good and well written article! I liked a lot how you started from basic concepts and were able to introduce also many of the more advanced and powerful features so clearly.
Two comments: 1) Underscore in the name of the init files have apparently been turned into bold formatting. 2) Using
Run Keyword And Expect Error
in negative cases, especially when you accept all errors, is a bit risky. I typically create explicit negative keyword instead.Hi Pekka,
Thanks a lot for the feedback, this is the training I’ve been using for other members of my SQE team and it’s been working well.
Regarding your comments:
I use similar approach in trainings I organize too but never had energy to write it down this well. You should try to get this published somewhere. And definitely send a note about this into robotframework-users list.
One thing I nowadays do differently in training is starting by writing the test in natural language. I noticed that when the first example we did used only low level keywords, people kept using that style also later. Now they only see tests done well and hopefully don’t learn the bad practices so easily.