Virtuous Programmer Adventures of an Autodidact



Posted by Frank Berthold

What's it for?

Markdown is a lightweight markup language which allows its user to create documents in a variety of formats. Unlike many other markup languages like HTML and LaTeX, readability of the sourcecode was one of the top priorities in Markdown's design. Its creator, John Gruber, based its syntax on how we format plain text emails to make it as readable as possible. For example: to emphasize text, you use "*important*" to get "important".

For people who are accustomed to working in plain text, this is an excellent tool to create documents others will want to read while still using text editors and other tools that they are accustomed to.

Where to get it

Markdown is only a syntax specification for which there are several implementations. For most purposes I've found Pandoc to be excellent. Pandoc reads a superset of Markdown and writes out in a multitude of formats including HTML, PDF(with pdflatex) and Docbook.

How to Write with Markdown

Here I'll give 4 commonly used formatting as an example. These should be be sufficient for many simple documents:

  • Section Headers
  • Links
  • Lists
  • Code

Section Headers

There are two ways to denote Section Headers which can be mixed freely. The two highest level headers of can be marked by underlining as in:


  1. First-level heading
  2. ===================
  4. Second-level heading
  5. --------------------

Renders as:

First-level heading

Second-level heading

All levels can be marked using # marks preceding the text:


  1. # First-level heading
  3. ## Second-level heading
  5. ### Third-level heading

Renders as:

First-level heading

Second-level heading

Third-level heading

The maximum number of levels depends on the format that it's converted to. As an example, HTML allows for 6 levels.


Links are written in the format [<link name>](<url> "<title>") where the title is optional. For example, [Python]( "Python's web site") renders as: Python


Lists may either be bulleted or numbered:

Bulleted List Markdown:

  1.    * First
  2.   * Second
  3.   * Third

Renders as:

  • First
  • Second
  • Third

Numbered List Markdown:

  1.    1. First
  2.   2. Second
  3.   3. Third

Renders as:

  1. First
  2. Second
  3. Third


Code can be entered inline by surrounding it with back quotes, `like this` or by indenting by 4 or more spaces:


  1.     if inCodeBlock:
  2.       return True

Renders as:

if inCodeBlock:
  return True

Generating Documents from Markdown Using Pandoc

Once you've downloaded Pandoc from its site and installed it, you can generate a variety of documents from your Markdown text. To see the complete list type "pandoc --help" at the command line.

To generate HTML documents at the command line use "pandoc <markdown document>.mkd -o <html document>.html". Pandoc determines what the format of the input and output types are from their extensions.

Final Words

Markdown is a readable markup language that covers a significant proportion what you need to generate formatted documents. Unfortunately it does lack some features. There is no notation for tables, footnotes and several other fairly common features in documents available in the original Markdown standard. To compensate for this, a variety of supersets have been create. Pandoc's extensions, Markdown Extra and MultiMarkdown are all commonly used.

If you want to play with Markdown a bit to get your feet wet, this blog is also written using Markdown and the comments section supports it. Feel free to experiment.


Filed under: Markup 1 Comment

Robot Framework

Posted by Frank Berthold

Test Automation with Robot Framework

The Plan

Robot Framework is a functional test framework that makes it possible for its users to:

  • Write tests in easily understood language
  • Organize what tests are run for specific purposes
  • Generate both high level and detailed test results

Notice that what it does not do is any specific testing activity. Instead it acts as a front end for libraries like Selenium and AutoIT which perform the actions themselves. For the purpose of this tutorial I'll be focusing on web automation using Selenium as the back-end.

What to Expect

When this article is complete:

  • You will have:

    • An installation of Robot Framework with Selenium
    • Several tests for a basic login page
  • You will know:

    • How to use Robot Framework(RF):
      • To drive Selenium
      • To write functional tests
      • To write data-driven tests
      • To select or exclude tests by their tags
    • How to read an RF results file
    • How to read an RF log file
    • To find identifiers on an HTML page

Installing Robot Framework and Selenium

Before installing Robot Framework, you need to install Python 2.5 or later. Robot Framework hasn't been updated to work with Python 3, so currently 2.7 is your best bet. If you don't already have Python installed you can download it from here.

Next you'll need to download Robot Framework and Selenium. For each of the installs select the default values.

Finally, add your python installation and robot installation to your path. If you've installed Python 2.7 it will be: "c:\Python27\;c:\Python27\Scripts\;".

Files Used in this Project

Executing Robot Framework Tests

In Robot Framework each file contains one or more tests. That file is treated as a test suite. Every directory that contains a test suite file or directory is also a test suite. When RF is executed on a directory it will recurse through all files and directories of the correct kind except those that start with an underscore character.

To run the tests in "SimpleTest.txt" navigate to the directory that contains it in the command line and type:

  1. pybot SimpleTest.txt

There are arguments that will give you greater control over what tests are executed, but I'll get to those later in this article.

Notice that while the test is running there are two windows. One is the window in which the test is being executed, the other is the controller window that Selenium is executing the JavaScript that controls the other window from.

When the test is finished executing four files will have been generated: report.html, log.html, output.xml and selenium_server_log.txt. Of these the only one's you'll generally need to be aware of are report.html and log.html.

Reading the Results

Your first stop after a test run is report.html. When you open it, if you're lucky, you'll see this happy image:

Passed Report

The background is green when all tests have passed, red if any have failed. If you need more detail than that the "Test Statistics" section gives how many tests have passed and failed sorted by criticality, tag and suite. The "Test Details" section gives how long the test took to run and, if it failed, what the fail message was, as can be seen here:

Failed Report

When a test fails, unless the message makes it immediately obvious, you'll generally next need to read the log. If you click the failed test name, it will then open up the test logs:

Failed Log

The log gives a detailed view of the execution of each of the tests. While the information for all of the tests are available, the tests and steps that are failed are marked in red and already expanded.

The Code

Syntax and Formats

Robot Framework files are sorted into tables. The format of those tables is flexible, you may create them in tab-separated, html, ReStructured text, or just plain text. I've tried writing tests in each of these, HTML makes the prettiest files, and is also a pain to work with. In the end, plain text has been the best combination of nice to read for my customer and easy for me to maintain.

There are up to four tables in a test file, the "Testcases" table is mandatory, it contains all the tests you're going to run, the optional tables are:

  • Settings: Data needed across tests, like what libraries to use and how to setup and teardown tests.
  • Variables: Variable names and values
  • Keywords: These are function definitions, RF calls them keywords, there's no difference.

The Test

Given a Login Page verify that when the correct username and password are supplied, it is possible to login.

A First Attempt


  1. *** Settings ***
  2. Library       Selenium Library

Each Table is marked out by *** <table name> *** and continues until the next table name is reached. The text version of the RF syntax is whitespace delimited. Each item under a table is flush to the left of the screen. If a keyword takes arguments then the follow it on the same line, separated by either a tab or two or more spaces. I generally space them out more than two for readability.

The Library setting imports the named library into the test suite. The Selenium Library allows its users to control a web browser automatically and ideally suited to the task at hand.

Identifying Elements on a Web Page

When automating any system, you have to have a way for your test software to find the elements that you're interacting with. Selenium does this through one of several methods, depending on how automation friendly the software you're testing is. If it's very friendly then every element on the page has either a unique name or id attribute. If it's more hostile then you can still identify any element through its xpath. I recommend using xpath only as a last resort. It's a powerful tool, but makes your code hard to understand.

I wrote the login page, so it's very test friendly. For simpler pages like this, I general just look at the sourcecode of the page to find out how to identify the elements, if you look at the Login Page, you'll find that the identifiers are:

Login Page

Test Cases

  1. *** Testcases ***
  2. Login Should Succeed When the Correct Username and Password are Entered
  3.   Start Selenium Server
  4.   Open Browser  ie
  5.   Maximize Browser Window
  6.   Input Text    uname  BUser
  7.   Input Text    pwd    TestPass
  8.   Click Button  login
  9.   Page Should Contain  Welcome
  10.   Close Browser
  11.   Stop Selenium Server

I've named this file "SimpleTest.txt". Here's a line By line look at it, I've informally labeled the below steps with their purpose: Setup, Action, Verification, Teardown

  1. A Testcases table contains one or more tests.
  2. The first line will be the name of the test as used in the reports and logs.
  3. Setup Task: To use Selenium a server has to be started that will receive commands and control a browser. Notice that keyword names can have single spaces in them.
  4. Setup Task: A browser is opened to the page given, by default Firefox is used. If specified as it is here it can also use Internet Explorer or Chrome.
  5. Setup Task: By default the browser window isn't maximized. This fixes that.
  6. Test Action: The Input Text keyword enters text into a field, it takes two arguments, the identifier for the element and the text to enter. This fills the username field.
  7. Test Action: The next Input Text call fills the password field.
  8. Test Action: One of the nice things about robot framework is that its keyword names tend to be pretty obvious. The argument passed to it identifies the Login Button.
  9. Test Verification: Page Should Contain searches through all the text on the page and passes if it contains the given text and fails otherwise.
  10. Teardown Task: Close Browser, this will shutdown the browser opened in line 4, if not run then it just leaves all the windows opened between tests, each test will ad more leading to clutter and confusion.
  11. Teardown Task: Shuts down the Selenium Server, if this isn't done then it can occasionally lead to trouble when you try to start the Selenium Server in a future test.

Suite Settings

The test as it's written will faithfully perform the tests given, but has a couple of problems. When a test fails Robot Framework stops running the test to save time and avoid performing activities on a system that's in a bad state. This means that if the verification step fails, then our teardown tasks will never be executed. To deal this problem, and make neater more readable test cases there are settings at both the test and suite level to setup and teardown tests.


  1. *** Settings ***
  2. Library         Selenium Library
  3. Suite Setup     Start Selenium Server
  4. Suite Teardown  Stop Selenium Server
  5. Test Setup      Open Browser  ie
  6. Test Teardown   Close All Browsers

Setup and Teardown can be set at both the test and the suite level. Each fires off at different points during a test run:

  • Suite Setup: Fires once before any tests during the suite are run.
  • Test Setup: Fires before each test is executed.
  • Test Teardown: Fires at the end of each test execution, even when the test fails.
  • Suite Teardown: Fires when all tests in the suite have completed, even when one of them has failed.

This slims our testcase down considerably:

  1. *** Testcases ***
  2. Login Should Succeed When the Correct Username and Password are Entered
  3.   Maximize Browser Window
  4.   Input Text    uname  BUser
  5.   Input Text    pwd    TestPass
  6.   Click Button  login
  7.   Page Should Contain  Welcome

But we still have the Maximize Browser Window keyword, which really doesn't improve the reader's understanding of the intent of the test, and is likely to be used in the execution of any future testcase. Unfortunately the setup/teardown keywords can only take one keyword.

The Keywords Table

You can solve this by adding a keyword that calls both of these:

  1. *** Keywords ***
  2. Setup Test
  3.   Open Browser  ie
  4.   Maximize Browser Window

And change the Test Setup setting to use it:

  1. *** Settings ***
  2. Library         Selenium Library
  3. Suite Setup     Start Selenium Server
  4. Suite Teardown  Stop Selenium Server
  5. Test Setup      Setup Test
  6. Test Teardown   Close Browser

Making Your Tests More Readable

  1. *** Testcases ***
  2. Login Should Succeed When the Correct Username and Password are Entered
  3.   Input Text    uname  BUser
  4.   Input Text    pwd    TestPass
  5.   Click Button  login
  6.   Page Should Contain  Welcome

Now that I've gotten rid of the steps that obscure the intent of my test, it's time to make the purpose of the tests themselves more clear by creating keywords with more meaningful names than those given, so we'll add a couple more keywords to our Keywords table:

  1. Enter Username  [Arguments]  ${username}
  2.   Input Text  uname  ${username}
  4. Enter Password  [Arguments]  ${password}
  5.   Input Text  pwd  ${password}
  7. Click the Login Button
  8.   Click Button  login
  10. Login Is Successful
  11.   Page Should Contain  Welcome

Which introduces the use of settings in individual keywords. The syntax used for keyword and test settings(to be seen soon) is the same. A special element surrounded in square brackets, you can find a full listing of these in Robot Framework's User Guide. Here I'm able to specify arguments that will be passed into the keywords from the test case that calls it. The syntax for variables is ${<variable name>} and can contain spaces.

Now my testcase is considerably more readable:

  1. Login Should Succeed When the Correct Username and Password are Entered
  2.   Enter Username  AUser
  3.   Enter Password  TestPass
  4.   Click the Login Button
  5.   Login Is Successful

The Power of Tags

While it's useful to be able to run every test in your test suite at once, often you only need to run a small subset of them to perform a smoke test, or to test only a specific feature. To this end you can tag tests and suites to indicate their function.

The code below is the same as what I've constructed with one more test, which should fail every time, and a couple of tags (Keywords left out because they don't change):

  1. *** Settings ***
  2. Library         Selenium Library
  3. Resource        LoginKeywords.txt
  4. Suite Setup     Start Selenium Server
  5. Suite Teardown  Stop Selenium Server
  6. Test Setup      Login Test Setup
  7. Test Teardown   Login Test Teardown
  8. Force Tags      FunctionalTest
  9. Default Tags    ValidTest
  11. *** Testcases ***
  12. Login Should Succeed When No Username Is Given
  13.   [Tags]  InvalidTest
  14.   Enter Username  ${empty}
  15.   Enter Password  TestPass
  16.   Click the Login Button
  17.   Login Is Successful
  19. Login Should Succeed When the Correct Username and Password are Entered
  20.   Enter Username  AUser
  21.   Enter Password  TestPass
  22.   Click the Login Button
  23.   Login Is Successful

One Test level setting has been added, tagging the first test with InvalidTest. At the suite level there are two more. Force Tags ensures that every test in the suite will be tagged as a FunctionalTest. Default Tags set's a test's tags to ValidTest unless the test has a tag setting of its own like the first test.

There are two benefits from adding descriptive tags to your tests. When you run your tests your test results will be sorted by them. This will frequently help determine where an individual problem is. Additionally you can determine which tests get executed. For example when you use:

  1. pybot -i InvalidTest SimpleTest.txt

Will only include the tests with that tag, while:

  1. pybot -e InvalidTest SimpleTest.txt

Will exclude any test with that tag.

Keyword Files

Keywords like Enter Username will be used in nearly every test you write for this page. It's often helpful to break up your test suites into multiple files, or even multiple directories. To avoid copy pasting the same set of keywords into every file, you can break them out into a separate resource file. The easiest way in our current case is to just cut everything from *** Keywords *** on down and put it in a new file called "LoginKeywords.txt"

We're getting to a point where this is well beyond a simple test, so while you're at it rename "SimpleTest.txt" to "LoginTest.txt" and add one more line to the Settings section of LoginTest.txt, like so:

  1. *** Settings ***
  2. Library         Selenium Library
  3. Resource        LoginKeywords.txt
  4. Suite Setup     Start Selenium Server
  5. Suite Teardown  Stop Selenium Server
  6. Test Setup      Login Test Setup
  7. Test Teardown   Login Test Teardown
  8. Force Tags      FunctionalTest
  9. Default Tags    ValidTest

Directories as Suites

I'm going to add more test files, so to organize them I'm going to create a directory "LoginTest" and put "LoginTest.txt" and "LoginKeywords.txt" in it. Now if I execute pybot from the directory above the "LoginTest" directory as:

  1. pybot LoginTest

It will execute all of the test files in LoginTest.

Init Files

As stated above, both directories and files are treated as test suites by Robot Framework. You can create a Settings section for a directory by adding the file "__init__.txt" and putting a Settings table in it. Since I'm about to add more tests and they're all going to need the same setup and teardowns, I'll move a few more settings from "LoginTest.txt" to "__init__.txt" giving us:

  1. *** Settings ***
  2. Library         Selenium Library
  3. Resource        LoginKeywords.txt
  4. Suite Setup     Start Selenium Server
  5. Suite Teardown  Stop Selenium Server
  6. Test Setup      Login Test Setup
  7. Test Teardown   Login Test Teardown

And now the Settings section of "LoginTest.txt" is:

  1. *** Settings ***
  2. Library         Selenium Library
  3. Resource        LoginKeywords.txt
  4. Force Tags      FunctionalTest
  5. Default Tags    ValidTest

Data Driven Tests

Data driven tests are designed to repeatedly run the same test with changes in the data is being used for inputs. This allows the person writing the tests to create a large number of tests scenarios with minimal effort.

Robot Framework has a special setting called Test Template that lets you create files that are dedicated to Data Driven test suites:

  1. *** Settings ***
  2. Library         Selenium Library
  3. Resource        LoginKeywords.txt
  4. Test Template   Login Should Fail When
  5. Force Tags      DataDriven  ValidTest
  7. *** Testcases ***       Username  Password
  8. The Password is Wrong   AUser     TestFail
  9. The Password is Empty   AUser     ${empty}
  10. Both Fields are Empty   ${empty}  ${empty}
  12. *** Keywords ***
  13. Login Should Fail When   [Arguments]  ${username}  ${password}
  14.   Enter Username  ${username}
  15.   Enter Password  ${password}
  16.   Click the Login Button
  17.   Login Is Unsuccessful

Using Login Should Fail When as the template means that each of the three testcases use it as their only keyword, and apply the arguments given to it. You can see on the first line of the Testcases table there are two columns set "Username" and "Password". Robot Framework ignores anything in the first line of the table after the name, the column names are strictly for the reader's benefit. The other new element is the ${empty} variable, this indicates an empty string.

For this suite you can see I've added one more new keyword to "LoginKeywords.txt" called "Login Is Unsuccessful." It is the negation of "Login Is Successful":

  1. Login is Unsuccessful
  2.   Run Keyword And Expect Error  *  Login Is Successful

Run Keyword and Expect Error does exactly what it says. The second argument is the error expected, * will allow for any error. The third argument is the keyword to run, then the follow arguments get passed to the keyword in the third argument. This is a useful way to define negative test cases.


When your tests get even moderately complicated there's going to be data that needs to change from one run to the next or depending on the context you're running it in. Use a variable table to minimize the pain of dealing with this. For my tests I have two variables that I'm concerned with, the URL under test and the browser used, pulling these out as variables in the keyword file gives me:

  1. *** Variables ***
  2. ${login address}=
  3. ${browser}=  ie
  6. *** Keywords ***
  7. Enter Username  [Arguments]  ${username}
  8.   Input Text  uname  ${username}
  10. Enter Password  [Arguments]  ${password}
  11.   Input Text  pwd  ${password}
  13. Click the Login Button
  14.   Click Button  login
  16. Login Is Successful
  17.   Page Should Contain  Welcome
  19. Login is Unsuccessful
  20.   Run Keyword And Expect Error  *  Login Is Successful
  22. Login Test Setup
  23.   Open Browser  ${login address}  ${browser}
  24.   Maximize Browser Window

Notice that when they're assigned the variable names have an = appended to them, this is optional, but it makes the code a little more clear.

Final Summary

I've found Robot Framework to be powerful tool for designing and executing tests. By adding functional tests to your project you can improve its overall quality. Enjoy.

Coming Up

Next week I'll start with Smalltalk, using the Pharo environment which is redesigned from Squeak to be a powerful professional development tool.


Filed under: Testing 3 Comments