Automated Robot Framework tests for embedded Linux devices

When working with embedded systems, the software is only a part of the whole product or device. Usually there is also customized hardware involved, and some parts of the software are very low-level and hardware dependent. With embedded Linux most parts of the software can quite easily be covered by unit testing on a Linux host. Yet, there are always some parts which need to be tested with the actual physical hardware. Using the right tools this testing can also be automated.

The test tools and process

Robot Framework is a generic test automation framework for automated acceptance tests. It has lots of libraries and can be used to test all kinds of systems. It can also be used to run tests against embedded devices. The following figure shows a simple test flow using Jenkins to orchestrate the tasks.

diagram

The code base is managed in a version control system such as git. The Jenkins automatically detects new commits to the repository and checks out the latest project. It then builds the project and archives the software binaries. Build tasks are followed by several testing tasks which can include both unit and acceptance tests.

Robot testing in practice

First step in robot testing is to update the device with the latest firmware. Jenkins can archive build results so the latest binaries are easily available from the Jenkins build tasks and the firmware update can be the first robot test case in the test suite. Robot framework provides libraries for SCP and SSH which can be used to implement the firmware update.

The easiest way to use robot framework with embedded Linux device is to build test scripts and applications into the firmware. This way the robot test case can

  1. Connect to the device using SSH
  2. Invoke test script / binary
  3. Parse and check the result

Using this scheme it is quite simple to test even the low-level components such as device drivers because the actual interaction with the driver can be implemented in C and wrapped to a test binary.

Example

To get started, the whole automated test system is not necessarily needed. Easiest way is just to install robot framework to the development host machine. The installation instructions can be found here. If python and pip are already installed, the robot installation is really easy, basically couple of commands:

sudo pip install robot
sudo pip install robotframework-sshlibrary
sudo pip install robotframework-scplibrary

Note: When installing on Ubuntu 14.04 LTS the SSHLibrary failed on missing headers. These errors were fixed by running commands:

sudo apt-get install python-dev
sudo pip install --upgrade --force-reinstall paramiko

Below is a simple Robot test suite with one example test case. The test opens a SSH connection to device and executes a single command. The test then asserts that the command returned correct output. Save the test suite to, for instance, ssh.robot file.

*** Settings ***
Library                SSHLibrary
Suite Setup            Open Connection And Log In
Suite Teardown         Close All Connections

*** Keywords ***
Open Connection And Log In
   Open Connection    ${HOST}
   Login    ${USER}    ${PASSWORD}

*** Test Cases ***
Check ps Utility
    ${output}=    Execute Command    ps
    Should Contain    ${output}    ps

Invoke the test from command line using the following command. It is possible to set user defined variables such as HOST, USER and PASSWORD from command line so the same test can easily be executed with different target devices.

robot --variable HOST:192.168.0.10 --variable USER:user --variable PASSWORD:password 
--result result.xml ssh.robot

The test output should be something like:

==============================================================================
Ssh                                                                           
==============================================================================
Check ps Utility                                                      | PASS |
------------------------------------------------------------------------------
Ssh                                                                   | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /home/devel/result.xml
Log:     /home/devel/log.html
Report:  /home/devel/report.html

This example test can be used as a starting point for an actual test. Just substitute the ‘ps’ with test binary/script and verify the output. The tests should be saved to the source code repository so they are easily accessible for other developers and Jenkins.

Integrating Robot tests to Jenkins is also pretty straight forward:

  1. Install Python, Robot and Robot libraries to Jenkins server
  2. Install Robot Jenkins plug-in (for xml results)
  3. Create Jenkins task and use Execute Shell to run Robot test command
  4. Configure the result file(s) to Robot Plug-in

Robot framework can help automate manual test tasks, and when integrated to Jenkins, all the tests are automatically executed whenever something is committed to the source code repository.

One thought on “Automated Robot Framework tests for embedded Linux devices

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s