Cucumber JVM 4 Parallel execution using JUnit

Introduction

This article deals with running Cucumber JVM in parallel using JUnit4 and Maven. The Maven Failsafe plugin is used for this purpose. In JUnit4 the feature files are run in parallel rather than scenarios, which means all the scenarios in a feature file will be executed by the same thread. Refer to Cucumber-JVM 4 announcement for more details.

To get a better understanding of parallel running using Maven Failsafe plugin refer to official documentation. For details about the thread configuration settings refer to this link. Though it mentions Surefire plugin, the settings are similar for Failsafe.

The runners generate the json and junit reports. Cluecumber aggregated html reports are also generated.

To run scenarios in parallel using an out of the box solution without JUnit or TestNG refer to this article. To run Cucumber in parallel using TestNG refer to this article.

Source Code

The source code is located here. The Cucumber version is 4.2.6, JUnit version is 4.12, Maven Failsafe plugin version is 3.0.0-M3.

The Default runner will execute all six feature files inside the features folder. This will be used as an example of a single runner project. The FirstRunner runs the three features in the first folder. Similarly SecondRunner runs three features in the second folder. These two runners will be used as an example of a project with multiple runners.

The steps are pretty basic and print out the executing thread id along with scenario description text and step number. There is a BeforeStep hook which introduces a two second sleep.

The scenarios1.feature and scenarios4.feature files are the same and contain one scenario and one senariooutline. The scenarios2.feature and scenarios5.feature are similar and contain one scenario each. The scenarios3.feature and scenarios6.feature are similar and contain one scenariooutline each.

Cluecumber report is an aggregated representation of the executed scenarios in html format. It uses the json report and is created in the folder specified by the generatedHtmlReportDirectory parameter. Refer to the plugin section of POM for configuration details.

The sample results of the executions using single and multiple runners with different configurations are stored in the reports folder. The duration in these reports is the time between the @AfterClass method and @BeforeClass method of the runner. In case of multiple runners, the duration is the difference between the last @AfterClass and first @BeforeClass of any of the runners.

The POM needs to be executed using the Maven command “mvn clean install” or a similar one which executes the failsafe plugin as well as the cluecumber plugin in the post-integration-test phase.

Failsafe Plugin Configuration

The settings for Maven Failsafe plugin needs to be added to the build plugins section of the pom. Refer to the POM for complete configuration.

The “PARALLEL EXECUTION SETTINGS” section will be detailed in the remaining article. The POM contains all the various settings and comments to enable or disable them.

The inclusion pattern would be specific to the project, depending on the name and number of runner classes. For automatic inclusion and exclusion of test classes refer here. The inclusion pattern for a single runner for this article is shown below. For multiple runner the inclusion pattern will be something like “**/*Runner.java”.

<configuration>
    <includes>
        <include>**/Default.java</include>
    </includes>               
        PARALLEL EXECUTION SETTINGS
 </configuration>

Parallel Execution

This uses the parallel configuration parameter of the Failsafe plugin. This can be set to ‘methods’ or ‘classesAndMethods’. The number of threads is set by using the threadCount parameter. As an example, below command creates 2 threads per core. This can be confusing as the actual number of threads will be larger than 2 depending on the number of cores.

<parallel>methods</parallel>
<threadCount>2</threadCount>

When there is a single runner it is ideal to use the “methods” setting. If multiple runners are present then one can use either “classesAndMethods” or “methods”. The “classes” setting does not give any benefit for a single runner, as it uses one thread it is similar to a sequential execution. Refer to the sample times for single runner and multiple runners for the above settings.

To limit the thread count to the mentioned setting across all cores set the perCoreThreadCount parameter to false. This is set to true by default. Refer to the sample times for single runner and multiple runners for thread count across all cores.

<parallel>methods</parallel>
<threadCount>4</threadCount>
<perCoreThreadCount>false</perCoreThreadCount>

One can also set the useUnlimitedThreads parameter to true instead of specifying threadCount and perCoreThreadCount options.

<parallel>methods</parallel>
<useUnlimitedThreads>true</useUnlimitedThreads>

Forked Execution

The forkCount parameter is the maximum count of JVM processes that Failsafe plugin will create. Each test class will be executed in its own forked process which will be reused depending on the value of reuseForks parameter.

<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<reportsDirectory>
    ${project.build.directory}/failsafe-reports_${surefire.forkNumber}
</reportsDirectory>

In case there is a single runner in the project then Failsafe plugin creates a new JVM process. There will be no benefit in terms of reduced execution times.

When there are multiple runners and the forkCount is more than 1, multiple JVM are spawned for the runners. These runners are then executed in parallel though by a single thread inside one JVM, similar to “classes” setting.

If the number of runners are more than the forkCount, then the forks can be reused depending on the reuseForks setting. Reusing forks is a more efficient technique unless greater separation is required.

To easily figure out more details about which scenarios are being run in which fork, one can set the reportsDirectory setting by including the surefire.forkNumber in the folder structure. This generates a xml report which mentions the scenarios run in that specific JVM process fork. This is entirely optional. If this is not set then the xml files are anyways generated in the target/failsafe-reports folder.

Combined Execution

To reduce the execution times in case of JVM process forking one can also configure the parallel setting. The “methods” or “classesAndMethods” setting can be used. This ensures that the feature files are executed by multiple threads in each forkRefer to the sample times for single runner and multiple runners for the combination settings.

<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<reportsDirectory>
    ${project.build.directory}/failsafe-reports_${surefire.forkNumber}
</reportsDirectory>

<parallel>methods</parallel>
<threadCount>2</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>

Feature File Parallel Execution

As mentioned in the introduction section, JUnit runs feature files in parallel rather than scenarios. This is true for running with the parallel setting or forked execution.

In the image, the line sequence have been modified to show the scenarios in a feature file to be clubbed together. The thread id 12 runs the scenario and sceariooutline present in the scenarios1.feature file in the first folder. Similarly the same logic can be extended to the other threads and feature files.

 

4 comments

  1. I’m using the above project but when running, they all seem to run in series. Console below. The pom is set to create multiple threads but it only seems like it’s running on a single thread. Any ideas? These should all be different threads right. I added the timeline plugin and it also shows just one thread

    Thread 1 -> Scenario 1 – FIRST STEP
    Thread 1 -> Scenario 1 – SECOND STEP
    Thread 1 -> Scenario Outline 1 – FIRST STEP
    Thread 1 -> Scenario Outline 1 – FIRST STEP
    Thread 1 -> Scenario Outline 1 – SECOND STEP
    Thread 1 -> Scenario Outline 1 – SECOND STEP
    Thread 1 -> Scenario 4 – FIRST STEP
    Thread 1 -> Scenario 4 – SECOND STEP
    Thread 1 -> Scenario Outline 4 – FIRST STEP
    Thread 1 -> Scenario Outline 4 – FIRST STEP
    Thread 1 -> Scenario Outline 4 – SECOND STEP
    Thread 1 -> Scenario Outline 4 – SECOND STEP
    Thread 1 -> Scenario 2 – FIRST STEP
    Thread 1 -> Scenario 2 – SECOND STEP
    Thread 1 -> Scenario 5 – FIRST STEP
    Thread 1 -> Scenario 5 – SECOND STEP
    Thread 1 -> Scenario Outline 3 – FIRST STEP
    Thread 1 -> Scenario Outline 3 – FIRST STEP
    Thread 1 -> Scenario Outline 3 – SECOND STEP
    Thread 1 -> Scenario Outline 3 – SECOND STEP
    Thread 1 -> Scenario Outline 6 – FIRST STEP
    Thread 1 -> Scenario Outline 6 – FIRST STEP
    Thread 1 -> Scenario Outline 6 – SECOND STEP
    Thread 1 -> Scenario Outline 6 – SECOND STEP

    12 Scenarios (12 passed)
    24 Steps (24 passed)
    0m48.173s

    DURATION – 48092
    Thread Id | Scenario Num | Step Count

    Process finished with exit code 0

  2. My project has a single runner and multiple step defination files , using Picocontainer as DI , Added POM configuration as directed in github, but for me using Mvn lifecycyle command, Feature files are running one by one, not parallely.

    any idea ?

Leave a Reply

Your email address will not be published. Required fields are marked *