Acceptance Test Automation with Cucumber / SpecFlow in Visual Studio

Acceptance Criteria, Test Automation and Gherkin

What do Acceptance Criteria, Test Automation and a Cucumber have in common? Well for the uninitiated in may seem that these 3 things have nothing in common however if you are an “Agilist” or Test Automation specialist than you are probably very familiar with the similarities. Let’s start with the Cucumber specifically the Gherkin. The Gherkin as it happens is the most common type of cucumber used to make pickels. Gherkin is also the language used to Automated Acceptance Tests in a tool called Cucumber. Acceptance Criteria are the conditions for success for a new feature or part of a feature being added to solution. If the Acceptance Criteria is written in Gherkin format most of the heavy lifting of Test Automation has already been done. To automate acceptance tests in Visual Studio we use the Visual Studio version of Cucumber: SpecFlow. To get started using Gherkin Acceptance Criteria to Automate Acceptance Testing the Visual Studio IDE needs to be configured to use the Cucumber / SpecFlow extension.

Create a Virtual Machine in AWS install SpecFlow Extension in Visual Studio Community Edition and create an Automated Acceptance Test for a basic Stack object. Detailed Step By Step Instructions below video

Install SpecFlow Extension for Visual Studio

  1. In Visual Studio 2019 Community Edition
  2. In the Launch dialog choose Continue without code
    VS 2019 Continue without code
    VS 2019 Continue without code
  3. In the Extensions Menu Select Manage Extensions
    VS 2019 Manage Extensions
    VS 2019 Manage Extensions
  4. Select Online from the Navigation menu
  5. In the Search box type “SpecFlow”
  6. Select SpecFlow for Visual Studio 2019 and click Download
    VS 2019 Install SpecFlow Extension
    VS 2019 Install SpecFlow Extension
  7. Click Close to close the Manage Extensions dialog
    VS 2019 Close Visual Studio to complete SpecFlow Extension Installation
    VS 2019 Close Visual Studio to complete SpecFlow Extension Installation
  8. Exit Visual Studio
  9. Click Modify in the VSIX Installer dialog
    VSIX Installer
    VSIX Installer – Modify
  10. When Installation completes click Close in the VSIX Installer dialog
    VSIX Installer
    VSIX Installer – Installation Complete
  11. Restart Visual Studio

Note: Visual Studio will not Install the Cucumber Extension until all Visual Studio windows are closed.

Create a Unit Test Project to Map Acceptance Criteria

  1. In order to Add Gherkin Feature Descriptions we will need to add a Unit Test Project.
  2. In Visual Studio from the File menu select New Project
  3. In the New Project Dialog under Language select C# under Platform select Windows under Project Type select Test
  4. Select Unit Test Project (.NET Framework) and click Next
    Create new Test Project
    Create new Test Project
  5. Name the Project Stack App Acceptance Tests
  6. Name the Solution Stack App
  7. Click Create

    Configure new Project and Solution
    Configure new Project and Solution names
  8. In the Solution Explorer
  9. Right Click UnitTest1.cs and select Delete
  10. Right click the Project and select Add New Item
  11. Select the SpecFlow folder on the left
  12. Select SpecFlow Feature File
  13. Name the File StackAppFeatures.feature
  14. Click Add

    Create new Feature File
    Create new Stack Feature File
  15. In the Solution Explorer select StackAppFeatures.feature
  16. In the Properties windows remove SpecFlowSingleFileGenerator from the Custom Tool Property
    Remove Custom Tool Property value
    Remove SpecFlowSingleFileGenerator from the Custom Tool property

Note: The Custom tool error will disappear once the SpecFlowSingleFileGenerator text is removed from the Custom Tool Property of the StackAppFeatures.feature file.

Add SpecFlow NuGet Packages to Test Project

  1. Right click the Stack App Acceptance Tests Project
  2. Select Manage NuGet Packages… from the Menu
  3. In the NuGet Stack App Acceptance Tests window select Browse form the menu
  4. In the Search (Ctl+L) box type SpecFlow and press enter
  5. Select SpecFlow by TechTalk
  6. In the Detail window to the right click Install

    Install SpecFlow NuGet package
    Install SpecFlow NuGet package in Test Project
  7. In the License Acceptance dialog click I Accept

    License Acceptance - I Agree
    License Acceptance – I Agree to accept license terms
  8. Select SpecRun.SpecFlow
  9. In the Detail window to the right click Install
  10. In the License Acceptance dialog click I Accept
  11. In the Search (Ctl+L) box type SpecFlow.Tools and press enter
  12. Select SpecFlow.Tools.MsBuild.Generation
  13. In the Detail window to the right click Install

Step Definitions

In the Solution explorer select StackAppFeatures.feature and replace the Math feature (User Story) and Acceptance Criteria with text below:

Feature: StackAppFeatures
    As a User
    I need to Stack stuff
    So that I can move it around

@EmptyStack
Scenario: IsEmpty should be true
Given An Empty Stack
When I check IsEmpty
Then IsEmpty should be “true”

@NonEmptyStack
Scenario: IsEmpty should be false
Given A nonEmpty Stack
When I check IsEmpty
Then IsEmpty should be “false”

@PushTests
Scenario: Push Check IsEmpty
Given A Empty Stack
When I Push “Bugga Boo” onto the Stack
And I check IsEmpty
Then IsEmpty should be “false”

@PushPopTests
Scenario: Push Gets Popped
Given An Empty Stack
When I Push “Item 1” onto the Stack
And I Pop a value off the Stack
Then The result should be “Item 1”

@PushPeekTests
Scenario: Push Gets Peeked
Given An Empty Stack
When I Push “Item 1” onto the Stack
And I Peek at a value on the Stack
And I check IsEmpty
Then The result should be “Item 1”
And IsEmpty should be “false”

The purple text indicates that the Gherkin statement does not yet have associated step definitions. We’ll use the Code Generations tools in the SpecFlow extension generate the step definitions from the Gherkin Acceptance Criteria.

  1. Right click on Given an Empty Stack and select Generate Step Definitions
  2. In the Generate Step Definition Skeleton – SpecFlow dialog click Generate

    Generate Step Definitions
    Generate Step Definitions
  3. In the Select target step definition class file dialog accept the defaults and click Save

    Accept Default location for Step Definitions class
    Accept Default location for Step Definitions class file

Feature Implementation – Test First

Now we will replace the ScenarioContext.Current.Pending(); placeholder code in each of the Given…When…Then… functions in the StackAppFeaturesSteps.cs class file with calls to our code under test

  1. In the Solution Explorer select the StackAppFeaturesSteps.cs class file
  2. Add the following code to the StackAppFeaturesSteps class
    Stack stack;
    String _actual;
    Boolean _isEmpty;
  3. In the public void GivenAnEmptyStack() function replace the ScenarioContext.Current.Pending(); with the code below
    stack = new Stack();
  4. Replace the ScenarioContext.Current.Pending(); code in the GivenANonEmptyStack function with the code below
    stack = new Stack();
    stack.Push(“Hello, World!”);
  5. Replace the ScenarioContext.Current.Pending(); code in the WhenICheckIsEmpty function with the code below
    _isEmpty = stack.IsEmpty();
  6. Replace the ScenarioContext.Current.Pending(); code in the WhenIPushOntoTheStack function with the code below
    stack.Push(p0);
  7. Replace the ScenarioContext.Current.Pending(); code in the WhenIPopAValueOffTheStack function with the code below
    _actual = stack.Pop();
  8. Replace the ScenarioContext.Current.Pending(); code in the WhenIPeekAtAValueOnTheStack function with the code below
    _actual = stack.Peek();
  9. Replace the ScenarioContext.Current.Pending(); code in the ThenTheResultShouldBe function with the code below
    Assert.AreEqual(p0, _actual);
  10. Replace the ScenarioContext.Current.Pending(); code in the ThenIsEmptyShouldBe function with the code below
    Assert.AreEqual(Boolean.Parse(p0), _isEmpty);

Now that we have our tests we can use code generation tools to create the class under test. At this point we will also have 5 syntax / reference errors, we will resolve these in the next steps.

  1. In the solution explorer right click Solution ‘Stack App’ (1 of 1 project) select Add > New Project
    In the Solution Explorer you should now see the Stack Utility project
    Add new Class Library project
    Add new Class Library project
  2. In the Solution Explorer select the StackAppFeaturesSteps.cs class file
  3. In the StackAppFeaturesSteps.cs class file hover the mouse over the Stack type with the red squiggles
    In the Quick Actions menu (The lightbulb) select Generate type ‘Stack’ > Generate new type…

    Generate Stack Type
    Use Quick Actions to Generate Stack Type
  4. In the Generate Type dialog select Stack Utility from the Project dropdown, select the Create new file radio button and click OK
    Note:
    the red squiggles under Stack are now gone
    No more red Squiggles
    Stack exists – No more red Squiggles
  5. In the StackAppFeaturesSteps.cs class file hover the mouse over the call to the stack.Push method and select Generate method ‘Stack.Push’ from the Quick Actions menu
    Generate Push method
    Use Quick Actions to Generate Push method
  6. Repeat the same process for the IsEmpty, Push, Pop and Peek methods
  7. In the StackAppFeaturesSteps.cs class file hover the mouse over the Assert in Assert.AreEqual and select using Microsoft.VisualStudio.TestTools.UnitTesting; from the Quick Actions menu
    Note: You may now run the tests and see them fail
  8. From the Test menu select Run > All Tests
    Note:
    In the Test Explorer that of 6 Tests 3 failed. Only 5 were actual tests 1 was a SpecRun delay for the evaluation version. 3 Failed and the 2 Tests are ignored. The ignored tests call methods that were already tested and failed in this test run.
    Run the Tests and see them Fail
    Run the Tests and see them Fail

Now that class skeleton has been created, we can implement the 4 methods, IsEmpty, Push, Pop and Peek.

  1. In the Solution Explorer select the Stack.cs class file in the Stack Utility project
  2. In the Stack class create a new stack variable of type ArrayList by adding the code below:
    ArrayList stack = new ArrayList();
  3. Hover the mouse over the ArrayList type and select using System.Collections; from the Quick Actions menu
  4. In the Push method replace throw new NotImplementedException(); with the code below
    stack.Insert(0,v);
  5. In the IsEmpty method replace throw new NotImplementedException(); with the code below
    return stack.Count == 0;
  6. In the Pop method replace throw new NotImplementedException(); with the code below
    String result = stack[0].ToString();
    stack.Remove(0);
    return result;
  7. In the Peek method replace throw new NotImplementedException(); with the code below
    return stack[0].ToString();
    Note:
    You may now run the tests and see them pass
  8. From the Test menu select Run > All Tests
    Note:
    All test should display a green check mark indicating a pass.
    Run the Tests and see them Pass
    Green = Good Red = Bad – Run the Tests and see them Pass


    Now that we have the Step Definitions (Glue Code) defined adding new Automated Acceptance tests is a simple a pasting in new Gherkin statements.  If the Product Owner adds new Acceptance Criteria to a user story we can simply copy and paste from the Work Item Tracking (Project Management) tool into our Feature file and we are done.  No new coding.  For example the Gherkin statement below can simply be added to the feature file with

    @MultiPushPopTests
    Scenario: Multi Push Pop
    Given An Empty Stack
    When I Push “Item 1” onto the Stack
    And I Push “Item 2” onto the Stack
    And I Pop a value off the Stack
    Then The result should be “Item 2”

no additional changes necessary.

Advertisements

The Test Driven Development Rhythm

The Test-Driven Development (TDD) Processes follows a pattern known as the TDD Rhythm which dictates the order in which elements of the solution should be created / edited.

Before we can successfully implement TDD a few key agile constructs must exist.  Most importantly we must have ​Tasks derived from a User Stories (or requirements) that define the details of required system feature.  These defined details would include the Acceptance Criteria for feature described in the User Story.  We would then use the task details and acceptance criteria to define our tests.

The TDD Rhythm

1.  Write a Failing Test

The first step in the TDD Rhythm is to Write a Failing Test.  Using the Task Details, we write a test that exercises the functionality defined by the Users Story and expects that the value that is returned is the same as the value that is expected based on the Acceptance Criteria defined in the Task Details.

2. Run the Failing Test

Run the test to see it fail. This is an interesting step as depending on your application architecture may require some minimal project structure be created and project references made for your Failing Tests to even compile before they can run and fail. For example, if you are storing all of your Business Logic in a Class Library Project called BusinessRules that compiles as a Windows .dll and your Tests are centrally stored in a Test Project then your Class Library Project will have to Exist and the Namespace, Class and Method will have to exist before your Test Project will compile and the Tests will run and fail. Fortunately, Visual Studio includes code generation tools that will create the Classes and Methods as long as the Class Library Project Exists and at least one class with a Namespace statement exists. The Failing Test generated by Visual Studio will throw a NotImplementedException which will obviously cause the method to fail.

3.  Write just enough code to pass the test

This can be a difficult concept to get your mind around for especially when the common simple TDD example code is used.  For example, take a method simply returns a Boolean value to illustrate simple TDD method creation if we start with a test that runs the required Boolean method and expect the method to return a true the code to pass the test would simply be return True;

[TestMethod]


public
void TestGetBool()

{


Assert.IsTrue(BoolApp.BoolHost.GetBool());

}

Figure A. Test Method to test GetBool Method


public
static
bool GetBool()

{


return
true;

}

Figure B. Minimal code needed to pass the test

With this example it may seem like a waste of time to write this minimal code to pass the test as it is clear that the test needs a returned value of true in order to pass so where is the value is writing this useless passing test? Without writing the test that tests the “unhappy path” through our method (aka the test that expects a return value of false) it is hard to see the value is a method that simply returns true with no additional implementation logic. An eager developer may want to just skip to writing implementation logic without wasting time on the simplest code step of the TDD Rhythm but follow the pattern young Jedi. As seen in a slightly more complex method that returns a formatted string, understanding how the output should be formatted in order to pass the test can potentially be much more difficult than just returning a Boolean value of true.


public
void TestWelcomeBack()

{


string expected = “Welcome Back Antoine! Your last visit to the site was 02/01/2016.”;


string actual = WebSite.BizRules.WelcomeBack(user);


string message = “We should get “ + expected;


Assert.AreEqual(expected, actual, message);

}

Figure C. Test Method to test WelcomeBack Method


public
static
string WelcomeBack(object user)

{


return
“Welcome Back Antoine! Your last visit to the site was 02 / 01 / 2016.”;

}

Figure D. Minimal code needed to pass the test

In this example the Literal String returned including the user name Antoine and the last visit date would obviously need to be updated for each user and on each daily visit but the formatting and welcome statement may also be important and could possibly come from a configuration file somewhere. The point is that the minimal code required to pass the test in this case acts as documentation for the method including formatting requirements for returned values. On the next refactoring pass we would update the code in the method to include the code necessary to extract the user name from the user object passed to the method and retrieve the date of their last visit from the membership database and return it in the expected format. The String Literal is our formatting template as we create the implementation code we know what the expected result format looks like.

4. Run the Passing Test

At this point our method has just enough code to pass the test but does not necessarily meet the business requirement nor does it allow us to perform the task described by the user story. This will become obvious as more tests are developed to tests the “unhappy path” or as varying return values are expected by other tests. But at this point we understand what must be done for the test to pass and can keep that in mind as we refactor the code to make it meet the business requirement or for optimization purposes.

5. Refactor the Code

Depending on “minimal” code we wrote to pass the test our first refactoring pass may be to add required functionality or if the required functionality already exist we may be refactoring for Maintainability, Scalability or Performance Optimization. In any event as we refactor the code for whatever reason we can do so with the confidence that any changes that we make have tests in place to ensure that we have not made a change that would break existing functionality already passing tests. If you make a change and all of the sudden tests that were passing stop passing you know you have a problem. The tests can also be used for Gated Check-ins that require that any changes a developer makes to code must pass existing tests before it can be checked into Source Control allow bugs to be identified before they make into our build and potentially out to customers.

The Expanded TDD Rhythm

6. Run All Tests

Once we have refactored our code to include desired functionality or optimize for maintenance, scalability or performance we need to run all tests to ensure that our changes did not break the method that we were working on but also for any methods that depend on this method or its results. This is a necessary step to avoid failed check-ins on Source Control or Continuous Integration (CI) Servers where Gated Check-ins are used. With gated check-ins your check-in cannot break the automated build and all tests must pass or your check-in will be rejected and your code not allowed into source control until the issues are resolved.

7. Repeat

As changes are required we can continue to repeat this process of writing failing tests, coding, passing tests and refactoring until our code for the features we are adding for our updates is “perfect”

https://en.wikipedia.org/wiki/Test-driven_development

https://en.wikipedia.org/wiki/User_story

https://en.wikipedia.org/wiki/Code_refactoring