Testing (Camunda)-DMN Tables automatically V.2

Two years ago, I created a tool to test DMNs automatically. You find the why’s and so on here: Testing (Camunda)-DMN Tables automatically

Due to technical problem of updating it, I tried some other technologies, made some improvements, et voilà:

This blog shows you how to use it with Docker, as this is the simplest way so far. If you want to use it as application — please check Testing (Camunda)-DMN Tables automatically.

If you use the DMN Tester already, there are some improvements and the Results are displayed differently — hopefully simpler to understand.

New: Integrated Tests

Yes it was also possible to do Integrated Tests (test with all its required Tables) in Version 1. However it turned out this was limited to very simple cases.

I also try to explain the difference between Unit- and Integrated Tests. You should definitely check the chapter Integrated Tests below.

Breaking Changes

• Integrated Tests are new persisted with the postfix -INT. Just open the existing Dmn Config and save it. Then delete the old configuration.

Requirements

All you need is a Docker Container and some basic knowledge on how to use it.

To follow this easily you can check out the DMN Tester and go to the demo folder:

git clone https://github.com/camunda-community-hub/camunda-dmn-tester.git
..
cd camunda-dmn-tester/demo
// make the start script executable
chmod +x runDmnTester.sh        
ls
- runDmnTester.sh   // script to start Docker
- dmns              // the DMNs
- dmnConfigs        // the DMN Configurations
...

With the standard runDmnTester.sh you will get the demo Dmn Table Beverages, as well as some examples for the different cases you find in this blog.

Run the DMN Tester

You can start the Tester by running (in demo):

./runDmnTester.sh

Or you create a bash file yourself with the following content:

docker run \
  --name camunda-dmn-tester \
   --rm \
   -it \
   -e TESTER_CONFIG_PATHS="/dmnConfigs,/server/src/test/resources/dmn-configs" \
   -v $(pwd)/dmns:/opt/docker/dmns \
   -v $(pwd)/dmnConfigs:/opt/docker/dmnConfigs \
   -v $(pwd)/../server/src/test/resources:/opt/docker/server/src/test/resources \
   -p 8885:8883 \
   pame/camunda-dmn-tester

Ok, line by line:

• --name camunda-dmn-tester Gives a readable name for the container:
• --rm removes the container after finishing the tester.
• -it is optional — the interactive mode gives me nice colors on my mac.
• -e TESTER_CONFIG_PATH="/dmnConfigs,/server/src..."a comma-separated list where you have the DMN configurations.
• -v $(pwd)/dmnConfigs:/opt/docker/dmnConfigs adds the DMN configurations to the container.
• -v $(pwd)/dmns:/opt/docker/dmns adds the DMNs to the container.
• -p 8885:8883 the port of the DMN Tester App — 8885 is the exposed port.
• pame/camunda-dmn-tester is the image you need. This will take the latest version. So if you have troubles — you can use the first version like pame/camunda-dmn-tester:0.14.0.

Open http://localhost:8885 in your browser and you should see something like this:

Let’s Test DMN Tables

We take the DMN from the Decision Requirements Graph (DRG) Example.

So we start with the following file structure:

|-testRunner.sc
|-dmns
  |-dinnerDecisions.dmn
|-dmnConfigs

If you want to try it yourself, just delete the folder dmnConfigs and follow the next simple steps.

1. Select Path where your DMN Configurations are.

You see the base path, where the tester was started. All path must be relative to that. To go up a folder you can use ../myDmns.

You have all paths you configured in runDmnTester.sh > TESTER_CONFIG_PATH. It is also possible to add another path on the fly. The Tester will search in it for DmnConfigs. If the folder does not exist, the Tester creates it.

2. Select the DMN Configurations you want to test

We start be creating our first DMN Configuration. Here is the DMN we want to test:

Dmn Tables Beverages and Dish

First we have to decide if we want to test

• each table by itself >> Test is Unit
• both tables together >> Test is Integrated (or NOT Unit) — Beverages can we test with its dependent table Dish.

2.1 Create a Unit Test

As we want to go easy, we make a Unit Test for the Beverages Table > Hit the button ‘Add Dmn Config’.

DMN Config — Basic Info

You see our base path ‘opt/docker’. This path is important, as the path to the DMN must be relative to it.

• Test is Unit: checked, as we want to test the Beverages table without its dependent table Dish. We will do this in the next chapter.
• Decision Id: You find the decisionId in the DRD in the Camunda Modeler — see screenshot above.
• Path to DMN: The relative path to the dinnerDecisions.dmn. The DMN must be there, otherwise you get a warning.

Now we define the Test Inputs. You find them in the Modeler:

As you can see, that is just a variable in a feel expression.

You have to define any variable that is used in your DMN. Variables can be used in FEEL expression of columns, input cells or output cells.

Here is how we can add Test Inputs:

• Key: The variable you use in the input variable (I haven’t tested to have more than one variable per column expression).
• Type: The type is automatically retrieved from the values you define. Make sure that the type reflects the type of your column definition.
• Values: List all possible input values in a comma-separated list.
• Null value: If it is possible that the input value might be ‘null’ check this box.

The tester will mix all inputs with all its possible values (cartesian product). So try to keep the number of Test Inputs and its values to a minimum.
TIPP: Split your Table in dependent Tables to break the complexity.

Same you can do with Variables. We separate them, as they have no influence in the matching process. However that the engine works correctly, you need to define them. So in Values you normally have only one value.

As we have no Variables in our example, hit the ‘Save’ button.

A Variable is used in the Rule expressions — mostly outputs, not like the Test Inputs that are used in the Column expression (I hope this makes sense).

Here a somewhat stupid example😉:

• Test Inputs: season and numberOfGuests
• Variables: fall and kind

2.2 Create an Integrated Test

So we want to test the Beverages table also as it is used in the process. This means we test it with all its dependent tables — Dish in this example.

Just open the existing ‘beverages’ Dmn Config and uncheck the box ‘Test is Unit’. As the Unit Tests are persisted like ‘decisionId.conf’, Integrated Tests are persisted like ‘decisionId-INT.conf’. So it is possible to have a Unit- and an Integrated Test for a table.

This is NEW! So if you have Integrated Tests already, you need to get rid of the extra config.

Now we have to remove the output from Dish (desiredDish) and replace it with its inputs (season, guestCount).

That’s it! We discuss the result in the the chapter Integrated Tests below.

2.3 Select the the Configs

This is the easiest part — just check the boxes you want to run. Usually when you develop a DMN you would run only this one. Otherwise you just check all.

This will evaluate the DMNs immediately and show the results in collapsed Panels — see next chapter.

3 Check the Test Results

You should see right away, if the evaluation was successful.

The result starts with some basic information about the tested DMN.

New, we have different tables for different types of results. I go through them, one by one. We start with the successful evaluated rows.

3.1 Successful Tests

This table will list all rules that were correctly evaluated and display the according result.

For these inputs your DMN returns a correct result. If the according output is also semantically correct, the business specialist must know. We discuss in a later chapter, how you can easily create a test case after your manual check.

Each Test Case is one row, that gives you the following information:

• blue: The test inputs the DMN table was evaluated with (from the values you defined).
• grey: The DMN Row that matched for these inputs (this row number you find in the Camunda Modeler).
• bisque: The Inputs that matched (these inputs you find in the Camunda Modeler).
• beige: The Outputs that matched (these outputs you find in the Camunda Modeler).

3.2 No matching Rows

If you have inputs, for which there are no matching inputs in the DMN, you get a warning and a list with all these input combinations.

You can do two things to get rid of the warning:

1. Remove the value from the values list in the Dmn Config.
2. Add a row in your DMN, that handles this use case.

3.3 Rules with no matching Test Inputs

This is the opposite to ‘No matching Rows’. You have rows in your DMN that are not used with any of the configured input values.

Again you can get rid of the in two ways:

1. Add additional values to the Dmn Config.
2. Remove these rows from the DMN.

3.4 Tests with Errors

Know errors in general means, that the syntax of your DMN is not correct. Or in other words the DMN Engine throw an exception. Depending on the errors we have more or less information.

Here some examples what can go wrong:

As the message suggests, you chose the Hitpolicy UNIQUE, but here more than one row matched for the inputs.

You have the type date selected in the DMN, but your input is not in the correct format.

Btw. the DmnConfig editor tells you, that you have a string, if the format is not correct - check it out (2021–12–23T00:00 is a correct date).

This is maybe the most confusing problem. You forgot to wrap a string with “….”. So the DMN Engine thinks it must be a variable. Ok, if it is a variable than you must add it to the Dmn Config.

The next examples have errors, that made it impossible to evaluate the DMN at all. So you need to figure out the problem yourself.

The first example is easy:

Know the next one is more tricky, so I added some possible solutions:

As the message tells you, there is a bad FEEL expression, but sadly you do not get any glue where this might be.

Integrated Tests

With Integrated Tests, we evaluate the Dmn Table, including its required Dmn Tables.

The Camunda DMN Engine handles this in the following way for our example ‘Beverages’:

1. Put all Test Inputs and Variables to the context.
2. Get the required tables > Dish
3. Evaluate the Dish table and add its outcome (output Variables) to the context.
   Lesson learnt: If you have only one output, the variable name is the id of the table not the output. So always name the output variable equal to the dependent table id!
4. Evaluate the Beverage table and return the output.

Our example looks then like this:

As you see, this is a bit more involved.

• First the inputs are not directly reflected in the matching inputs.
• We have guestWithChildren, but season and guestCount are missing. On the other hand we have an additional one: Dish.
• If you move the mouse over the button on the right, you see the required tables. Here you find some of the missing inputs.
• Also you find the extra input desiredDish (Dish) — which is the output of the required Dmn Table.

TIPP: As the DMN Engine returns the name, and we defined the expression names (test inputs) >>> Always use the same identifier for the name and id (expression name). So it easier to check the result.

Some points, why I prefer Unit Tests:

• It is way harder to reason about what is going on, than doing one step at the time.
• The more inputs you have the more results you have to check. This grows exponentially (Cartesian Product).

☠️ Disclaimer: I only tested this with a few rather simple examples.

Test Case Creation

As you can imagine, the number of Test Cases can be overwhelming. So what we can do is to mark every Result that is correct and persist it to the DMN Configuration.

Just check the boxes of manually validated (correct) Rows and hit the ‘Create Test Cases from the checked Rows’ button:

Running the Tests again gives you now the following result:

The DMN Row and the outputs are now checked if they haven’t changed. So if we change our DMN that changes the Result we see this right away:

Make sure you also checked the first version, if you want learn more about DMNs and how to test them — Testing (Camunda)-DMN Tables automatically. And also Regression Tests with the DMN Tester is still working.

What’s next

• If you find any bugs or you have any improvements, please create an issue on Github — so I can get rid of the BETA😉.
• The creation of the Dmn Configs, and adding some Test Cases could be done more or less automatically with Camundala. So my next project will be the integration of the Dmn Tester into Camundala .

That’s it — I hope you like version 2 of the Dmn Tester!