Important Update 📣: Breaking changes in release of testframework lib v2.0.0

Introduction

This is an announcement of our upcoming release of the testframework library. This release will introduce breaking changes in the way that you will write tests.

We understand that breaking changes can be challenging, so we want to provide you with all the information you need to make a smooth transition. This post outlines the details of the changes, the impact they will have, and how you can prepare.

Summary of changes

• Moving of Testframework: moving all testframework related code to IntegrationTests.TestFramework

• Renaming of TestOpsController & TestOpsControllerInflux: we’ve heard the feedback and will in the future use OpsClient & OpsClientRecorded.

• New “TestClient”: Introduction of a new client for all things related to the testbrain

• Removal of .new_from_testbench method: The clients will from here on require a testbench to be instantiated.

These changes are necessary for us to centralize the development of our client code, since we were developing features directly in this repository as well as in IntegrationTests.TestFramework.

Furthermore this change allows us to remove the user-facing influxdb and mqtt clients, hopefully reducing the complexity and barriers of entry.

Details of the changes

1. Moving of Testframework

What?

As some of you may know the utils/testframework directory was only meant as a temporary solution while we worked on finalizing the clients and the release process for the library. We’re now officially ready to release this library and will be removing this directory when we rollout the release.

Why?

Originally there was just one repository, namely IntegrationTests.TestFramework and we didn’t have any way to import the client code to this repository, so this was copy pasted originally. This is where our problems began 🙈. We eventually added some features to the ops client in this repo and other features to the original repository. This escalated once other users started using the repository and contributing to the client, sometimes writing new functions to implement features that we already implemented in the original repository. This made it difficult to maintain and gives a false impression that it is a requirement to contribute to the client code in order to write tests. This is not true, since we want to provide high level abstractions that are intuitive to call.

Impact

Instead of importing from utils.testframework you’ll in the future import directly from testframework. This is a private library uploaded to an azure artifact feed, which means you’ll have to authenticate towards this feed. Details on this is already documented in the readme.

If you have feature request please talk directly to us or you can directly make a contribution in IntegrationTests.TestFramework

Example:

An example on creating clients for the

# import from utils
from utils.testframework.ops.tester.tester_controller import TestOpsController
from utils.testframework.ops.tester.tester_influx_controller import TestOpsControllerInflux

@pytest.mark.parametrize(
    "testbench",
    BENCHES,
    ids=get_testbench_id,
)
def test_example_filter_on_inventory(testbench: TestBench):
    # from the given testbench we can create a ops client
    ops_client = TestOpsController.new_from_testbench(default_timeout=5, tb=testbench)

From Release 2.0.0 onwards, you will do like this instead

from testframework.clients.ops import OpsClient # This can used for no brainers

# if you like the annotations in the dashboard, use OpsClientRecorded
from testframework.clients.ops.tester.tester_client_recorded import OpsClientRecorded

@pytest.mark.parametrize(
    "testbench",
    BENCHES,
    ids=get_testbench_id,
)
def test_example_filter_on_inventory(testbench: TestBench):
    assert testbench is not None
    # from the given testbench we can create a ops client
    ops_client = OpsClient(default_timeout=5, tb=testbench)

2. Renaming of TestOpsController & TestOpsControllerInflux:

What

Not much to remark on here other than the fact we’re renaming from TestOpsController & TestOpsControllerInflux to something (hopefully) descriptive of what the class is.

• OpsClient

• OpsClientRecorded

Why

Similar to the previous point, we had the “Controller” name due to legacy reasons in the testframework, but agree with recent feedback that “Client” fits better. We also chose to go with “Recorded” over influx as we want to “hide” away the internals, since the user shouln’t need to know, and also in case we decide on different technologies in the future.

Examples

Before:

client = TestOpsController(**args)
client_with_annotations = TestOpsControllerInflux(**args)

Now:

client = OpsClient(default_timeout=10, tb=testbench)
client_with_annotations = OpsClientRecorded(default_timeout=5, tb=testbench)

3. New “TestClient”

What

We’re happy to finally release the TestClient which is our new abstraction for all things related to the testbrain. This client will be the only object you’ll need to be able to get measurements and control testbrain actuators. These being:

• car

• pvsystem

• grid

• solar

• wallbox

• houseload

This is aligned with the physical architecture of a testbench. Take the inverter bench as an example:

In the above diagram only the house load is depicted but there can be multiple “sub-components”.

In the testlab, we have the “IOT” + the “Testbrain” which make up the testbench. With this release, we are essentially providing a client for each of these objects. OpsClient is the client to interact with the IOT’s API and the TestClient interacts with the testbrain’s API.

This means no more influx/mqtt clients that you have to jungle with in order to read power values and no more creating a contactor interface.

Why

It was a natural decision following the new and improved OpsClient to also have a corresponding TestClient with a similar interface. This is built the same way as the OpsClient i.e you construct it from a testbench item in the inventory and we have tried to collect all the required items into this client and make nice abstractions on top of it.

Impact

The influx_client and mqtt_client will be removed - these have been abstracted away under the TetsClient. So you’ll no longer have to instantiate instances of these and have to know which sampling points you need to pass into them.

Example

Before, if you wanted to set the pv system into self consumption and read it’s value it would look something like this, where you’d have to use the pytest fixtures. So first you’d have to check in settings if the bench you had was defined in the enum

class TestBenchTarget(Enum):
    """Available Testbenches as Target for tests"""

    TESTBENCH_9 = "9_Inverter_Huawei_Luna2000"
    TESTBENCH_10 = "10_Inverter_FoxESS_H3"
    TESTBENCH_13 = "13_Inverter_Sungrow_SHRT"
    TESTBENCH_14 = "14_Inverter_FoxESS_H3"
    TESTBENCH_15 = "15_Inverter_FoxESS_Smart"
    TESTBENCH_19 = "19_Inverter_FoxESS_Smart"
    TESTBENCH_23 = "23_Wallbox_Starcharge_Aurora"


# Change value here for testing against a spercific bench
TESTBENCH_TARGET = TestBenchTarget(os.getenv("TESTBENCH_TARGET", TestBenchTarget.TESTBENCH_23.value))

Then once you had that you could write a test

def test_self_consumption(ops_interface, mqtt_client):
    ops_interface.post_pv_action_sc(OpsAction.SELFCONSUMPTION)
    # You have to know which MQTTVariable and SamplingPoint you'd want
    brain_powermeter_import_watts = mqtt_client.get_number_value(MQTTVariables.POWERTOTAL, SamplingPoint.LOADL1)

def test_self_consumption(testbench: TestBench):
    client = OpsClient(testbench)
    test_client = TestClient(testbench)

    client.post_pv_action_sc(OpsAction.SELFCONSUMPTION)
    # method to get power is a simple call:
    brain_powermeter_import_watts = test_client.pysystem.get_power_l1()

The beauty of the new implementation is that we are no longer dependent on having the right configuration inside config.py i.e we no longer have to maintain:

    case TestBenchTarget.TESTBENCH_13.value:
        INFLUXDB_TOKEN = INFLUXDB_DEFAULT_TOKEN
        IP_TESTBRAIN = "http://brain13.lab.enpal.io"
        OPS_API_BASE = "http://bench13.lab.enpal.io/api/"
        MODBUS_CONTAINER_API_BASE = "http://bench13.lab.enpal.io:7080/api/v1"
        MQTT_IP = "brain13.lab.enpal.io"
        MQTT_UUID = "54773416f4a8fec9ec3c0f5dec697ab9"

4. Removal of .new_from_testbench method

What

The classmethod new_from_testbench will be removed, since this feature has now become the default. Meaning when you init a client object, you must give a TestBench instance. From the instance it will resolve URLs and other things necessary to construct the client.

Why

We want to leverage the information available in the inventory model, to create clients from these without having to know parameters such as the ones listed in the config.py, every time you wanted to initiate a new testbench client. Initially the method was added as a classmethod so that you the client was backwards compatible with the old interfaces. We’d like to see more use of the inventory and will now make it a requirement going forward.

Impact

Tests that were already using the method can do an easy migration by replacing TestOpsController.new_from_testbench(tb, 5) with just OpsClient(tb, 5). This creates your client instance with a default timeout of 5 seconds for instance.

Note:

This change also means creating standalone, no brainer clients from a URL won’t be possible. We’ve received the feature request to enable this for connecting to remote client systems and are in process of refining this.

def test_example_filter_on_inventory(testbench: TestBench):
    # BEFORE:
    ops_client = TestOpsController.new_from_testbench(default_timeout=5, tb=testbench)
    # AFTER:
    ops_client = OpsClient(default_timeout=5, tb=testbench)

Migration Plan

Before rolling these changes out we want to collect your feedback, and evaluate if there are some last minute changes to be added to the release.

We want to make this transition as seamless as possible for the users of the repository and will be migrating existing tests to the new client ourselves, before adding version 2.0.0 to the requirements.