Skip to main content

Creating a Python Transformation

Within Infrahub a Transformation is defined in an external repository. However, during development and troubleshooting it is easiest to start from your local computer and run the Transformation using infrahubctl transform.

The goal of this guide is to develop a Python Transformation and add it to Infrahub, we will achieve this by following these steps.

  1. Identify the relevant data you want to extract from the database using a GraphQL query, that can take an input parameter to filter the data
  2. Write a Python script that use the GraphQL query to read information from the system and transform the data into a new format
  3. Create an entry for the Transformation within an .infrahub.yml file.
  4. Create a Git repository
  5. Test the Transformation with infrahubctl
  6. Add the repository to Infrahub as an external repository
  7. Validate that the Transformation works by using the Transformation API endpoint

1. Loading a schema​

In this guide we are going to work with a very simplistic network device model. It won't provide a Transformation that is very useful, the goal is instead to show how Transformations are created. Once you have mastered the basics you will be ready to go on to create more advanced Transformations.

---
version: "1.0"
nodes:
  - name: Device
    namespace: Network
    display_label: "{{ name__value }}"
    attributes:
      - name: name
        kind: Text
        label: Name
        optional: false
        unique: true
      - name: description
        kind: Text
        label: Description
        optional: true

Store the schema as a YAML file on your local disk, and load the schema into Infrahub using the following command

infrahubctl schema load /path/to/schema.yml

2. Creating a query to collect the desired data​

As the first step we need to have some data in the database to actually query.

Create three devices, called "switch1", "switch2", "switch3", either using the frontend or by submitting three GraphQL mutations as per below (just swapping out the name of the color each time).

mutation CreateDevice {
  NetworkDeviceCreate(
    data: {name: {value: "switch1"}, description: {value: "This is device switch1"}}
  ) {
    ok
    object {
      id
    }
  }
}

The next step is to create a query that returns the data we just created. The rest of this guide assumes that the following query will return a response similar to the response below the query.

Convert query to Infrahub SDK objects

We provide convert_query_response option to be toggled to be able to access objects from the GraphQL query as Infrahub SDK objects rather than the raw dictionary response.

This allows you to manage the returned data with helper methods on the SDK objects such as save, fetch, etc. on the returned data rather than having to build a payload to send back to Infrahub to manage the objects.

Read more on the Infrahub Python SDK.

query DeviceQuery {
  NetworkDevice {
    edges {
      node {
        name {
          value
        }
        description {
          value
        }
      }
    }
  }
}

Response to the query:

{
  "data": {
    "NetworkDevice": {
      "edges": [
        {
          "node": {
            "name": {
              "value": "switch1"
            },
            "description": {
              "value": "This is device switch1"
            }
          }
        },
        {
          "node": {
            "name": {
              "value": "switch2"
            },
            "description": {
              "value": "This is device switch2"
            }
          }
        },
        {
          "node": {
            "name": {
              "value": "switch3"
            },
            "description": {
              "value": "This is device switch3"
            }
          }
        }
      ]
    }
  }
}

While it would be possible to create a Transformation that targets all of these devices, for example if you want to create a report, the goal for us is to be able to focus on one of these objects. For this reason we need to modify the query from above to take an input parameter so that we can filter the result to what we want.

Create a local directory on your computer.

mkdir device_config_render

Then save the above query as a text file named device_config_render/device_config.gql.

The query will require an input parameter called $name what will refer to the name of each device. When we want to query for device switch1, the input variables to the query would look like this:

{
  "name": "switch1"
}

3. Create the Python Transformation file​

The next step is to create the actual Python Transformation. The Transformation is a Python class that inherits from InfrahubTransform from the Python SDK. Create a file called device_config_render/device_config.py:

from infrahub_sdk.transforms import InfrahubTransform
class DeviceConfigTransform(InfrahubTransform):
    query = "device_config_query"
    async def transform(self, data):
        device = data["NetworkDevice"]["edges"][0]["node"]
        device_name = device["name"]["value"]
        device_description = device["description"]["value"]
        return {
            "device_hostname": device_name,
            "device_description": f"*{device_description}*"
        }

The example is simplistic in terms of what we do with the data, but all of the important parts of a Transformation exist here.

  1. We import the InfrahubTransform class.
from infrahub_sdk.transforms import InfrahubTransform
  1. We define our own class based on InfrahubTransform.
class DeviceConfigTransform(InfrahubTransform):

Here we need to note the name of the class as we will need it later, optionally we can just call it Transform which is the default name.

  1. We define where data comes from and what API endpoint to use.
    query = "device_config_query"

The query part refers to the the query that we will define in the .infrahub.yml repository configuration file later in the guide.

With this configuration, the endpoint of our Transformation will be http://localhost:8000/api/transform/python/device_config_transform.

  1. The Transformation method
async def transform(self, data):
    device = data["BuiltinTag"]["edges"][0]["node"]
    device_name = device["name"]["value"]
    device_description = device["description"]["value"]
    return {
        "device_hostname": device_name,
        "device_description": f"*{device_description}*"
    }

When running the Transformation, the data input variable will consist of the response to the query we created or the webhook payload if using the Transformation with a Custom Webhook.

In this case, the Transformation returns a JSON object consisting of two keys device_hostname and device_description where we have modified the data in some way. Here you would return data in the format you need.

info

If you are unsure of the format of the data you can set a debug marker when testing the Transformation with infrahubctl:

    async def transform(self, data):
        breakpoint()
        device = data["BuiltinTag"]["edges"][0]["node"]
        device_name = device["name"]["value"]
        device_description = device["description"]["value"]

4. Create an .infrahub.yml file​

The .infrahub.yml file allows you to define both the Transformation and query.

Convert query to Infrahub SDK objects

We provide convert_query_response option to be toggled to be able to access objects from the GraphQL query as Infrahub SDK objects rather than the raw dictionary response.

This allows you to manage the returned data with helper methods on the SDK objects such as save, fetch, etc. on the returned data rather than having to build a payload to send back to Infrahub to manage the objects.

Read more on the Infrahub Python SDK.

See this topic for a full explanation of everything that can be defined in the .infrahub.yml file.

.infrahub.yml
# yaml-language-server: $schema=https://schema.infrahub.app/python-sdk/repository-config/latest.json
---
python_transforms:
  - name: device_config_transform
    class_name: DeviceConfigTransform
    file_path: "device_config_render/device_config.py"
queries:
  - name: device_config_query
      file_path: "device_config_render/device_config.gql"

Two parts here are required, first the name of the Transformation which should be unique across Infrahub and also the file_path that should point to the Python file within the repository. In this example we have also defined class_name, the reason for this is that we gave our class the name DeviceConfigTransform instead of the default Transform.

See this topic for a full explanation of everything that can be defined in the .infrahub.yml file.

5. Test the Transformation using infrahubctl​

Using infrahubctl you can first verify that the .infrahub.yml file is formatted correctly by listing available Transformations.

❯ infrahubctl transform --list
Python Transformations defined in repository: 1
device_config_transform (device_config.py::DeviceConfigTransform)
info

Trying to run the Transformation with just the Transformation name will produce an error.

❯ infrahubctl transform device_config_transform
{'message': "Variable '$name' of required type 'String!' was not provided.", 'locations': [{'line': 1, 'column': 19}]}

Here we can see that our query is missing the required input for $name which is needed to filter the data.

Run the Transformation and specify the variable name along with the tag we want to target.

❯ infrahubctl transform device_config_transform name=switch2
{
  "device_description": "*This is device switch2*",
  "device_hostname": "switch2"
}

We have now successfully created a Transformation. Most of the Transformations you will create would be more complex than this, however the main building blocks will always remain the same. It could be that you need the output in OpenConfig format, as Terraform input variables or any other kind of format.

6. Create a Git repository​

Within the device_config_render folder you should now have 3 files:

  • device_config_render/device_config.gql: Contains the GraphQL query
  • device_config_render/device_config.py: Contains the Python code for the Transformation
  • .infrahub.yml: Contains the definition for the Transformation

Before we can test our Transformation we must add the files to a local Git repository.

git init --initial-branch=main
git add .
git commit -m "First commit"

7. Adding the repository to Infrahub​

In order to avoid having the same instructions over and over please refer to the guide adding a repository to Infrahub in order to sync the repository you created and make it available within Infrahub.

8. Accessing the Transformation from the API​

Once the repository is synced to Infrahub you can access the Transformation from the API:

❯ curl http://localhost:8000/api/transform/python/device_config_transform?name=switch2
{
  "device_hostname":"switch2",
  "device_description":"*This is device switch2*"
}
❯ curl http://localhost:8000/api/transform/python/device_config_transform?name=switch3
{
  "device_hostname":"switch3",
  "device_description":"*This is device switch3*"
}