How to run toddler pdc_import_compose.py locally and get the payload obtained from koji that is to be pushed to pdc.

There are three places in the code that do the push:

• two inside the function _import_compose:

pdc["compose-images"]._(
    {
        "release_id": release_id,
        "composeinfo": composeinfo,
        "image_manifest": images,
    }
)

pdc["compose-rpms"]._(
    {
        "release_id": release_id,
        "composeinfo": composeinfo,
        "rpm_manifest": rpms,
    }
)

• one inside the function ensure_release_exists:

 pdc["releases"]._(release_payload)

What I need is to check what the payload pushed to pdc/compose_images and pdc/compose_roms is while making sure nothing gets pushed there for real.

To solve this, I will run a local server and redirect the post requests to the localhost. First considered option for testing/sending data across network connections, netcat, I was not able to make POST work nc -l -p 8080 POST /post

Checking another option: httpbin Running httpbin in a container as described in the documentation (inside toddlers vagrant VM):

docker run -p 9000:80 kennethreitz/httpbin

Rewriting the post requests to point to localhost/9000:

data={
    "release_id": release_id,
    "composeinfo": composeinfo,
    "image_manifest": images,
}
requests.post("http://localhost:9000/post", data=data)

and

data={
    "release_id": release_id,
    "composeinfo": composeinfo,
    "rpm_manifest": rpms,
}

requests.post("http://localhost:9000/post", data=data)

and in ensure_release_exists:

requests.post("http://localhost:9000/post", data=release_payload)

From another window inside the vagrant, I will run the toddler I’m interested in:

python3 -m toddlers.plugins.pdc_import_compose toddlers.toml 818affe5.txt

In the toddlers.toml config all of the urls are chanded to point to stg for additional safety, but the rest of the requests should be only GET.

Now I see how the POST requests look like, while not posting anything anywhere, but the problem is, that the payload is not visible, I see only “payload” in the request. How to actually see what the payload is?

Setting logging on DEBUG mode, modifying http.client to debuglevel=1, stuff I found on stackoverflow that worked for other folks:

import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
_log = logging.getLogger(__name__)
requests_log = logging.getLogger("urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

But does not really work for me, I still see only “payload”. And what about to use good old print after the post request, to see what it is trying to push? (Heh, could have this idea a slight bit earlier)

Adding print(f"Compose-images data: {data}") after the request and now I get a lot of data printed (and forwarded into a file, from where I split it into two files, image-compose and rpm-compose, removing the header lines and such). Would be nice to see it with some json viewer. jq is complaining it’s not really a json format. Turns out it doesn’t like the single quotes, so turning them all to double quotes. The image-compose is not that voluminous, so it’s easy to change it in vim: :%s/\'/\"/g The second file, rpm-compose keeps crashing vim, in vi better, but still I cannot manage to make the swap. Trying sed, but can’t escape the quotes well. Finally trying cat single-quoted-file.txt | tr "'" '"' | tee double-quoted-file.txt and that works well. jq is still complaining about one False without quotes, adding quotes and jq finally pretty prints it. The output is too large to see on the terminal, so opening it in firefox to see the structure.

And I see the structure well now.