K6 http2+tls error_code 1000 unexpected EOF

Grafana k6 OSS Support
1

Hi all,

I’d be glad to have some help on tracking down an issue I’m having with k6 + tls.

My arquitecture is as follows:
home-sep22-Page-4.drawio

I’m running K6 from a machine that works with http but not with https.

the code is simple like:

this works

import http from 'k6/http';
import { check } from "k6";

export default function() {
  let res = http.post("http://lab003:4318/v1/traces", JSON.stringify(payload), {
        headers: {
            "Content-Type": "application/json"
        }});
    console.log(res);
}

when using NLB’s endpoint + SSL it breaks

  let res = http.post("https://<NLB DNS>:4318/v1/traces", JSON.stringify(payload), {

res output =

"error": "unexpected EOF",
"error_code": 1000,

and i’m running it as

k6 run --insecure-skip-tls-verify test.js

can anyone help me?

edit:
also worth mentioning that it works with python
import requests

requests.post("<NLB URL>", json=payload, verify=False)

edit2:
I’m using OS redhat 8.5
K6 version v0.40.0

regards,
Carlos.

3

Hmm this is quite weird :thinking: Which k6 version and OS are you using?

4

Hi @ned ,

thank you for your reply.

I’m using OS redhat 8.5
K6 version v0.40.0

regards,C.

5

I just noticed something in your original post. The port number in both the HTTP and the HTTPS requests was 4318, which doesn’t seem correct. Shouldn’t the HTTPS port be 443, or at least something else different from your HTTP port?

6

I’m using it like this.
you can set different configuration to each port.

nlbconsole
nlbconsole1201×445 29.8 KB

7

Interesting… I found the cause. But I still don’t know why and how to solve it.
It seems that there’s some issue regarding HTTP2 in AWS NLB. I’m not an expert in TLS and HTTP1,2 versions.
But when using TLS + HTTP1 it works
TLS + HTTP2 even chrome and curl fails

can anyone help?

$ curl https://mydomain.com:4318/
curl: (16) Error in the HTTP2 framing layer

with k6

import http from 'k6/http';
import { check, sleep } from 'k6';

export default function () {
    let url = "https://test-api.k6.io/"
    url = "https://mydomain.com:4318/"
  const res = http.get(url);
    console.log(JSON.stringify(res, null, 4))
  sleep(1);
}

{
    "remote_ip": "xxxxxxx",
    "remote_port": 4318,
    "url": "https://mydomain.com:4318/",
    "status": 0,
    "status_text": "",
    "proto": "",
    "headers": {},
    "cookies": {},
    "body": null,
    "timings": {
                "duration": 152.277919,
                "blocked": 455.039955,
                "looking_up": 0,
                "connecting": 151.581168,
                "tls_handshaking": 303.261604,
                "sending": 0.162776,
                "waiting": 152.115143,
                "receiving": 0
            },
    "tls_version": "",
    "tls_cipher_suite": "",
    "ocsp": {
                "produced_at": 0,
                "this_update": 0,
                "next_update": 0,
                "revoked_at": 0,
                "revocation_reason": "",
                "status": ""
            },
    "error": "unexpected EOF",
    "error_code": 1000,
    "request": {
                "method": "GET",
                "url": "https://mydomain.com:4318/",
                "headers": {
                    "User-Agent": [
                        "k6/0.40.0 (https://k6.io/)"
                    ]
                },
                "body": "",
                "cookies": {}
            }
}

nlbconsole2
nlbconsole21145×231 20.8 KB

ty,c.

8

Shot in the dark, but out of curiosity, which TLS versions are supported by the components of your architecture (NLB, and the others)?

I believe k6 uses Go crypto/TLS and hence version 1.2 of the protocol. That could also be something for investigating.

9

Something that might help is to capture the traffic and inspect it with WireShark or something like that. Since a few versions ago, k6 supports the SSLKEYLOGFILE environment variable, so you can dump the TLS keys to a file and inspect the traffic. It seems that we still haven’t documented that feature fully (Document the usage of SSLKEYLOGGER · Issue #754 · grafana/k6-docs · GitHub), but there are tutorials on how to use it in other tools that should also apply for k6, for example: myF5

10

If even chrome and curl fails I would argue this is an AWS issue :thinking:

11

From a quick google search I got this Which seems to suggest that NLB just forwards TCP forward but does not actually handle TLS or actually HTTP/2.

So my guess will be that the application behind the NLB does not handle this correctly.

It also suggests that ALB will do this correctly.

This in my mind raises more questions than it answers around why NLB even has “http2” option … but :person_shrugging:

12

Closing this at least for now as it solved my needs. feel free to continue the discussion though.

Answering @mstoykov

So my guess will be that the application behind the NLB does not handle this correctly.
It also suggests that ALB will do this correctly.

yes it works on ALB and yes it seems that the application behind it needs to have support for http2. just successfully tested with https2 tls grpc backed endpoint.

thank you all for your help!

regards,c.