Hi,
I use a Minikube k8s setup on my local machine. I have an Ingress with the hostname “k6.test”. The Ingress has two backends which can be reached with different paths. When I run the K6 test script locally everything works fine, but when I execute the test script in Minikube, the connection is refused. The Ingress can be reached with a browser too. How can I solve this problem?
Hi @MariusB,
Welcome to the community forums! 
Have you also configured an in-cluster DNS server to resolve local DNS names inside the cluster? (see option four Ingress DNS | minikube).
Let me know if that answers,
Cheers!
Hi @olegbespalov ,
Thank you for your reply.
Yes I have the ingress-DNS addon active:
I have also an entry in the local etc/hosts file with 127.0.0.1 k6.test.
I normaly use the “minikube tunnel” command and then I can reach the API at k6.test with an get command or with my browser.
Do I need to configure an additional DNS entry in the minikube core-dns config-map for applications inside the cluster?
Hi @MariusB
I believe so, as the documentation (the link that I sent) said:
Sometimes it’s useful to access other applications inside cluster via ingress and by their local DNS name - microservices/APIs/tests. In such case ingress-dns addon should be used by in-cluster DNS server - CoreDNS to resolve local DNS names.
So you need a block, like:
test:53 {
errors
cache 30
forward . 192.168.99.169
}
where 192.168.99.169 is your minikube ip
Let me know if that helps,
Cheers
Hi @olegbespalov ,
I tested some configurations with the DNS. I added a DnsClientNrptRule (Add-DnsClientNrptRule -Namespace “.default” -NameServers “minikube ip”) like they described in the link you sent me. But now my K6 doesnt work anymore. I couldn’t get any logs from the K6-sample Pods.
I found the following errors in my kube-dns:
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:36425->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:48639->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:39893->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:49073->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:40277->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:57768->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:44683->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:55877->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:45956->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:46514->192.168.65.2:53: i/o timeout
And in my K6-Starter Pod I have the following Errors:
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
curl: (6) Couldn’t resolve host ‘k6-sample-service-1’
I deleted the DnsClientNrptRule but I have still the same errors. After this I reinstalled the K6 Operator but the problem is still the same. What can I do?
Hi @MariusB
Now, I’m a bit confused and probably misunderstood you:thinking:
In the first message, you mentioned the hostname k6.test. I thought it was a domain of the service that you are going to load-test with k6, where k6 is running inside the minikube.
The
That’s why I suggested configuring an in-cluster DNS server to resolve local DNS names inside the cluster (it’s option four of the manual from the page Ingress DNS | minikube).
kubectl edit configmap coredns -n kube-system
and adding the block:
test:53 {
errors
cache 30
forward . 192.168.99.169
}
That should help resolve the hostname k6.test (if it’s also configured ) from inside the cluster, in case when you run the pod with the k6 and it tries to access the k6.test service.
I added a DnsClientNrptRule (Add-DnsClientNrptRule -Namespace “.default” -NameServers “minikube ip”
This is part of the instruction that is adding the minikube IP as DNS server for windows, and not with the .test domain, but .default which I’m not sure that it’s right.
Could you maybe please tell me more about your setup?
Hi @olegbespalov ,
Sorry for the confusion. I haven’t any experience with DNS configuration and I think I mixed up some topics.
My Setup is the following:
I have an application in Minikube. This application can be accessed externally over an Ingress. This Ingress has the hostname k6.test. I configured an entry in the etc/hosts file on my windows computer with 127.0.0.1 k6.test. To access the API I usually run “minikube tunnel” in a powershell and afterwards I can access my API without any problem. I tested it with my browser and postman.
My K6 samples run in Minikube too. But the K6 tests couldn’t reach the API I use on k6.test. So I searched for a solution.
I tried the Add-DnsClientNrptRule -Namespace “.default” -NameServers “”. That was obvious a mistake and wrong. But now my K6 doesn’t work anymore at all. I got now the Errors I sent in the message before. I cant get K6 to work again. I reinstalled the operator in a new created namespace, but it doesn’t work. How can I get my K6 to work again? The rest of my applications in Minikube still run without any problem.
The whole reason behind this is I want to check out how good my API works, so I want to loadtest the API on this hostname (or any other hostname, this doesn’t matter). The setup on minikube is just for developing, the real test will be set in a real cluster after I figured my Minikube setup out.
Edit:
Important Info: In the real Test the K6 instances and my Application will be in two different clusters. Because of that I want to access my Application over the Ingress externally in my Minikube setup too.
Hi @MariusB,
Thanks for the details! What’s the namespace you run your app in? Is it test?
It seems to me as if there are two separate issues involved here by now 
Here’s my attempt at keeping the record straight (correct me if I’m wrong):
Add-DnsClientNrptRule -Namespace “.default” -NameServers “”. Can you run the opposite command Remove-DnsClientNrptRule to remove that rule for default namespace? It’s the most likely cause of failure of k6-operator, i.e. this one:Once this failure disappears and k6-operator runner pods are started properly, then it make sense to return to the 1st issue if it still persists.
Also, one more thing, related to:
It might be best to try to avoid relying on anything specific to Minikube or PowerShell so that it’s easier to switch to the future 2-cluster setup. TBH, for your use case, minikube might be even harder to set up correctly than the real cluster due to these limitations.
Hi @olha ,
Thanks for your answer!
All my applications run in the namespace “default”.
I created a new namespace called “test” and installed the k6 operator and my samples in there to check if they work there, but they dont.
I already removed the DNS rule with Remove-DnsClientNrptRule, but my K6 still dont work. Would it help to delete the whole Minikube-cluster? Or is there a mechanic to reset all DNS rules or something?
TBH, for your use case, minikube might be even harder to set up correctly than the real cluster due to these limitations.
Yeah I think at this point It would really be easier, but the real clusters doesn’t exist yet, so i have to continue to work with Minikube for the moment.
Interesting.
For the sake of clean setup I’d say it surely makes sense to start the cluster from scratch if it doesn’t cost much: NPRT now seems like a separate beast entirely 
Operator should work with scripts testing https://test.k6.io even without Ingress: I think it would be beneficial to have that working as a baseline.
All my applications run in the namespace “default”.
I created a new namespace called “test” and installed the k6 operator and my samples in there to check if they work there, but they dont.
If your app is running in default then it should be k6.default instead of k6.test, right?
Hi @olha ,
I try to get k6 to work again at the moment. Let’s forget about my other application and the Ingress at the moment.
I wrote a test script for k6 with one of your example tests.
This test has nothing to do with my ingress or anything else. The Problem is, this test wont start at all too. I got these error messages:
Logs from the k6-sample starter:
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
curl: (6) Couldn’t resolve host ‘k6-sample-service-1’
Logs from kube-dns:
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:36425->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:48639->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:39893->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:49073->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:40277->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:57768->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:44683->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:55877->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:45956->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 k6-sample-service-1. A: read udp 172.17.0.2:46514->192.168.65.2:53: i/o timeout
The k6 service is not able to find the k6 sample pods I guess. But why? How can this Add-DnsClientNrptRule disturb the communication inside a Kubernetes cluster?
I started a complete new cluster with only the K6 operator and test-script inside and I got the same errors again.
I searched for other DnsClientNrptRules which could effect the namespace default but there are no further rules.
output of powershell:
PS C:\Users\becker\Desktop> Get-DnsClientNrptRule | Where-Object {$_.Namespace -eq ‘.default’}
PS C:\Users\becker\Desktop>
Is there anything I could do or test?
Edit:
I tested the DNS resolve with a dummy pod from inside Kubernetes:
PS C:\Users\becker\Desktop> kubectl logs k6-sample-starter-tlm6j
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
curl: (6) Couldn’t resolve host ‘k6-sample-service-1’
PS C:\Users\becker\Desktop> kubectl exec -i -t dnsutils – nslookup k6-sample-service-1
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: k6-sample-service-1.default.svc.cluster.local
Address: 10.98.108.218
The dummy container can find the k6 service without any problem, but the k6-sample starter can’t find it.
Name: k6-sample-service-1.default.svc.cluster.local
This service is in default namespace: is the starter pod also in default? Starter expects runners to be in the same namespace.
It doesn’t make sense that NPRT would impact internal DNS of Kubernetes but just in case, it might make sense to also check /etc/resolve.conf on the starter pod: whether it was changed, etc.
Yes they are all in the default namespace:
PS C:\Users\becker\Desktop> kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
dnsutils 1/1 Running 0 50m
k6-sample-1-8mgf8 1/1 Running 0 2m44s
k6-sample-starter-krz6k 1/1 Running 0 29s
PS C:\Users\becker\Desktop> kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k6-sample-service-1 ClusterIP 10.99.195.121 6565/TCP 3m16s
kubernetes ClusterIP 10.96.0.1 443/TCP 3h12m
The cat /etc/resolv.conf from the starter-pod:
PS C:\Users\becker\Desktop> kubectl exec -ti k6-sample-starter-krz6k – cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
resolve.conf seems to be fine… k8s has a DNS troubleshooting guide, with actions that can be tried in such cases. Linking just in case: Debugging DNS Resolution | Kubernetes
I wonder, a few days ago you mentioned getting timeout errors in kube-dns like this:
[ERROR] plugin/errors: 2 k6-sample-service-1. AAAA: read udp 172.17.0.2:36425->192.168.65.2:53: i/o timeout
Do you still get them now, when starter pod is not resolving while dnsutils is resolving? I’d also compare the IPs in the logs against resource descriptions. And maybe a full restart of everything 