Hickup / unresponsiveness when running on RasPi in Docker

[murphy83]

This is a: Run Issue

Details

I am experiencing a lagginess / unresponsiveness in my (maybe rather unusual) setup. Docker-Pi-Hole is running on a Raspi 3, with a DNS-Loadbalancer (dnsdist) in front of it to work as a proxy and to route DNS-Requests by certain criteria eg. route requests for specific domains / subdomains to a dedicated PowerDNS-Server-Container which also runs on the Raspi. The reason why I have this setup is the need for an updateable DNS-Server for some of my projects (RFC2136) which is not supported by PiHole yet.
To check the health of downstream servers dnsdist querys the downstream server every second (I also tried to reduce this cycle, but it did not change anything).
I also checked for load issues and replaced a Raspi2 with a more recent Raspi3 available, but nothing changed and the average load remains stable without any peaks that could explain the behavior.
I also can rule out heavy traffic / load on the powerdns server - I disabled all requests that would go there and the logs / status of DNSDist also shows no pakets hitting the loadbalancer or even the powerdns server downstream.

The log extracted from dnsdist looks like this:

2022-01-06T19:49:14.937456931Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:49:24.233204958Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:49:39.851252439Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:49:49.333743040Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:49:56.604145829Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:50:06.009935536Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:50:18.539178287Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:50:18.539269272Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:50:40.388395015Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:50:43.568515721Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:51:01.315384742Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:51:37.610656936Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:52:01.588221213Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:52:02.675526701Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:52:20.465449811Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:52:21.547588358Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:52:51.864176408Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:52:57.178850830Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:53:17.087816433Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:53:18.725784457Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:53:33.797885038Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:53:36.978885744Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:54:11.479380734Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:54:16.821364208Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:54:26.099655259Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:54:33.525292816Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:55:01.786483283Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:55:03.159795360Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:55:12.246406465Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:55:19.758467584Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:56:02.607865021Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:56:03.663914445Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:58:02.005286883Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:58:03.097759726Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:58:10.391958351Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:58:13.561893967Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:58:29.249370078Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:58:32.932089703Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'
2022-01-06T19:59:15.431592140Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'down'
2022-01-06T19:59:22.833101913Z Marking downstream Pi-Hole ([fdbd:f077:ab57:53::5]:53) as 'up'

I would not worry about this log, but I also recognise issues with day-to-day usage (lookups in browser take ages or fail for non blacklisted domains). I also checked the logs of pihole if this issues happen at expected times (updates of db) but no pattern or entries in the log showed up.

Related Issues

• I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar

How to reproduce the issue

1. Environment data

• Operating System: Rasbian (Bullseye)
• Hardware: RasPi 3B
• Kernel Architecture: ArmV7, ArmV8 32bit, ArmV8 64bit, etc -->
• Docker Install Info and version:
  • Software source: official docker-ce
  • Supplimentary Software: dnsdist, powerdns-server, mysql
• Hardware architecture: ARMv7

2. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here
   docker run -d --network dns --ip6='fdbd:fa49:ab57:53::5' -v '/etc/pihole/pihole:/etc/pihole' -v './etc/pihole/dnsmasq.d:/etc/dnsmasq.d' pihole/pihole:latest

3. any additional info to help reproduce

These common fixes didn't work for my issue

• I have tried removing/destroying my container, and re-creating a new container
• I have tried fresh volume data by backing up and moving/removing the old volume data
• I have tried running the stock docker run example(s) in the readme (removing any customizations I added)
• I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me)
• I have tried running without my volume data mounts to eliminate volumes as the cause

I did not use the port-mapping of docker in the command as I want to run a "as most ipv6 as possible" network which means no portmapping is required for the PiHole (I only have on on 53/UDP) for dnsdist to allow legacy clients to use the service.

I also shut down the powerdns-sever and the mysql-server running on the Raspi to further rule out on load but the situation remains unchanged.

[murphy83]

Talking with someone also running pihole in docker on a pi revealed a potential issue outside the scope of the pihole image.
As I setup the raspi fresh from the start with a bare image I chose the bullseye version of Raspian. So today I took the time to setup another pi with an identical setup (in fact I recycled a Raspi2 that was still waiting for a useful task). For the time being this looks quite promising and I will further investigate in this direction.

I also experienced issues with building the docker image for dnsdist on a bullseye base image when running on a buster-based raspian - not yet an idea why this happens, but it seems like there are some issues with the bullseye based raspian, at least when running on a Raspi2 or Raspi3 (of course this is not the fault of the pi-hole image)

I will keep you posted about the further details and answers I hope to find in the days to come.

[murphy83]

Seems I was a bit to optimistic last evening. Having a look after a relatively calm weekend with the raspi just sitting there waiting for me to continue I just found the following things in the log:

2022-01-09T20:23:38.392532679Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'down'
2022-01-09T20:23:41.587431435Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'up'
2022-01-09T20:24:01.461030041Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'down'
2022-01-09T20:24:03.017839903Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'up'
2022-01-09T20:24:09.836958511Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'down'
2022-01-09T20:24:10.941088350Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'up'
2022-01-09T20:24:32.913297060Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'down'
2022-01-09T20:24:36.099124661Z Marking downstream Pi-Hole ([fdbd:f177:ab57:53::5]:53) as 'up'

This is just an excerpt of the whole log the issues is not on a regular but frequent basis.

Any Ideas on how to further narrow things down are highly welcome.

[PromoFaux]

Is that your complete run script in the OP? It seems a little sparse :)

[murphy83]

docker run -d --network dns --ip6='fdbd:fa49:ab57:53::5' -v '/etc/pihole/pihole:/etc/pihole' -v './etc/pihole/dnsmasq.d:/etc/dnsmasq.d' pihole/pihole:latest
This is my startup script and I don't think it looks sparse.
In comparison to the orginal docker_run.sh I left out some elements but I don't think they are relevant in my setup.
The port-mapping is not required in an IPv6-only setup with routing - I can access the Pi-Hole directly via its Ipv6-Address, the port 53 is already occupied by dnsdist to have the default dns port exposed to the network.
Same goes for the environment-settings about the virtual host and proxy location - as far as I looked at those, they are not required in the IPv6 environment.
The restart policy might be useful, but as the container is running now and is not stopped by the docker-Daemon (except for restarting the whole Raspi).
So the only things that I still have doubts on are the dns-settings for the container.

I will try the changes once I am back home to access the Raspi.

[murphy83]

Ok checked and restarted the Docker Containerwith all the parameters listed in the docker_run.sh except the port-mappings but issue persists.

[murphy83]

I also had another issue but could rule out that one as well: The power supply seemed a bit unstable on higher loads, so I replaced it with a more powerful one - no more Voltage warnings now, but still no solution.

[PromoFaux]

I was just wondering about whether or not some of the environment variables could help, I was thinking along the lines of ServerIPv6 (soon to be deprecated, mind), but it's probably unrelated.

Do you have some other hardware you can replicate your setup on to rule that out?

[murphy83]

Ok, I did some further research / hardware testing.
I had both versions (buster/bullseye) on both of my available RasperyPis (Model 2 / Model 3).
The only thing I noticed was that there are issues to get docker-bullseye images running on a buster-based machine for some issues with not signed repositories. But this is most likely a different story, because it only affected building a dnsdist-image.

So I decided to ditch docker to see if I have any hardware issues I did not see before. So I installed the pihole natively on one of the RaspberryPi and dnsdist on the other one, using an IPv6-Setup wherever possible.
This setup works, so for the time being I think I can rule out any hardware issues.

Next thing I will try is to run pihole in docker on one of the machines, leaving dnsdist native on the other one and than try to run dnsdist in a docker container as well and see what happens (I think I will get some issues with routing out of the docker-subnets but this should be solveable).

[rdwebdesign]

(I think I will get some issues with routing out of the docker-subnets but this should be solveable).

Try using a macvlan for your container.

"... you can use the macvlan network driver to assign a MAC address to each container’s virtual network interface, making it appear to be a physical network interface directly connected to the physical network." (From docker docs)

[murphy83]

So another weekend and time for testing / debugging. And good news is: Currently it seems like I solved the issue.
Turns out to be something totally different than I originally thought, but #980 got me pointed in the right direction.

When setting up dnsdist and powerdns I decided to pick a baseimage that reflects Raspian as this seemed logical to me: Running on Raspi in a Raspbian environment containers based on somehing Raspbian would be as smooth as possible (no break in the architecture) - so I used "alenalib/rpi-raspbian" as a base image to setup dnsdist and powerdns.

Triggered by the idea in #980 thought about it and decided to give the idea of alpine based containers a spin on my Raspi. As I am not that familiar with Alpine and PiHole yet, I wanted to reduce complexity by doing something I am already knowledgeable about. So an obvious idea was to refactor the images for powerdns and dnsdist.

And what a neat suprise: Changing the images and reconstructing the docker images turned out to be a breeze, only some slight adjustments to match the binary and the different location of config files in Alpine and the containers were ready to start. After the restart of dnsdist with a local authorative DNS and the pihole in a container I was expecting to see the old error again. But: everything just worked :) - no more strange entries in the logs of dnsdist and the responsiveness is good as well.

So I will watch this issue a bit longer to see if really everything is stable and then close this issue here. Thanks for the great work and all the suggestions.

[bahag-schlachterk]

ok issue is resolved, needs to be closed

[PromoFaux]

What were you findings in the end?

[murphy83]

The basic finding was: use the right images as basis (in this case alpine for Raspi did the trick).