Director service is not running

[Muchaszewski]

I had successfully set up your image once but forgot to set DEFAULT_MYSQL_PASS and decided it would be easier to make a clear rerun of the container. Removed all containers, and all volumes data.

After re-run following issue has appeared:

The Icinga Director Background Daemon is not running. Please check our documentation in case you need step by step 
instructions showing you how to fix this.

Because systemctl does not exist on docker images I have no idea how I can start this daemon.

Without the demon director does not work as it freezes on deployments.

[jjethwa]

Hi @Muchaszewski

That's strange, it should start automatically and is not something we usually start/stop manually. Just to check, you did not set ICINGA2_FEATURE_DIRECTOR to false?

If you docker exec into the container, what happens if you run
supervisorctl status director
and
supervisorctl start director

This container is based on the standard debian image and it does not include systemctl as is the best practice for containers. I use supervisord to help manage the multiple daemons required since it's a little more complicated of an image that the usual 😛

[Muchaszewski]

Hi @jjethwa

That's strange, it should start automatically and is not something we usually start/stop manually. Just to check, you did not set ICINGA2_FEATURE_DIRECTOR to false?

I double-checked but I didn't set this by accident, especially because it's not in the default docker-compose file there was no possibility for me to even find this flag :D

If you docker exec into the container, what happens if you run
supervisorctl status director
and
supervisorctl start director

That's something unexpected... I expected the director service to be stopped...

start obviously thrown an error so I restarted it:
supervisorctl start director director: ERROR (already started)

After restart, the issue persists, where IcingaWeb Deamon throws this error, and the status is "Running"

[jjethwa]

Thanks for the information, @Muchaszewski but now I feel even more confused since it shows as running hehe. Can you try stopping and starting the director service with supervisorctl and see if it reports an issue? If it doesn't help, we'll need to start looking at logs. Do you have any other details that might help me reproduce the issue?

[Muchaszewski]

Can you try stopping and starting the director service with supervisorctl and see if it reports an issue?

I did last time and it starts without any issue, because it's not running lol

I look through the logs but see nothing that would point to this issue. If you need other log files then ones I provided here let me know, but I will try to do another clean run (remove docker container and volumes and start again)

startup.log
icinga2.log
access.log

[Muchaszewski]

Just to make sure that no data is corrupted on the Docker images, volumes, etc I run docker system prune -a while Icinga was stopped. I rerun docker-compose, with and to the same effect...

So my only way to reproduce this issue is to provide you my docker-compose file

version: "3.3"
services:
  icinga2:
    image: jordan/icinga2:latest
    container_name: icinga2
    restart: on-failure:5
    # Set your hostname to the FQDN under which your
    # sattelites will reach this container
    hostname: icinga2
    env_file:
      - ./icinga/secrets_sql.env
    environment:
      - ICINGA2_FEATURE_GRAPHITE=1
      # Important:
      # keep the hostname graphite the same as
      # the name of the graphite docker-container
      - ICINGA2_FEATURE_GRAPHITE_HOST=graphite
      - ICINGA2_FEATURE_GRAPHITE_PORT=2003
      - ICINGA2_FEATURE_GRAPHITE_URL=http://graphite
      # - ICINGA2_FEATURE_GRAPHITE_SEND_THRESHOLDS=true
      # - ICINGA2_FEATURE_GRAPHITE_SEND_METADATA=false
      - ICINGAWEB2_ADMIN_USER=<LOGIN>
      - ICINGAWEB2_ADMIN_PASS=<PASSWORD>
      #- ICINGA2_USER_FULLNAME=Icinga2 Docker Monitoring Instance
      - DEFAULT_MYSQL_HOST=mysql
    volumes:
      - ./icinga/cache:/var/cache/icinga2
      - ./icinga/certs:/etc/apache2/ssl
      - ./icinga/etc/icinga2:/etc/icinga2
      - ./icinga/etc/icingaweb2:/etc/icingaweb2
      - ./icinga/lib/icinga:/var/lib/icinga2
      - ./icinga/lib/php/sessions:/var/lib/php/sessions
      - ./icinga/log/apache2:/var/log/apache2
      - ./icinga/log/icinga2:/var/log/icinga2
      - ./icinga/log/icingaweb2:/var/log/icingaweb2
      - ./icinga/log/mysql:/var/log/mysql
      - ./icinga/spool:/var/spool/icinga2
      # Sending e-mail
      #  See: https://github.com/jjethwa/icinga2#sending-notification-mails
      #  If you want to enable outbound e-mail, edit the file mstmp/msmtprc
      #  and configure to your corresponding mail setup. The default is a
      #  Gmail example but msmtp can be used for any MTA configuration.
      #  Change the aliases in msmtp/aliases to your recipients.
      #  Then uncomment the rows below
      - ./icinga/msmtp/msmtprc:/etc/msmtprc:ro
      - ./icinga/msmtp/aliases:/etc/aliases:ro
    ports:
      - "5665:5665"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.icinga.rule=Host(`<ADDRESS>`)"
      - "traefik.http.routers.icinga.entrypoints=websecure"
      - "traefik.http.routers.icinga.tls.certresolver=myresolver"
  graphite:
    image: graphiteapp/graphite-statsd:latest
    container_name: graphite
    restart: on-failure:5
    hostname: graphite
    volumes:
      - ./icinga/graphite/conf:/opt/graphite/conf
      - ./icinga/graphite/storage:/opt/graphite/storage
      - ./icinga/graphite/log/graphite:/var/log/graphite
      - ./icinga/graphite/log/carbon:/var/log/carbon
  mysql:
    image: mariadb
    container_name: mysql
    env_file:
      - ./icinga/secrets_sql.env
    volumes:
      - ./icinga/mysql/data:/var/lib/mysql
      # If you have previously used the container's internal DB use:
      #- ./data/icinga/lib/mysql:/var/lib/mysql

I don't change anything in msmtp and secrets_sql.env besides providing credentials.

Just note that this configuration was working until I provided DEFAULT_MYSQL_PASS in secrets_sql.env which is weird considering what this have in common with director.

For completeness log from docker-compose ... build/up
docker-compose-up.txt

[jjethwa]

Thanka, @Muchaszewski I'll go through the logs and see if I can reproduce it using your compose file, really appreciate you adding all of the details. Just need a bit of time to wrap up some other tasks and I'll get down to it 👍

[jjethwa]

@Muchaszewski going to take me a bit longer to get to it today, do you mind trying something? If you start the container up by itself with a normal docker run, does the daemon connectivity work? No volumes or anything, just a fresh container run to make sure there's nothing on the host side that might be causing an issue.

[Muchaszewski]

Just confirmed:

docker run -p 80:80 -h icinga2 -t jordan/icinga2:latest - works as expected

[jjethwa]

Thanks @Muchaszewski

Do you mind sharing the director configs from the volumes you are using?

[Muchaszewski]

I'm not modifying any other settings, everything is generated after docker-compose is run.

Steps:

• Create icinga folder
• Create icinga/secrets_sql.env, icinga/msmtp/msmtprc, icinga/mstp/aliases files
• docker-compose build
• docker-compose up -d

I can provide director configs that are created but if you could point to where they are as by default I have access denied on all the config folders, and searching through them could take me a while.

[Muchaszewski]

@jjethwa I have found an issue, it was either of the 2 things.

Because Director was not working after I added DEFAULT_MYSQL_PASS to secrets_sql. My idea was that it was either a new line ending or DEFAULT_MYSQL_PASS does not support one of those special signed !*%@ (yes my password had all of them).

So I have changed line endings from WINDOWS to UNIX and removed all special characters from the secrets_sql and it resolved my issue! 😃

Line endings could be an issue because \r is not escaped properly which is present only on windows, but any special characters could be an issue as well.

[jjethwa]

That's great news, @Muchaszewski ! That would definitely do it! Really glad you found the issue 😃