The database just crashed, I am about to investigate the problem.
What I know so far: Two of the 3 mongodb instances got corrupt. I do not know yet and I am pretty sure that I will never know. By recreating the 2 instances I made a big mistake and have accidentally deployed an old version of mongodb, because I have picked up the wrong yaml file. It took a long time until the mistake was found and I had to start a repair of the remaining mongo instance to check for other potential corruption. I hope that when this is done, I can just connect the other 2 instances to the cluster and when they have synced everything it should be up and running again.
I will publish a more detailed postmortem later and will keep you up to date.