Member-only story
This Is Why Our 3000 Apache Servers Went Down On The First Day of 2022
Imagine a beautiful Saturday morning and the first day of the new year, you wake up and see a notification that says your whole infrastructure is down!!!
This happened to a colleague of mine. Real story.
Early Morning
The most important action to take was recovering all services to minimize the outage. We restarted all apache servers and it seemed that it was functioning without any problems. Now it was the time to answer the question of why; Why on earth are all servers suddenly going down on the first day of the year? It couldn’t be a coincidence, could it?
This error log was all we had on all our servers:
AH00171: Graceful restart requested, doing restartlibgomp: could not create thread pool destructor.
libgomp!? To respect programmers' culture first thing we did was google this issue. We found the same issue opened in serverfault. But no answer, at least not something that we can use! Although something about the issue was weird for us; as the reporter mentioned the crash occurs every 24–36 hours!
Excogitation
Back to the error. Every morning we do a simple log rotation just to start a fresh new file to write logs on. The restart is for that. It seems that the apache has been reloaded successfully but crashed by the libgomp error immediately.
Devastated by opening and reading countless pages on Google, we started to dive into the libgomp source code to see what is going on there. What is libgomp? According to its home page:
The GOMP project has developed an implementation of OpenMP for the C, C++, and Fortran compilers … GOMP simplifies parallel programming for all GNU system variants.
