Yaten Kou - Forking Perl POE with console to background eats 100% CPU
Forking Perl POE with console to backgro ...
03/11/2014 18:41:42 +0000
Yaten Kou (yaten)
Our VIOXYS project has one of these daemon made from Perl with POE module that communicates with the cgi scripts for handling request that may need access to server assets and resources such as registering an entry to apache config whenever a new sub-domain is spawned and other similar features. In the past, when we used to host the server with us, we normally just run this daemon on one of the consoles of linux box. It has its own console handler using POE::Wheel::Readline so we can type on the console of the running daemon. When we moved VIOXYS to a remote server (a data center in US), we had to fork this daemon to background so that it does not exit when we logout our console. Initially, all we did was just use 'nohup'. It worked well until we noticed that the server is slowed a bit whenever our daemon is running. After checking the cpu utilization, we noticed that our forked to background daemon is eating 100% cpu utilization on one of the cores. So we searched the internet about the possible cause of this. We thought at first that using 'nohup' is causing the problem, so we search on methods of 'properly daemonizing a program to background'. We found various methods such as using Proc::Daemon and as simple as using the 'fork' command. Both method successfully forked our daemon to background, but both solutions did not prevent our daemon from eating up our CPU resources. So we tried to isolate the problem by disabling the functions of the daemon one at a time. We noticed that when we disabled the initialization for console, CPU utilization went back to normal. So whenever we pass a parameter to our daemon that forks it to background, we disable initialization of console (POE::Wheel::Readline). Also, instead of calling the 'put' function directly for output to our console, we created a new subroutine that will automatically switch between using 'put' function and ordinary 'print' function depending on whether the console is running or not.
I hope this post would help others that might encounter the same problem as we did.