Debugging Segfaults in PHP

In rare cases it is possible that you encounter a segfault in PHP due to the Tideways extension. We will thoroughly test the extension on demo projects and our own production projects before releasing it, but nevertheless it can happen due to your code running in ways that we haven’t foreseen.

The first step when encountering segfaults is to disable the Tideways extension immediately until you have prepared to debug the problem, so that there is no impact for your users. If you have a segfault on your machine, please send an e-mail to [email protected] immediately so that we can help you find the cause and fix it.

To debug segfaults you need the program GDB installed. Depending on your scenario there are various ways to get information about why a segfault happened.

Segfaults are visible in your syslog file at /var/log/syslog and are log messages that contain the word "segfault". A PHP script that segfaults usually leads to a 502 Error from PHP and the webserver.

1. Using "tideways.log_segfaults" INI Setting

The easiest way to get some information about a segfault in PHP is to use the Tideways extension’s backtrace logging for segfaults. This functionality is disabled by default, because it usually interferes with the handling that the Linux kernel has setup itself, so we recommend to activate it only to debug an immediate segfault.

To use this logging, set the PHP.ini setting "tideways.log_segfaults=1" and the "error_log" INI setting to a file of your choice or "syslog". Then restart PHP (FPM, Apache, …​). When the segfault happens a backtrace should be printed into the error log that looks like this:

[tideways.ERROR] PHP received segfault signal: 11
[tideways.ERROR] PHP Backtrace:

The lines afterwards contain the backtrace that you should send to [email protected]

This feature is available since PHP Extension version 5.0.52 released on August 7th 2019.

2. Segfault in a PHP CLI Script

In this case you only need to run the script wrapped with GDB to get access to the backtrace:

$ gdb php
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
---- (lots of output)
(gdb) run script.php arg1 arg2
Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt

The bt command then renders the backtrace. You can copy the whole output to a support ticket.

3. Segfault in Apache With mod_php or PHP-FPM

If PHP running inside Apache segfaults or when using PHP-FPM, then you can attach GDB to the running process. There are a few downsides to this approach:

  • GDB will first stop the process indefinitely until you call "continue" in the GDB console. If a real user gets served between attaching and continue, then he will wait for the response.

  • In addition the process that is attached to gets slower because GDB starts collecting a lot of information about it.

  • Both Apache and php-fpm usually start a lot of workers/children. You can/should only attach to one of them, so it might take a while until this specific one exhibits the segfault. You can make it easier by redoing the number of workers to 1, but of course this is only a good decision on a development or staging system.

ps aux | grep php-fpm
# select child process id
gdb -p ${process_id}
(gdb) continue
(gdb) bt

Once the attached process segfaults, you can again enter "bt" in the GDB console to see the stacktrace and paste it into a support ticket.

4. Collecting Core Dumps From PHP-FPM

If a segfault is happening within PHP-FPM, then the only debug information you get by default is an entry in "/var/log/syslog". To find out the exact line and backtrace the segfault happened without attaching to the process using GDB you need Linux to keep a core memory dump around and then analyze it with GDB. The benefit here is that you don’t affect the process as much compared to the GDB attach case.  The downside is that its a bit more tricky to get working and generated coredumps can become very large very fast and clutter your disk space. If you are not careful you run out of disk space.

First, you must enable core dump system-wide on Linux with:

sudo -i
echo '/tmp/core-%e.%p' > /proc/sys/kernel/core_pattern
echo 0 > /proc/sys/kernel/core_uses_pid
ulimit -c unlimited

Then set the rlimit_core directive in /etc/php/7.1/fpm/php-fpm.conf to unlimited:

rlimit_core = unlimited

Restart php-fpm and trigger the segfault again.

Use GDB to load up core dumps from /tmp:

$ gdb /usr/sbin/php7.1-fpm /tmp/core-php7.1-fpm.4330
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
...
Core was generated by `php-fpm: pool profiler'.
Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
Still need help? Email [email protected]