Debugging Segfaults in PHP

There is currently a known segfault in the Tideways PHP Extension caused by a Generator-related bug in PHP 8.3.0-8.3.12, 8.2.8-8.2.24, and 8.1.21+. A workaround has been implemented in Tideways PHP Extension in version 5.12.0. Recent Twig versions make extensive use of Generators and applications using these are likely to be affected.

There is currently a known segfault in the interaction between Tideways PHP Extension and Ioncube. You can work around it by disabling Zend Observers with tideways.features.observer=0.

In rare cases it is possible that you encounter a segfault in PHP due to the Tideways extension. We will thoroughly test the extension on demo projects and our own production projects before releasing it, but nevertheless it can happen due to your code running in ways that we haven’t foreseen.

The first step when encountering segfaults is to disable the Tideways extension immediately until you have prepared to debug the problem, so that there is no impact for your users. Also please send an e-mail to [email protected] so that we can help you find the cause and fix it.

Segfaults are usually visible in your syslog file at /var/log/syslog or /var/log/messages and are log messages that contain the word "segfault". A PHP script that segfaults usually leads to a 502 Error from PHP and the webserver.

1. Using "tideways.log_segfaults" INI Setting

The easiest way to get some information about a segfault in PHP is to use the Tideways extension’s backtrace logging for segfaults. This functionality is disabled by default, because it usually interferes with the handling that the Linux kernel has setup itself, so we recommend to activate it only to debug an immediate segfault.

To use this logging, set the following PHP.ini settings:

tideways.log_segfaults=1
; configure error_log only if not previously done. Can be a file path or "syslog"
error_log=/path/to/file.log

Then restart PHP (FPM, Apache, …). When the segfault happens a backtrace should be printed into the error log that looks like this:

[tideways.ERROR] PHP received segfault signal: 11
[tideways.ERROR] PHP Backtrace:

The lines afterwards contain the backtrace that you should send to [email protected]

This feature is available since PHP Extension version 5.0.52 released on August 7th 2019.

Setting this flag can disable other processes hooking into segfaults, specifically it may disable core dumps from being stored on disk.

2. Segfault in a PHP CLI Script

In this case you only need to run the script wrapped with GDB to get access to the backtrace:

$ gdb php
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
---- (lots of output)
(gdb) run script.php arg1 arg2
Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt

The bt command then renders the backtrace. You can copy the whole output to a support ticket.

3. Segfault in Apache With mod_php or PHP-FPM

If PHP running inside Apache segfaults or when using PHP-FPM, then you can attach GDB to the running process. There are a few downsides to this approach:

GDB will first stop the process indefinitely until you call "continue" in the GDB console. If a real user gets served between attaching and continue, then he will wait for the response.
In addition the process that is attached to gets slower because GDB starts collecting a lot of information about it.
Both Apache and php-fpm usually start a lot of workers/children. You can/should only attach to one of them, so it might take a while until this specific one exhibits the segfault. You can make it easier by redoing the number of workers to 1, but of course this is only a good decision on a development or staging system.

ps aux | grep php-fpm
# select child process id
gdb -p ${process_id}
(gdb) continue
(gdb) bt

Once the attached process segfaults, you can again enter "bt" in the GDB console to see the stacktrace and paste it into a support ticket.

4. Collecting Core Dumps From PHP-FPM

If a segfault is happening within PHP-FPM, then the only debug information you get by default is an entry in "/var/log/syslog". To find out the exact line and backtrace the segfault happened without attaching to the process using GDB you need Linux to keep a core memory dump around and then analyze it with GDB. The benefit here is that you don’t affect the process as much compared to the GDB attach case. The downside is that its a bit more tricky to get working and generated coredumps can become very large and quickly reduce your disk space.

If you collect coredumps in production make sure to disable the collection quickly after collecting one or a handful of coredumps, because they are very large and can fill up the harddrive quickly. Disable coredump collection by removing the rlimit_core=unlimited line from php-fpm.conf and restart php-fpm.

First, you must enable core dump system-wide on Linux with:

sudo -i
echo '/tmp/core-%e.%p' > /proc/sys/kernel/core_pattern
echo 0 > /proc/sys/kernel/core_uses_pid
ulimit -c unlimited

Then set the rlimit_core directive in php-fpm.conf for example in /etc/php/7.1/fpm/php-fpm.conf for PHP 7.1 on Ubuntu/Debian to unlimited:

rlimit_core = unlimited

Then restart php-fpm and trigger the segfault again. A coredump file will be written to /tmp whenever a segfault triggers.

Use GDB to load up core dumps from /tmp:

$ gdb /usr/sbin/php7.1-fpm /tmp/core-php7.1-fpm.4330

You can install gdb on Debian/Ubuntu with "apt-get install gdb" or on Alma/Rocky/Fedora/RHEL/CentOS with "yum install gdb"

Enter the gdb command "bt" for backtrace on the console to see the programs backtrace at the time it crashed:

GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
...
Core was generated by `php-fpm: pool profiler'.
Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt

Still need help? Email [email protected]