10

Last week, I upgraded my Linux file server from Fedora 39 to Fedora 40, and several CGI applications written in Perl stopped working. I first noticed it when Foswiki could not show any pages, because it was unable to open its log file.

After unsuccessfully pursuing a theory that the system upgrade had resulted in some incompatibility between (updated) perl libraries and the (same old) Foswiki application, I discovered that an application that I had written myself had the same problem.

I have now reduced it to a very small program, the core of which is just these few lines:

my $file_to_write = "/tmp/writetest.txt";    
unless (open(OUTFILE, ">>", $file_to_write)) {
    print "Failed to open (for append) $file_to_write.<BR>\n";
}
printf "%s %s Write test to $file_to_write\n", ljpDate(), ljpTime();
printf OUTFILE "%s %s Write test\n", ljpDate(), ljpTime();
close OUTFILE;
print "Write completed<BR>\n";

It appears that the open succeeds (I do not get the "Failed .." message), but nothing is written to the file, even though it has mode 666 (-rw-rw-rw-) and it is owned by apache:apache. If the file exists, it is untouched, and if it does not exist, it is not created.

If I run the script from the command line (./writetest.cgi) everything works as expected.

This worked last week before the update. Is there some new sandboxing feature that kills my applications?

Update 1: From the answers/replies/comments below I have learned that the reason my SIMPLE demonstration problem did not work, is that httpd on Fedora runs with PrivateTmp enabled in systemd. I will probably be turning that off by doing a systemctl edit httpd to create a configuration override containing

[Service]
PrivateTmp=false

In fact, this makes the demonstration program work.

But this does not resolve the original problem.

The most severe version happens in the Foswiki web service; however, that is also the most complex one to work on, because the failure to open the log file is wrapped in a net of exception handlers providing Perl traceback from the failure point. I will continue to put this on hold, while I work on the occurance in my own program, which is a lot simpler.

I will work on this by adding a couple of lines that allow me to set the target write location from the URL typed into the web browser, and adding logging to the generated web page. This will allow me to see error codes and how they vary with properties of the targeted location, such as

file ownership

  • "htaccess" properties set by directives in /etc/httpd/cont/httpd.conf
  • ?? ideas ??

I will keep you posted on my progress.

Update 2: This whole idea of systemd sandboxing Apache httpd is mind-boggling. Just as discovering that the place where anybody can create a file is explicitly not working for a CGI program by default makes me wonder how much code is needed in the kernel to support that feature.

My workaround was to create a folder under my home directory, owned by apache and world readable and writable. And amazingly, I now got the same error code (Read-only filesystem) that Foswiki got in the episode that started this quest. So I noticed a hint from one of the people that had looked at my problem: Maybe /home is protected similarly to how /tmp is protected. So I looked at the systemd run unit for httpd again and found a parameter named ProtectHome=read-only ... which sound a lot like my problem. Indeed, the way my system is laid out, every subsystem is installed as a subdirectory under /home. So the solution was simple: Edit the file I created above to override the options for httpd and add the line

ProtectHome=no

right after PrivateTmp=false.

And now it works!

Giacomo1968
  • 58,727
Lars Poulsen
  • 325
  • 1
  • 2
  • 13

3 Answers3

14

Programs may have a different view of the filesystem set up for them by the service manager (or rarely by PAM for user logins, too).

For example, it is very common to configure the Apache2 .service in systemd so that it has an isolated /tmp mounted which is only visible to processes inside the .service – you will see PrivateTmp= enabled in systemctl cat apache2, and if you run findmnt in your CGI script it'll report a different output than findmnt from an interactive logon.

When PrivateTmp is enabled, the service's /tmp is really mapped to /tmp/systemd-private-foo/ on the host (using a bind mount that systemd sets up), so the file creation will succeed¹ – unlike with SELinux, which would return an access denied error (and would log it to dmesg) – but the file is located elsewhere than you think.

¹ That is to say, this sounds to me like a completely different situation than your original "Foswiki can't open log file" issue. If the open function failed, then that most likely was some kind of permissions or SELinux problem. (Of course, the first step should be to check what error code it returned instead of just "it failed" – ideally the wiki should log the failure to systemd journal using the syslog() function, which is always accessible to services and even CGI scripts.)

use Sys::Syslog;
if (open(my $fh, ">>", "/path/to/log")) {
    ...
} else {
    syslog("err", "Could not open /path/to/log: %m");
    die("Could not open log: $!");
}

Often network-based services are configured with a whole ton of filesystem namespacing options, including making their view of /home read-only (or invisible), so that they won't reach out to where they generally shouldn't.

makes me wonder how much code is needed in the kernel to support that feature.

In absolute terms – a lot, but relatively speaking – zero, as it relies on a pre-existing kernel feature: the same filesystem namespacing is what powers "container" systems such as Docker or LXC, and its kernel implementation predates the creation of systemd by some 3 or 4 years.

grawity
  • 501,077
4

This sounds like an SE Linux headache.

Check if SE Linux is enabled by doing this:

sestatus

The output should look something like this:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

If so, you can enable CGI access for HTTPD like this:

sudo setsebool -P httpd_enable_cgi on;

Then change the file system stuff for SE Linux like this:

sudo semanage fcontext -a -t httpd_sys_script_exec_t /var/www/cgi-bin/script.cgi
sudo restorecon -vR /var/www/cgi-bin/script.cgi

And things should work fine.

Notes: Just note that you might have to install policycoreutils-python to run those commands. I do this to install that package on Red Hat 7:

sudo yum install policycoreutils-python

Otherwise, you can disable SE Linux like this to run tests on the script:

sudo setenforce permissive

To permanently disabled SE Linux, edit the SE Linux config (/etc/selinux/config) and change the value of:

SELINUX=enforcing

To this:

SELINUX=permissive

And restart the machine for that value to take.

Giacomo1968
  • 58,727
4

I assume that the script is in fact running via CGI (so you see the other outputs all the way to print "Write completed<BR>\n";).

Could it be that in the upgrade Fedora¹ has switched to having per-user temporary directories, either for all accounts or just for service accounts like apache, using something like pam-tmpdir or SystemD's PrivateTmp option? If so then your script may be successfully be writing, but to a different location (perhaps a tmpfs based one that is never written to any real drive) than the one you are reading from as your own user.

To test for this, create a world-writable directory elsewhere and try to write to that instead.

This would explain your simple example though, as it is explicit writing to /tmp. With regard to

Foswiki could not show any pages, because it was unable to open its log file.

This might be a separate permissions issue. Do you get any further diagnostic detail for “unable to open” (permission denied?, not found?, etc.)? Also, where is that log file stored? Did you install the application manually (in which case a permissions change is most likely) or via a package (in which case perhaps the package has changed default config and the log is trying to be written somewhere different to prior to the upgrade).

[1] Caveat: I don't use Fedora at all, so don't know if this is a change that has been made between releases or at all, though I have seen such behaviour elsewhere, so it jumped to mind as a possibility

David Spillett
  • 23,830