Background
A friend runs a locked-down server with a Grsecurity-hardened kernel with CONFIG_GRKERNSEC_PROC enabled, which limits the visibility of a normal user much more than the average Linux system. Similar /proc-hiding features are also available in the vanilla kernel. This makes standard commands such as ps completely oblivious to other user's processes. I wanted to see whether there's some kernel APIs that'd allow us to peek behind this "curtain". No new exploits are presented, but I thought I'd share some tips and tricks.
TPE, no binaries for thee?
The first obstacle to poking around on this server was Grsecurity's Trusted Path Execution (TPE), a mechanism that limits the execution/mapping of binary images: with this option enabled, binaries can only be loaded from directories owned by root. While we can't bring in new executables, interpreters like Python can of course be used to execute a file as a script, which would be a natural start.
Python, as a high level language, doesn't directly expose the raw syscall interface, so I tried to reach for glibc functions via the ctypes FFI. However, this was unsuccesful as ctypes appears to be broken under Grsecurity (see Debian bug #781578). A quick skim through the ctypes Python source code reveals that the underlying native module is called _ctypes. By skipping the Python module and cherry-picking classes and functions from this underlying C library, I was able to cobble together some Python code that gets us access to the libc "syscall" function.
from _ctypes import RTLD_LOCAL, dlopen, CFuncPtr, FUNCFLAG_CDECL, _SimpleCData class c_long(_SimpleCData): _type_ = "l" class FP(CFuncPtr): _restype_, _flags_ = c_long, FUNCFLAG_CDECL class M(object): _handle = dlopen(None, RTLD_LOCAL) sc = FP(("syscall", M())) # sc(syscall_no, param1, param2, ...)
With this sc() function, we're able to invoke abitrary syscalls. We may now either write our tools in Python, build a mechanism to proxy syscalls and responses back and forth from our machine, or maybe even combine this with a Python x86-64 emulator plus some ELF loader code to "execute" unmodified custom tools on the host.
Give me the numbers - listing PIDs, SIDs, UIDs
As described above, the Grsecurity option CONFIG_GRKERNSEC_PROC (like the hidepid=2 mount option) hides /proc/PID entries for processes not running under our UID. Thus ps does not even give us the PIDs of other users' processes.
Interestingly, many syscalls return errors if you pass them a non-existent PID. So a simple brute-force loop around e.g. the capget() function will reveal the existence of a certain PID as well as the capability set of that PID. There are a few other APIs such as getsid() and getpgpid() which allow us to group PIDs together into sessions and process groups. This gives visibility to e.g. which PIDs might've been executed from the same screen/tmux window (in the case of an interactive user).
The capabilities returned by capget() indicate whether a process is in possession of various capabilities (~root-ish), but is there anything else we could learn about UIDs active on the system? One trick is to perform a call such as getpriority(PRIO_USER, uid), which returns an ESRCH error in case the given UID does not have any processes. By looping over all possible UIDs, we can detect e.g. cron jobs that run as specific users.
Filesystem snooping: inotify + side-channel
Whilst the above methods give us numerical visibility into what UIDs and PIDs/sessions/proces groups are running, we don't really know what those processes are. I found no direct API to do this, but we can approach this from another direction: instead of going from PID to binary, maybe we can look at the excutable binaries themselves? This leads us to inotify, a mechanism to subscribe to filesystem actions. The only permission requirement is that we have read access to the directory that contains the files in question. Therefore we can recursively subscribe to /bin, /usr/bin, and other directories along PATH.
This gets us practically real-time notifications whenever a process is executed (OPEN event) and terminated (CLOSE_NOWRITE event). If you need to filter for noise from other file open events, you can just set further subscriptions/watches for some libraries and correlate the accesses to those.
The sequence of binaries executed is not the only type of interesting data we can glean via inotify. Especially if the system uses textual log files, it's often useful to subscribe to all write events to e.g. the authentication log. We can then keep track of 'last seen size' for the log file and then derive the size of individual log writes by looking at the delta of the file size.
Putting it all together
All of the above signals can be correlated via time if the system is quite passive - heavier load would drown out the signal of which PIDs and inotify events are connected. As an example, here's an example log transcript from an example tool that does the inotify subscriptions and polls the various syscalls described above on each event:
15:40:42 Observed new PID 6824 (sid=6824) 15:40:42 Opening: bash 15:40:43 Observed new Session ID 6824 15:40:46 Observed new PID 6825 (ROOT!, sid=6824) 15:40:46 Opening: sudo 15:40:47 Auth log size increased by 153 bytes 15:40:55 No longer seeing PID 6825 () 15:40:55 Auth log size increased by 161 bytes 15:40:55 Closing: sudo 15:40:56 Observed new PID 6828 (ROOT!, sid=6824) 15:40:56 Opening: sudo 15:40:58 Observed new PID 6829 (ROOT!, sid=6824) 6830 () 15:40:58 No longer seeing PID 6822 () 15:40:58 Observed new processes with UID 123 15:40:58 Auth log size increased by 231 bytes 15:40:58 Observed new PID 6832 (sid=6824) 15:40:58 No longer seeing PID 6830 () 15:40:58 Opening: apt-get 15:40:58 Opening: dpkg 15:40:58 Closing: dpkg 15:41:02 Observed new PID 6834 (sid=6824) 6835 () 15:41:02 No longer seeing PID 6835 () 15:41:02 Opening: apt-key 15:41:02 Opening: dash 15:41:02 Closing: apt-key ... snip ... 15:41:03 Opening: gpgv 15:41:03 Closing: gpgv 15:41:03 Opening: rm 15:41:03 Closing: rm 15:41:03 Closing: apt-key 15:41:03 Closing: dash 15:41:04 Observed new PID 6952 () 6953 () 6957 () 15:41:04 No longer seeing processes with UID 123 15:41:04 Opening: apt-key 15:41:04 No longer seeing PID 6832 () 6952 () 6834 () 6957 () 6953 () 15:41:04 Opening: dash 15:41:04 Closing: apt-key 15:41:04 Opening: apt-key ... snip ... 15:41:04 Closing: dash 15:41:04 Closing: dash 15:41:04 Opening: dpkg 15:41:04 Closing: dpkg 15:41:05 Auth log size increased by 87 bytes 15:41:05 Opening: dpkg 15:41:05 No longer seeing PID 6828 () 6829 () 15:41:05 Closing: dpkg 15:41:05 Closing: apt-get 15:41:05 Closing: sudo 15:41:07 No longer seeing PID 6824 () 15:41:07 No longer seeing Session ID 6824 15:41:07 Closing: bash
- Although we don't get anything resembling a full command line, we can piece together various facts:
- An interactive user tried to sudo (which is a set-uid binary)
- They mistyped the password a few times and ran sudo again
- After succeeding at 15:40:58, apt-get is run (it was very likely passed to the sudo command)
- Something (likely apt-get) dropped to UID 123 (= apt on this system)
- The sudo command finished at 15:41:05 and the shell/session finished 15:41:07
Not quite equivalent to ps or top -b, but still quite an improvement over the complete blindness standard tools would leave us with!