Go low - hardeninghttps://bling.kapsi.fi/blog/2017-02-11T17:40:00+02:00Process reconnaissance without /proc2017-02-11T17:40:00+02:002017-02-11T17:40:00+02:00Otto Ebelingtag:bling.kapsi.fi,2017-02-11:/blog/no-proc-process-recon.html<p class="first last">Servers with <tt class="docutils literal">/proc</tt> restrictions make it hard to see what processes other users executed and when. Can we we learn at least something of these other processes despite the restrictions?</p>
<div class="section" id="background">
<h2>Background</h2>
<p>A friend runs a locked-down server with a Grsecurity-hardened kernel with <tt class="docutils literal">CONFIG_GRKERNSEC_PROC</tt> enabled, which limits the visibility of a normal user much more than the average Linux system. Similar <tt class="docutils literal">/proc</tt>-hiding features are also available in the vanilla kernel. This makes standard commands such as <tt class="docutils literal">ps</tt> completely oblivious to other user's processes. I wanted to see whether there's some kernel APIs that'd allow us to peek behind this "curtain". No new exploits are presented, but I thought I'd share some tips and tricks.</p>
</div>
<div class="section" id="tpe-no-binaries-for-thee">
<h2>TPE, no binaries for thee?</h2>
<p>The first obstacle to poking around on this server was Grsecurity's Trusted Path Execution (TPE), a mechanism that limits the execution/mapping of binary images: with this option enabled, binaries can only be loaded from directories owned by root. While we can't bring in new executables, interpreters like Python can of course be used to execute a file as a script, which would be a natural start.</p>
<p>Python, as a high level language, doesn't directly expose the raw syscall interface, so I tried to reach for glibc functions via the <tt class="docutils literal">ctypes</tt> FFI. However, this was unsuccesful as ctypes appears to be broken under Grsecurity (see Debian bug #781578). A quick skim through the <tt class="docutils literal">ctypes</tt> Python source code reveals that the underlying native module is called <tt class="docutils literal">_ctypes</tt>. By skipping the Python module and cherry-picking classes and functions from this underlying C library, I was able to cobble together some Python code that gets us access to the libc "syscall" function.</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">_ctypes</span> <span class="kn">import</span> <span class="n">RTLD_LOCAL</span><span class="p">,</span> <span class="n">dlopen</span><span class="p">,</span> <span class="n">CFuncPtr</span><span class="p">,</span> <span class="n">FUNCFLAG_CDECL</span><span class="p">,</span> <span class="n">_SimpleCData</span>
<span class="k">class</span> <span class="nc">c_long</span><span class="p">(</span><span class="n">_SimpleCData</span><span class="p">):</span> <span class="n">_type_</span> <span class="o">=</span> <span class="s2">"l"</span>
<span class="k">class</span> <span class="nc">FP</span><span class="p">(</span><span class="n">CFuncPtr</span><span class="p">):</span> <span class="n">_restype_</span><span class="p">,</span> <span class="n">_flags_</span> <span class="o">=</span> <span class="n">c_long</span><span class="p">,</span> <span class="n">FUNCFLAG_CDECL</span>
<span class="k">class</span> <span class="nc">M</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="n">_handle</span> <span class="o">=</span> <span class="n">dlopen</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">RTLD_LOCAL</span><span class="p">)</span>
<span class="n">sc</span> <span class="o">=</span> <span class="n">FP</span><span class="p">((</span><span class="s2">"syscall"</span><span class="p">,</span> <span class="n">M</span><span class="p">()))</span> <span class="c1"># sc(syscall_no, param1, param2, ...)</span>
</pre>
<p>With this <tt class="docutils literal">sc()</tt> function, we're able to invoke abitrary syscalls. We may now either write our tools in Python, build a mechanism to proxy syscalls and responses back and forth from our machine, or maybe even combine this with a Python x86-64 emulator plus some ELF loader code to "execute" unmodified custom tools on the host.</p>
</div>
<div class="section" id="give-me-the-numbers-listing-pids-sids-uids">
<h2>Give me the numbers - listing PIDs, SIDs, UIDs</h2>
<p>As described above, the Grsecurity option <tt class="docutils literal">CONFIG_GRKERNSEC_PROC</tt> (like the <tt class="docutils literal">hidepid=2</tt> mount option) hides <tt class="docutils literal">/proc/PID</tt> entries for processes not running under our UID. Thus <tt class="docutils literal">ps</tt> does not even give us the PIDs of other users' processes.</p>
<p>Interestingly, many syscalls return errors if you pass them a non-existent PID. So a simple brute-force loop around e.g. the <tt class="docutils literal">capget()</tt> function will reveal the existence of a certain PID as well as the capability set of that PID. There are a few other APIs such as <tt class="docutils literal">getsid()</tt> and <tt class="docutils literal">getpgpid()</tt> which allow us to group PIDs together into sessions and process groups. This gives visibility to e.g. which PIDs might've been executed from the same screen/tmux window (in the case of an interactive user).</p>
<p>The capabilities returned by <tt class="docutils literal">capget()</tt> indicate whether a process is in possession of various capabilities (~root-ish), but is there anything else we could learn about UIDs active on the system? One trick is to perform a call such as <tt class="docutils literal">getpriority(PRIO_USER, uid),</tt> which returns an ESRCH error in case the given UID does not have any processes. By looping over all possible UIDs, we can detect e.g. cron jobs that run as specific users.</p>
</div>
<div class="section" id="filesystem-snooping-inotify-side-channel">
<h2>Filesystem snooping: inotify + side-channel</h2>
<p>Whilst the above methods give us numerical visibility into what UIDs and PIDs/sessions/proces groups are running, we don't really know <em>what</em> those processes are. I found no direct API to do this, but we can approach this from another direction: instead of going from PID to binary, maybe we can look at the excutable binaries themselves? This leads us to <tt class="docutils literal">inotify</tt>, a mechanism to subscribe to filesystem actions. The only permission requirement is that we have read access to the directory that contains the files in question. Therefore we can recursively subscribe to <tt class="docutils literal">/bin</tt>, <tt class="docutils literal">/usr/bin</tt>, and other directories along PATH.</p>
<p>This gets us practically real-time notifications whenever a process is executed (<tt class="docutils literal">OPEN</tt> event) and terminated (<tt class="docutils literal">CLOSE_NOWRITE</tt> event). If you need to filter for noise from other file open events, you can just set further subscriptions/watches for some libraries and correlate the accesses to those.</p>
<p>The sequence of binaries executed is not the only type of interesting data we can glean via inotify. Especially if the system uses textual log files, it's often useful to subscribe to all write events to e.g. the authentication log. We can then keep track of 'last seen size' for the log file and then derive the size of individual log writes by looking at the delta of the file size.</p>
</div>
<div class="section" id="putting-it-all-together">
<h2>Putting it all together</h2>
<p>All of the above signals can be correlated via time if the system is quite passive - heavier load would drown out the signal of which PIDs and <tt class="docutils literal">inotify</tt> events are connected. As an example, here's an example log transcript from an example tool that does the inotify subscriptions and polls the various syscalls described above on each event:</p>
<pre class="code text literal-block">
15:40:42 Observed new PID 6824 (sid=6824)
15:40:42 Opening: bash
15:40:43 Observed new Session ID 6824
15:40:46 Observed new PID 6825 (ROOT!, sid=6824)
15:40:46 Opening: sudo
15:40:47 Auth log size increased by 153 bytes
15:40:55 No longer seeing PID 6825 ()
15:40:55 Auth log size increased by 161 bytes
15:40:55 Closing: sudo
15:40:56 Observed new PID 6828 (ROOT!, sid=6824)
15:40:56 Opening: sudo
15:40:58 Observed new PID 6829 (ROOT!, sid=6824) 6830 ()
15:40:58 No longer seeing PID 6822 ()
15:40:58 Observed new processes with UID 123
15:40:58 Auth log size increased by 231 bytes
15:40:58 Observed new PID 6832 (sid=6824)
15:40:58 No longer seeing PID 6830 ()
15:40:58 Opening: apt-get
15:40:58 Opening: dpkg
15:40:58 Closing: dpkg
15:41:02 Observed new PID 6834 (sid=6824) 6835 ()
15:41:02 No longer seeing PID 6835 ()
15:41:02 Opening: apt-key
15:41:02 Opening: dash
15:41:02 Closing: apt-key
... snip ...
15:41:03 Opening: gpgv
15:41:03 Closing: gpgv
15:41:03 Opening: rm
15:41:03 Closing: rm
15:41:03 Closing: apt-key
15:41:03 Closing: dash
15:41:04 Observed new PID 6952 () 6953 () 6957 ()
15:41:04 No longer seeing processes with UID 123
15:41:04 Opening: apt-key
15:41:04 No longer seeing PID 6832 () 6952 () 6834 () 6957 () 6953 ()
15:41:04 Opening: dash
15:41:04 Closing: apt-key
15:41:04 Opening: apt-key
... snip ...
15:41:04 Closing: dash
15:41:04 Closing: dash
15:41:04 Opening: dpkg
15:41:04 Closing: dpkg
15:41:05 Auth log size increased by 87 bytes
15:41:05 Opening: dpkg
15:41:05 No longer seeing PID 6828 () 6829 ()
15:41:05 Closing: dpkg
15:41:05 Closing: apt-get
15:41:05 Closing: sudo
15:41:07 No longer seeing PID 6824 ()
15:41:07 No longer seeing Session ID 6824
15:41:07 Closing: bash
</pre>
<dl class="docutils">
<dt>Although we don't get anything resembling a full command line, we can piece together various facts:</dt>
<dd><ul class="first last simple">
<li>An interactive user tried to <tt class="docutils literal">sudo</tt> (which is a set-uid binary)</li>
<li>They mistyped the password a few times and ran <tt class="docutils literal">sudo</tt> again</li>
<li>After succeeding at <tt class="docutils literal">15:40:58</tt>, <tt class="docutils literal"><span class="pre">apt-get</span></tt> is run (it was very likely passed to the sudo command)</li>
<li>Something (likely <tt class="docutils literal"><span class="pre">apt-get</span></tt>) dropped to UID 123 (= <tt class="docutils literal">apt</tt> on this system)</li>
<li>The sudo command finished at <tt class="docutils literal">15:41:05</tt> and the shell/session finished <tt class="docutils literal">15:41:07</tt></li>
</ul>
</dd>
</dl>
<p>Not quite equivalent to <tt class="docutils literal">ps</tt> or <tt class="docutils literal">top <span class="pre">-b</span></tt>, but still quite an improvement over the complete blindness standard tools would leave us with!</p>
</div>