<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title><![CDATA[0xpebbles.org]]></title>
    <link><![CDATA[http://blog.0xpebbles.org]]></link>
    <description><![CDATA[0xpebbles blog]]></description>
    <lastBuildDate>Fri, 09 Apr 2021 08:37:23 +0200</lastBuildDate>
    <pubDate>Fri, 09 Apr 2021 08:37:23 +0200</pubDate>
    <language>en</language>

<!-- 20200911 -->    <item>
      <title>Simple OpenSMTPD filter example in awk</title>
      <link>http://blog.0xpebbles.org/Simple-OpenSMTPD-filter-example-in-awk</link>
      <pubDate>11 Sep 2020 00:00:00 +0000</pubDate>
      <content:encoded><![CDATA[

  

<p>Turns out that <i>awk</i> lends itself very nicely to writing OpenSMTPD filters with its
line-based filter protocol. Below is a simple example of how to implement a simple DNSBL check.</p>

<p><b>Note</b>, the snippet below is an example, only, and has some shortcomings for simplicity:</p>
<ul>
<li><b>only</b> handles IPv4 addresses</li>
<li><b>does not do any version detection of the filter protocol</b>, and doesn't work on filter protocol &lt; 0.6 (which was used in OpenSMTPD &lt; 6.7; see comments in the script for details)</li>
<li>SMTP error returned on blacklisting is hardcoded, but should be configurable</li>
<li>has hardcoded logging of every filter action, people might want to silence it</li>
</ul>

<style>
<!--
pre.awk { font-family: monospace; color: #b2b2b2; background-color: #000000; }
.String { color: #8787d7; }
.Comment { color: #626262; }
.Constant { color: #87afff; font-weight: bold; }
.Special { color: #00d700; font-weight: bold; }
.Identifier { color: #ffd700; font-weight: bold; }
.Statement { color: #ff8700; font-weight: bold; }
-->
</style>

<pre class='awk'>
<span class="Comment">#!/usr/bin/awk -f</span>
<span class="Comment">#</span>
<span class="Comment"># Usage in smtpd.conf:</span>
<span class="Comment">#   filter &lt;filter-name&gt; proc-exec &quot;/path/to/filter-dnsbl &lt;resolve_cmd&gt; &lt;dnsbl&gt; &lt;ip_bl&gt;&quot;</span>
<span class="Comment">#</span>
<span class="Comment"># Where:</span>
<span class="Comment"># - &lt;resolve_cmd&gt; is a string used to resolve the DNSBL query, returning</span>
<span class="Comment">#   only the response, use %s for assembled request, escape % with %%, e.g.:</span>
<span class="Comment">#     &quot;dig +short %s&quot;</span>
<span class="Comment">#     &quot;host -t A %s | sed 's/^.*has address //'&quot;</span>
<span class="Comment"># - &lt;dnsbl&gt; is the DNSBL address-suffix to look up, e.g.:</span>
<span class="Comment">#     &quot;ix.dnsbl.manitu.net&quot;</span>
<span class="Comment"># - &lt;ip_bl&gt; is a regex, if IP(s) returned by the lookup match they are</span>
<span class="Comment">#   considered &quot;blacklisted&quot;, e.g.:</span>
<span class="Comment">#     &quot;^127\.0\.0\.[234]$&quot;</span>
<span class="Comment">#</span>
<span class="Comment"># Examples (for smtpd.conf):</span>
<span class="Comment">#   filter dnsbl_nixspam proc-exec &quot;filter-dnsbl.awk \&quot;host -t A %s | sed 's/^.*has address //'\&quot; ix.dnsbl.manitu.net '^127\.0\.0\.2$'&quot;</span>
<span class="Comment">#   filter dnsbl_nixspam proc-exec &quot;filter-dnsbl.awk \&quot;dig +short %s A\&quot; bl.spamcop.net '^127\.0\.0\.2$'&quot;</span>

<span class="Special">BEGIN</span> {
    <span class="Statement">if</span> (<span class="Special">ARGC</span> <span class="Special">!</span><span class="Special">=</span> <span class="Constant">4</span>) {
        <span class="Statement">printf</span>(<span class="String">&quot;Error, 4 args expected, got </span><span class="Special">%d</span><span class="Special">\n</span><span class="String">&quot;</span><span class="Special">,</span> <span class="Special">ARGC</span>) &gt; <span class="String">&quot;/dev/stderr&quot;</span>
        <span class="Statement">exit</span> <span class="Constant">1</span>  <span class="Comment"># note, this will terminate smtpd</span>
    }
    RESOLVE_CMD <span class="Special">=</span> <span class="Special">ARGV</span>[<span class="Special">1</span>]
    DNSBL <span class="Special">=</span> <span class="Special">ARGV</span>[<span class="Special">2</span>]
    IP_BL <span class="Special">=</span> <span class="Special">ARGV</span>[<span class="Special">3</span>]
    <span class="Special">ARGC</span> <span class="Special">=</span> <span class="Constant">0</span> <span class="Comment"># no more input args / files</span>
    <span class="Special">FS</span> <span class="Special">=</span> <span class="String">&quot;|&quot;</span>
}

<span class="String">&quot;config|ready&quot;</span> <span class="Special">==</span> <span class="Special">$0</span> {
    <span class="Statement">print</span>(<span class="String">&quot;register|filter|smtp-in|connect&quot;</span>) &gt; <span class="String">&quot;/dev/stdin&quot;</span>
    <span class="Statement">print</span>(<span class="String">&quot;register|ready&quot;</span>) &gt; <span class="String">&quot;/dev/stdin&quot;</span>
    <span class="Statement">next</span> <span class="Comment"># don't exit as this will stop smtpd</span>
}
<span class="String">&quot;filter&quot;</span> <span class="Special">==</span> <span class="Special">$1</span> {
    <span class="Statement">if</span> (<span class="Special">NF</span> &lt; <span class="Constant">9</span>) {
        <span class="Statement">printf</span>(<span class="String">&quot;Error, filter line not having enough fields, 9+ expected, got </span><span class="Special">%d</span><span class="Special">\n</span><span class="String">&quot;</span><span class="Special">,</span> <span class="Special">NF</span>) &gt; <span class="String">&quot;/dev/stderr&quot;</span>
        <span class="Statement">next</span> <span class="Comment"># don't exit as this will stop smtpd</span>
    }
    sess_id <span class="Special">=</span> <span class="Special">$6</span>
    resp_token <span class="Special">=</span> <span class="Special">$7</span>
    <span class="Comment"># reverse on the fly and trim port</span>
    <span class="Comment"># !NOTE!: with version &lt; 0.6 (e.g. $2 == &quot;0.5&quot;) the connecting ip is in $10 - not handling version detection, here</span>
    <span class="Identifier">split</span>(<span class="Special">$9</span><span class="Special">,</span> x<span class="Special">,</span> <span class="String">&quot;[.:]&quot;</span>)
    req <span class="Special">=</span> x[<span class="Special">4</span>]<span class="String">&quot;.&quot;</span>x[<span class="Special">3</span>]<span class="String">&quot;.&quot;</span>x[<span class="Special">2</span>]<span class="String">&quot;.&quot;</span>x[<span class="Special">1</span>]<span class="String">&quot;.&quot;</span>DNSBL  <span class="Comment"># !NOTE!: works only with ipv4</span>

    ret <span class="Special">=</span> <span class="String">&quot;proceed&quot;</span>
    cmd <span class="Special">=</span> <span class="Identifier">sprintf</span>(RESOLVE_CMD<span class="Special">,</span> req)
    <span class="Statement">while</span>((cmd | <span class="Statement">getline</span> r) &gt; <span class="Constant">0</span>) {
        <span class="Statement">if</span>(r <span class="Special">~</span> IP_BL) {
            ret <span class="Special">=</span> <span class="String">&quot;reject|550 connecting server is blacklisted&quot;</span>
            <span class="Statement">break</span>
        }
    }
    <span class="Identifier">close</span>(cmd)

    <span class="Statement">print</span>(sess_id<span class="String">&quot; DNSBL check: &quot;</span><span class="Special">$8</span><span class="String">&quot; [&quot;</span><span class="Special">$9</span><span class="String">&quot;]: &quot;</span>cmd<span class="String">&quot; =&gt; &quot;</span>ret) &gt; <span class="String">&quot;/dev/stderr&quot;</span>
    <span class="Statement">print</span>(<span class="String">&quot;filter-result|&quot;</span>sess_id<span class="String">&quot;|&quot;</span>resp_token<span class="String">&quot;|&quot;</span>ret) &gt; <span class="String">&quot;/dev/stdin&quot;</span>
}
</pre>


]]></content:encoded>
    </item>

<!-- 20150911 -->    <item>
      <title>File rescue with dd and gawk</title>
      <link>http://blog.0xpebbles.org/File-rescue-with-dd-and-gawk</link>
      <pubDate>11 Sep 2015 00:00:00 +0000</pubDate>
      <content:encoded><![CDATA[

  

<p>
I recently had to undelete some accidentally deleted pictures on some SD card, after the owner of it was trying
out different tools, and even brought it to some computer store (which tried more tools), but was only able to
recover half of them. It was clear that the files have been deleted, only, but not overwritten, as he noticed
his mistake immediately and refrained from using the card afterwards. The way default deletion usually works,
means that pretty much everything still had to be recoverable.
</p>

<p>
Turns out it was, and even without any rescue tool. When I started looking into it, the first tool I saw in the
FreeBSD ports was <em>magicrescue</em>, but somehow no matter what I tried, it always exited with the same error.
Looking at the man-page I noticed right at the beginning:
</p>

<pre>
It looks at "magic bytes" in file contents, [...] It works on any file system,
but on very fragmented file systems it can only recover the first chunk of each
file.  These chunks are sometimes as big as 50MB, however.
</pre>

<p>
So, the tool is file-system agnostic, it seems to only look for some sequence of bytes and then to recover some
sequential number of bytes. This also means that very complex file-systems or features like compression and
deduplication will obviously not be suited for recovery with <em>magicrescue</em>.
</p>

<p>
Makes sense. And that applies also to my case: the card had a FAT32 file system on it (like probably most cameras
use), meaning there <b>won't</b> be any fancy file system features. Also, given that a camera stores one picture after
the other (and if people delete some it's often always the last right after taking it), there probably also is
<b>little fragmentation</b>.
</p>

<p>
So, basically, all I need to do is read all bytes off of the card, and split on certain patterns. <em>split(1)</em>
unfortunately doesn't help, as although you can use a pattern for splitting, it's only matching on entire lines.<br>
Inspecting the first few megabytes on the SD card revealed, that the images on there are stored as Exif-JPEG files
(starting with magic numbers 0xff 0xd8, and then 0xff 0xe1 for this subtype, details <a href="http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf">here</a>).
This is not something general purpose, of course. And even for this one type of JPEG file not something to rely on,
but I didn't want to split on 0xff 0xd8, only (to keep false positives low), and assumed that the camera wrote all
images in the same format/way.
</p>

<p>
Completely ignoring the end-markers of JPEG files, accepting that the recovered images might have some garbage data
appended, I started splitting the data on the SD card up on those 4 byte patterns. And that works quite nicely with
<em>dd</em> and <em>gawk</em> (note, POSIX <em>awk</em> won't work, as the record separator can only be one byte):
</p>

<style type="text/css">
<!--
pre.sh { font-family: monospace; color: #dadada; background-color: #000000; }
.shShellVariables { color: #87afff; }
.Comment { color: #626262; }
.Constant { color: #87afff; font-weight: bold; }
.Identifier { color: #ffd700; }
.Statement { color: #ff8700; font-weight: bold; }
.String { color: #8787ff; }
.Statement { color: #ff8700; font-weight: bold; }
-->
</style>
<pre class="sh">
dd <span class="Identifier">if</span>=<span class="shShellVariables">$SRC</span> <span class="Identifier">of</span>=/dev/stdout <span class="Identifier">bs</span>=1M | <span class="Statement">\</span>
  gawk <span class="Statement">'</span><span class="String">BEGIN { FS=&quot;fs is not important&quot;; RS=&quot;\xff\xd8\xff\xe1&quot; } { print RS$0 &gt; sprintf(&quot;%04d.jpg&quot;, NR) }</span><span class="Statement">'</span>
</pre>

<p>
Of course, set <b>$SRC</b> to the device you want to recover your files from.
</p>

<p>
That's it - I was able to recover every single image off of that card, with a shell one-liner! Of course, this is a
specific case that made this possible: simple file system, no fragmentation, only JPEG files to recover, and only
one JPEG type to look for, etc., but it can easily extended to suit other purposes.
</p>

<p>
Here's a little bit more convenient version as a shell script, allowing to seek, set the size to recover, and an optional
prefix for the recovered images (still only looking for the same 4 bytes to separate on, though):
</p>

<pre class="sh">
<span class="Comment">#!/bin/sh</span>
<span class="Statement">if </span><span class="Statement">[</span> <span class="shShellVariables">$#</span> <span class="Statement">-lt</span> <span class="Constant">3</span> <span class="Statement">]</span><span class="Statement">;</span> <span class="Statement">then</span> <span class="Statement">echo</span><span class="String"> Usage: </span><span class="shShellVariables">$0</span><span class="String"> DEV SIZE_MB SEEK_MB OUT_PREFIX</span><span class="Statement">;</span> <span class="Statement">exit</span><span class="Statement">;</span> <span class="Statement">fi</span>
dd <span class="Identifier">if</span>=<span class="shShellVariables">$1</span> <span class="Identifier">of</span>=/dev/stdout <span class="Identifier">bs</span>=1M <span class="Identifier">count</span>=<span class="shShellVariables">$2</span> <span class="Identifier">iseek</span>=<span class="shShellVariables">$3</span> | gawk <span class="Statement">'</span><span class="String">BEGIN { FS=&quot;fs is not important&quot;; RS=&quot;\xff\xd8\xff\xe1&quot; } { print RS$0 &gt; sprintf(&quot;</span><span class="Statement">'</span><span class="shShellVariables">$4</span><span class="Statement">'</span><span class="String">%04d.jpg&quot;, NR) }</span><span class="Statement">'</span>
</pre>

]]></content:encoded>
    </item>




  </channel>
</rss>

