<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title><![CDATA[0xpebbles.org]]></title>
    <link><![CDATA[http://blog.0xpebbles.org]]></link>
    <description><![CDATA[0xpebbles blog]]></description>
    <lastBuildDate>Fri, 09 Apr 2021 08:37:23 +0200</lastBuildDate>
    <pubDate>Fri, 09 Apr 2021 08:37:23 +0200</pubDate>
    <language>en</language>

<!-- 20150319 -->    <item>
      <title>shell script for S3-upload via curl using AWS version 4 signatures</title>
      <link>http://blog.0xpebbles.org/shell-script-for-S3-upload-via-curl-using-AWS-version-4-signatures</link>
      <pubDate>19 Mar 2015 00:00:00 +0000</pubDate>
      <content:encoded><![CDATA[

  

<p>
This is a more modern version of <a href="/shell-script-to-upload-to-S3-via-curl">this script</a>, switching
to AWS version 4 signatures that are mandatory for AWS regions created after January 2014. It also works with
older regions as they seem to support the new signature format, as well.
</p>

<p>
The script's interface is a bit easier and more intuitive, too, and allows setting the access permissions, now. See the script's
own help-text for information and examples on how to use it.
</p>

<style type="text/css">
<!--
pre.sh { font-family: monospace; color: #dadada; background-color: #000000; }
.Comment { color: #626262; }
.Constant { color: #87afff; font-weight: bold; }
.Special { color: #00d700; font-weight: bold; }
.Identifier { color: #ffd700; }
.Statement { color: #ff8700; font-weight: bold; }
.PreProc { color: #008787; }
.String { color: #8787ff; }
.shShellVariables { color: #87afff; }
-->
</style>
<pre class="sh">
<span class="Comment">#!/bin/sh</span>

<span class="Identifier">usage()</span>
<span class="Identifier">{</span>
    <span class="Statement">cat</span> <span class="Statement">&lt;&lt;USAGE</span>

<span class="String">Simple script uploading a file to S3. Supports AWS signature version 4, custom</span>
<span class="String">region, permissions and mime-types. Uses Content-MD5 header to guarantee</span>
<span class="String">uncorrupted file transfer.</span>

<span class="String">Usage:</span>
<span class="String">  </span><span class="Special">`basename </span><span class="shShellVariables">$0</span><span class="Special">`</span><span class="String"> aws_ak aws_sk bucket srcfile targfile [acl] [mime_type]</span>

<span class="String">Where &lt;arg&gt; is one of:</span>
<span class="String">  aws_ak     access key ('' for upload to public writable bucket)</span>
<span class="String">  aws_sk     secret key ('' for upload to public writable bucket)</span>
<span class="String">  bucket     bucket name (with optional @region suffix, default is us-east-1)</span>
<span class="String">  srcfile    path to source file</span>
<span class="String">  targfile   path to target (dir if it ends with '/', relative to bucket root)</span>
<span class="String">  acl        s3 access permissions (default: public-read)</span>
<span class="String">  mime_type  optional mime-type (tries to guess if omitted)</span>

<span class="String">Dependencies:</span>
<span class="String">  To run, this shell script depends on command-line curl and openssl, as well</span>
<span class="String">  as standard Unix tools</span>

<span class="String">Examples:</span>
<span class="String">  To upload file '~/blog/media/image.png' to bucket 'storage' in region</span>
<span class="String">  'eu-central-1' with key (path relative to bucket) 'media/image.png':</span>

<span class="String">    </span><span class="Special">`basename </span><span class="shShellVariables">$0</span><span class="Special">`</span><span class="String"> ACCESS SECRET storage@eu-central-1 </span><span class="Special">\\</span>
<span class="String">      ~/blog/image.png media/</span>

<span class="String">  To upload file '~/blog/media/image.png' to public-writable bucket 'storage'</span>
<span class="String">  in default region 'us-east-1' with key (path relative to bucket) 'x/y.png':</span>

<span class="String">    </span><span class="Special">`basename </span><span class="shShellVariables">$0</span><span class="Special">`</span><span class="String"> '' '' storage ~/blog/image.png x/y.png</span>

<span class="Statement">USAGE</span>
    <span class="Statement">exit</span> <span class="Constant">0</span>
<span class="Identifier">}</span>

<span class="Identifier">guessmime()</span>
<span class="Identifier">{</span>
    <span class="Identifier">mime</span>=<span class="Special">`file </span><span class="Special">-b</span><span class="Special"> </span><span class="Special">--mime-type</span><span class="Special"> </span><span class="shShellVariables">$1</span><span class="Special">`</span>
    <span class="Statement">if </span><span class="Statement">[</span> <span class="Statement">&quot;</span><span class="shShellVariables">$mime</span><span class="Statement">&quot;</span> <span class="Statement">=</span> <span class="String">&quot;text/plain&quot;</span> <span class="Statement">]</span>; <span class="Statement">then</span>
        <span class="Statement">case</span> <span class="shShellVariables">$1</span> <span class="Statement">in</span>
            *.css<span class="Statement">)</span>           <span class="Identifier">mime</span>=text/css<span class="Statement">;;</span>
            *.ttf<span class="Statement">|</span>*.otf<span class="Statement">)</span>     <span class="Identifier">mime</span>=application/font-sfnt<span class="Statement">;;</span>
            *.woff<span class="Statement">)</span>          <span class="Identifier">mime</span>=application/font-woff<span class="Statement">;;</span>
            *.woff2<span class="Statement">)</span>         <span class="Identifier">mime</span>=font/woff<span class="Constant">2</span><span class="Statement">;;</span>
            *rss*.xml<span class="Statement">|</span>*.rss<span class="Statement">)</span> <span class="Identifier">mime</span>=application/rss+xml<span class="Statement">;;</span>
            *<span class="Statement">)</span>               <span class="Statement">if </span>head <span class="shShellVariables">$1</span> | <span class="Statement">grep</span> <span class="Statement">'</span><span class="String">&lt;html.*&gt;</span><span class="Statement">'</span> <span class="Statement">&gt;</span>/dev/null; <span class="Statement">then</span> <span class="Identifier">mime</span>=text/html; <span class="Statement">fi</span><span class="Statement">;;</span>
        <span class="Statement">esac</span>
    <span class="Statement">fi</span>
    <span class="Statement">printf</span> <span class="Statement">&quot;</span><span class="shShellVariables">$mime</span><span class="Statement">&quot;</span>
<span class="Identifier">}</span>

<span class="Statement">if </span><span class="Statement">[</span> <span class="shShellVariables">$#</span> <span class="Statement">-lt</span> <span class="Constant">5</span> <span class="Statement">]</span>; <span class="Statement">then</span> usage; <span class="Statement">fi</span>

<span class="Comment"># Inputs.</span>
<span class="Identifier">aws_ak</span>=<span class="Statement">&quot;</span><span class="shShellVariables">$1</span><span class="Statement">&quot;</span>                                                              <span class="Comment"># access key</span>
<span class="Identifier">aws_sk</span>=<span class="Statement">&quot;</span><span class="shShellVariables">$2</span><span class="Statement">&quot;</span>                                                              <span class="Comment"># secret key</span>
<span class="Identifier">bucket</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="shShellVariables">$3</span><span class="Special"> | awk </span><span class="Statement">'</span><span class="String">BEGIN{FS=&quot;@&quot;}{print $1}</span><span class="Statement">'</span><span class="Special">`</span>                       <span class="Comment"># bucket name</span>
<span class="Identifier">region</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="shShellVariables">$3</span><span class="Special"> | awk </span><span class="Statement">'</span><span class="String">BEGIN{FS=&quot;@&quot;}{print ($2==&quot;&quot;?&quot;us-east-1&quot;:$2)}</span><span class="Statement">'</span><span class="Special">`</span>  <span class="Comment"># region name</span>
<span class="Identifier">srcfile</span>=<span class="Statement">&quot;</span><span class="shShellVariables">$4</span><span class="Statement">&quot;</span>                                                             <span class="Comment"># source file</span>
<span class="Identifier">targfile</span>=<span class="Special">`</span><span class="Statement">echo</span><span class="String"> -n </span><span class="Statement">&quot;</span><span class="shShellVariables">$5</span><span class="Statement">&quot;</span><span class="String"> </span><span class="Statement">|</span><span class="Special"> </span><span class="Statement">sed</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="String">s/</span><span class="Special">\/</span><span class="String">$/</span><span class="Special">\/</span><span class="shShellVariables">$(</span><span class="Special">basename </span><span class="shShellVariables">$srcfile</span><span class="shShellVariables">)</span><span class="String">/</span><span class="Statement">&quot;</span><span class="Special">`</span>            <span class="Comment"># target file</span>
<span class="Identifier">acl</span>=<span class="PreProc">${</span><span class="shShellVariables">6</span><span class="Statement">:-</span><span class="Statement">'</span><span class="String">public-read</span><span class="Statement">'</span><span class="PreProc">}</span>                                                  <span class="Comment"># s3 perms</span>
<span class="Identifier">mime</span>=<span class="PreProc">${</span><span class="shShellVariables">7</span><span class="Statement">:-</span><span class="Statement">&quot;</span><span class="Special">`guessmime </span><span class="Statement">&quot;</span><span class="shShellVariables">$srcfile</span><span class="Statement">&quot;</span><span class="Special">`</span><span class="Statement">&quot;</span><span class="PreProc">}</span>                                      <span class="Comment"># mime type</span>
<span class="Identifier">md5</span>=<span class="Special">`openssl md5 </span><span class="Special">-binary</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="shShellVariables">$srcfile</span><span class="Statement">&quot;</span><span class="Special"> | openssl base64`</span>


<span class="Comment"># Create signature if not public upload.</span>
<span class="Identifier">key_and_sig_args</span>=<span class="Statement">''</span>
<span class="Statement">if </span><span class="Statement">[</span> <span class="Statement">&quot;</span><span class="shShellVariables">$aws_ak</span><span class="Statement">&quot;</span> <span class="Statement">!=</span> <span class="Statement">&quot;&quot;</span> <span class="Statement">]</span> &amp;&amp; <span class="Statement">[</span> <span class="Statement">&quot;</span><span class="shShellVariables">$aws_sk</span><span class="Statement">&quot;</span> <span class="Statement">!=</span> <span class="Statement">&quot;&quot;</span> <span class="Statement">]</span>; <span class="Statement">then</span>

    <span class="Comment"># Need current and file upload expiration date. Handle GNU and BSD date command style to get tomorrow's date.</span>
    <span class="Identifier">date</span>=<span class="Special">`date </span><span class="Special">-u</span><span class="Special"> +%Y%m%dT%H%M%SZ`</span>
    <span class="Identifier">expdate</span>=<span class="Special">`if ! date </span><span class="Special">-v</span><span class="Special">+1d +%Y-%m-%d </span><span class="Constant">2</span><span class="Special">&gt;/dev/null; then date </span><span class="Special">-d</span><span class="Special"> tomorrow +%Y-%m-%d; fi`</span>
    <span class="Identifier">expdate_s</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="shShellVariables">$expdate</span><span class="Special"> | </span><span class="Statement">sed</span><span class="Special"> s/-//g`</span> <span class="Comment"># without dashes, as we need both formats below</span>
    <span class="Identifier">service</span>=<span class="Statement">'</span><span class="String">s3</span><span class="Statement">'</span>

    <span class="Comment"># Generate policy and sign with secret key following AWS Signature version 4, below</span>
    <span class="Identifier">p</span>=<span class="shShellVariables">$(</span><span class="Statement">cat</span><span class="Special"> &lt;&lt;POLICY | openssl base64</span>
<span class="Special">{ </span><span class="Statement">&quot;</span><span class="String">expiration</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="PreProc">${</span><span class="shShellVariables">expdate</span><span class="PreProc">}</span><span class="String">T12:00:00.000Z</span><span class="Statement">&quot;</span><span class="Special">,</span>
<span class="Special">  </span><span class="Statement">&quot;</span><span class="String">conditions</span><span class="Statement">&quot;</span><span class="Special">: [</span>
<span class="Special">    {</span><span class="Statement">&quot;</span><span class="String">acl</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="shShellVariables">$acl</span><span class="Statement">&quot;</span><span class="Special"> },</span>
<span class="Special">    {</span><span class="Statement">&quot;</span><span class="String">bucket</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="shShellVariables">$bucket</span><span class="Statement">&quot;</span><span class="Special"> },</span>
<span class="Special">    [</span><span class="Statement">&quot;</span><span class="String">starts-with</span><span class="Statement">&quot;</span><span class="Special">, </span><span class="Statement">&quot;</span><span class="Special">\$</span><span class="String">key</span><span class="Statement">&quot;</span><span class="Special">, </span><span class="Statement">&quot;&quot;</span><span class="Special">],</span>
<span class="Special">    [</span><span class="Statement">&quot;</span><span class="String">starts-with</span><span class="Statement">&quot;</span><span class="Special">, </span><span class="Statement">&quot;</span><span class="Special">\$</span><span class="String">content-type</span><span class="Statement">&quot;</span><span class="Special">, </span><span class="Statement">&quot;&quot;</span><span class="Special">],</span>
<span class="Special">    [</span><span class="Statement">&quot;</span><span class="String">content-length-range</span><span class="Statement">&quot;</span><span class="Special">, </span><span class="Constant">1</span><span class="Special">, `</span><span class="Statement">ls</span><span class="Special"> </span><span class="Special">-l</span><span class="Special"> </span><span class="Special">-H</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="shShellVariables">$srcfile</span><span class="Statement">&quot;</span><span class="Special"> | awk </span><span class="Statement">'</span><span class="String">{print $5}</span><span class="Statement">'</span><span class="Special"> | head </span><span class="Constant">-1</span><span class="Special">`],</span>
<span class="Special">    {</span><span class="Statement">&quot;</span><span class="String">content-md5</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="shShellVariables">$md5</span><span class="Statement">&quot;</span><span class="Special"> },</span>
<span class="Special">    {</span><span class="Statement">&quot;</span><span class="String">x-amz-date</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="shShellVariables">$date</span><span class="Statement">&quot;</span><span class="Special"> },</span>
<span class="Special">    {</span><span class="Statement">&quot;</span><span class="String">x-amz-credential</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="shShellVariables">$aws_ak</span><span class="String">/</span><span class="shShellVariables">$expdate_s</span><span class="String">/</span><span class="shShellVariables">$region</span><span class="String">/</span><span class="shShellVariables">$service</span><span class="String">/aws4_request</span><span class="Statement">&quot;</span><span class="Special"> },</span>
<span class="Special">    {</span><span class="Statement">&quot;</span><span class="String">x-amz-algorithm</span><span class="Statement">&quot;</span><span class="Special">: </span><span class="Statement">&quot;</span><span class="String">AWS4-HMAC-SHA256</span><span class="Statement">&quot;</span><span class="Special"> }</span>
<span class="Special">  ]</span>
<span class="Special">}</span>
<span class="Special">POLICY</span>
<span class="Special">    </span><span class="shShellVariables">)</span>

    <span class="Comment"># AWS4-HMAC-SHA256 signature</span>
    <span class="Identifier">s</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="shShellVariables">$expdate_s</span><span class="Statement">&quot;</span><span class="Special">   | openssl sha256 </span><span class="Special">-hmac</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="String">AWS4</span><span class="shShellVariables">$aws_sk</span><span class="Statement">&quot;</span><span class="Special">           </span><span class="Special">-hex</span><span class="Special"> | </span><span class="Statement">sed</span><span class="Special"> </span><span class="Statement">'</span><span class="String">s/(stdin)= //</span><span class="Statement">'</span><span class="Special">`</span>
    <span class="Identifier">s</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="shShellVariables">$region</span><span class="Statement">&quot;</span><span class="Special">      | openssl sha256 </span><span class="Special">-mac</span><span class="Special"> HMAC </span><span class="Special">-macopt</span><span class="Special"> hexkey:</span><span class="Statement">&quot;</span><span class="shShellVariables">$s</span><span class="Statement">&quot;</span><span class="Special"> </span><span class="Special">-hex</span><span class="Special"> | </span><span class="Statement">sed</span><span class="Special"> </span><span class="Statement">'</span><span class="String">s/(stdin)= //</span><span class="Statement">'</span><span class="Special">`</span>
    <span class="Identifier">s</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="shShellVariables">$service</span><span class="Statement">&quot;</span><span class="Special">     | openssl sha256 </span><span class="Special">-mac</span><span class="Special"> HMAC </span><span class="Special">-macopt</span><span class="Special"> hexkey:</span><span class="Statement">&quot;</span><span class="shShellVariables">$s</span><span class="Statement">&quot;</span><span class="Special"> </span><span class="Special">-hex</span><span class="Special"> | </span><span class="Statement">sed</span><span class="Special"> </span><span class="Statement">'</span><span class="String">s/(stdin)= //</span><span class="Statement">'</span><span class="Special">`</span>
    <span class="Identifier">s</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="String">aws4_request</span><span class="Statement">&quot;</span><span class="Special"> | openssl sha256 </span><span class="Special">-mac</span><span class="Special"> HMAC </span><span class="Special">-macopt</span><span class="Special"> hexkey:</span><span class="Statement">&quot;</span><span class="shShellVariables">$s</span><span class="Statement">&quot;</span><span class="Special"> </span><span class="Special">-hex</span><span class="Special"> | </span><span class="Statement">sed</span><span class="Special"> </span><span class="Statement">'</span><span class="String">s/(stdin)= //</span><span class="Statement">'</span><span class="Special">`</span>
    <span class="Identifier">s</span>=<span class="Special">`</span><span class="Statement">printf</span><span class="Special"> </span><span class="Statement">&quot;</span><span class="shShellVariables">$p</span><span class="Statement">&quot;</span><span class="Special">           | openssl sha256 </span><span class="Special">-mac</span><span class="Special"> HMAC </span><span class="Special">-macopt</span><span class="Special"> hexkey:</span><span class="Statement">&quot;</span><span class="shShellVariables">$s</span><span class="Statement">&quot;</span><span class="Special"> </span><span class="Special">-hex</span><span class="Special"> | </span><span class="Statement">sed</span><span class="Special"> </span><span class="Statement">'</span><span class="String">s/(stdin)= //</span><span class="Statement">'</span><span class="Special">`</span>

    <span class="Identifier">key_and_sig_args</span>=<span class="Statement">&quot;</span><span class="String">-F X-Amz-Credential=</span><span class="shShellVariables">$aws_ak</span><span class="String">/</span><span class="shShellVariables">$expdate_s</span><span class="String">/</span><span class="shShellVariables">$region</span><span class="String">/</span><span class="shShellVariables">$service</span><span class="String">/aws4_request -F X-Amz-Algorithm=AWS4-HMAC-SHA256 -F X-Amz-Signature=</span><span class="shShellVariables">$s</span><span class="String"> -F X-Amz-Date=</span><span class="PreProc">${</span><span class="shShellVariables">date</span><span class="PreProc">}</span><span class="Statement">&quot;</span>
<span class="Statement">fi</span>


<span class="Comment"># Upload. Supports anonymous upload if bucket is public-writable, and keys are set to ''.</span>
<span class="Statement">echo</span><span class="String"> </span><span class="Statement">&quot;</span><span class="String">Uploading: </span><span class="shShellVariables">$srcfile</span><span class="String"> (</span><span class="shShellVariables">$mime</span><span class="String">) to </span><span class="shShellVariables">$bucket</span><span class="String">:</span><span class="shShellVariables">$targfile</span><span class="Statement">&quot;</span>
curl                            <span class="Statement">\</span>
    <span class="Special">-#</span> <span class="Special">-k</span>                       <span class="Statement">\</span>
    <span class="Special">-F</span> <span class="Identifier">key</span>=<span class="shShellVariables">$targfile</span>            <span class="Statement">\</span>
    <span class="Special">-F</span> <span class="Identifier">acl</span>=<span class="shShellVariables">$acl</span>                 <span class="Statement">\</span>
    <span class="shShellVariables">$key_and_sig_args</span>           <span class="Statement">\</span>
    <span class="Special">-F</span> <span class="Statement">&quot;</span><span class="String">Policy=</span><span class="shShellVariables">$p</span><span class="Statement">&quot;</span>              <span class="Statement">\</span>
    <span class="Special">-F</span> <span class="Statement">&quot;</span><span class="String">Content-MD5=</span><span class="shShellVariables">$md5</span><span class="Statement">&quot;</span>       <span class="Statement">\</span>
    <span class="Special">-F</span> <span class="Statement">&quot;</span><span class="String">Content-Type=</span><span class="shShellVariables">$mime</span><span class="Statement">&quot;</span>     <span class="Statement">\</span>
    <span class="Special">-F</span> <span class="Statement">&quot;</span><span class="String">file=@</span><span class="shShellVariables">$srcfile</span><span class="Statement">&quot;</span>         <span class="Statement">\</span>
    https://<span class="PreProc">${</span><span class="shShellVariables">bucket</span><span class="PreProc">}</span>.s3.amazonaws.com/ | <span class="Statement">cat</span> <span class="Comment"># pipe through cat so curl displays upload progress bar, *and* response</span>

</pre>

]]></content:encoded>
    </item>




  </channel>
</rss>

