<![CDATA[0xpebbles.org]]> Fri, 09 Apr 2021 08:37:24 +0200 Fri, 09 Apr 2021 08:37:24 +0200 en Simple OpenSMTPD filter example in awk http://blog.0xpebbles.org/Simple-OpenSMTPD-filter-example-in-awk 11 Sep 2020 00:00:00 +0000 Turns out that awk lends itself very nicely to writing OpenSMTPD filters with its line-based filter protocol. Below is a simple example of how to implement a simple DNSBL check.

Note, the snippet below is an example, only, and has some shortcomings for simplicity:

  • only handles IPv4 addresses
  • does not do any version detection of the filter protocol, and doesn't work on filter protocol < 0.6 (which was used in OpenSMTPD < 6.7; see comments in the script for details)
  • SMTP error returned on blacklisting is hardcoded, but should be configurable
  • has hardcoded logging of every filter action, people might want to silence it
#!/usr/bin/awk -f
#
# Usage in smtpd.conf:
#   filter <filter-name> proc-exec "/path/to/filter-dnsbl <resolve_cmd> <dnsbl> <ip_bl>"
#
# Where:
# - <resolve_cmd> is a string used to resolve the DNSBL query, returning
#   only the response, use %s for assembled request, escape % with %%, e.g.:
#     "dig +short %s"
#     "host -t A %s | sed 's/^.*has address //'"
# - <dnsbl> is the DNSBL address-suffix to look up, e.g.:
#     "ix.dnsbl.manitu.net"
# - <ip_bl> is a regex, if IP(s) returned by the lookup match they are
#   considered "blacklisted", e.g.:
#     "^127\.0\.0\.[234]$"
#
# Examples (for smtpd.conf):
#   filter dnsbl_nixspam proc-exec "filter-dnsbl.awk \"host -t A %s | sed 's/^.*has address //'\" ix.dnsbl.manitu.net '^127\.0\.0\.2$'"
#   filter dnsbl_nixspam proc-exec "filter-dnsbl.awk \"dig +short %s A\" bl.spamcop.net '^127\.0\.0\.2$'"

BEGIN {
    if (ARGC != 4) {
        printf("Error, 4 args expected, got %d\n", ARGC) > "/dev/stderr"
        exit 1  # note, this will terminate smtpd
    }
    RESOLVE_CMD = ARGV[1]
    DNSBL = ARGV[2]
    IP_BL = ARGV[3]
    ARGC = 0 # no more input args / files
    FS = "|"
}

"config|ready" == $0 {
    print("register|filter|smtp-in|connect") > "/dev/stdin"
    print("register|ready") > "/dev/stdin"
    next # don't exit as this will stop smtpd
}
"filter" == $1 {
    if (NF < 9) {
        printf("Error, filter line not having enough fields, 9+ expected, got %d\n", NF) > "/dev/stderr"
        next # don't exit as this will stop smtpd
    }
    sess_id = $6
    resp_token = $7
    # reverse on the fly and trim port
    # !NOTE!: with version < 0.6 (e.g. $2 == "0.5") the connecting ip is in $10 - not handling version detection, here
    split($9, x, "[.:]")
    req = x[4]"."x[3]"."x[2]"."x[1]"."DNSBL  # !NOTE!: works only with ipv4

    ret = "proceed"
    cmd = sprintf(RESOLVE_CMD, req)
    while((cmd | getline r) > 0) {
        if(r ~ IP_BL) {
            ret = "reject|550 connecting server is blacklisted"
            break
        }
    }
    close(cmd)

    print(sess_id" DNSBL check: "$8" ["$9"]: "cmd" => "ret) > "/dev/stderr"
    print("filter-result|"sess_id"|"resp_token"|"ret) > "/dev/stdin"
}
]]> Global, static string->data associative array for a C program http://blog.0xpebbles.org/Static-associative-array-for-a-C-program 26 Jul 2020 00:00:00 +0000 I pondered the question whether there is a simple way to have a static (as in static lifetime) read-only associative array in C, with strings as keys, mapping to any kind of data, ideally without having to manually handle its lifetime. So, basically similar to what one would get with standard static, const arrays, like the following: 

// in some header
extern const int month_days[];

// in some translation unit
const int month_days[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };

Turns it it's not too hard, actually, at least on systems with dynamic linkers. We would need a dynamic symbol for each entry, allowing for lookups via dlsym(3), getting back a pointer to our data.

Staying with our example above, let's say we want to lookup the same by name: 

const int month_day_jan = 31;
const int month_day_feb = 28;
const int month_day_mar = 31;
const int month_day_apr = 30;
const int month_day_may = 31;
const int month_day_jun = 30;
const int month_day_jul = 31;
const int month_day_aug = 31;
const int month_day_sep = 30;
const int month_day_oct = 31;
const int month_day_nov = 30;
const int month_day_dec = 31;

Now when linking, we need to declare those symbols to be dynamic, this can be done for example with the --dynamic-list linker flag, and a file like the following: 

{
	month_day_jan;
	month_day_feb;
	month_day_mar;
	month_day_apr;
	month_day_may;
	month_day_jun;
	month_day_jul;
	month_day_aug;
	month_day_sep;
	month_day_oct;
	month_day_nov;
	month_day_dec;
};

Now we can do lookups like the following, for example: 

int x = *(int*)dlsym(NULL, "month_day_oct");

This is simple and straightforward, and doesn't come as a surprise - after all this is how object files are structured and how linkers work. However, I never considered (ab)using the dynamic linker as some sort of dynamic, associative array lookup equivalent.

The upsides are that the data is embedded, that its lifetime is static, that you can use strings as keys, that you don't have to do any memory management (like fill a map at startup and free it at the end), that dlsym(3) lookups are efficiently implemented (well, most likely at least), etc.

There are downsides, however:

  • the keys being symbol names are limited to alphanumeric characters and underscores, and cannot start with a number (some platforms might allow for other characters)
  • they are potentially name-mangled by the compiler (use objdump -t to check)
  • the names must be globally unique and are subject to name clashes with other unrelated symbols

Let's look at another example, embedding binary data directly, without using any C code, by also making sure that the data is in the .rodata section, and creating the dynamic symbol list from our object file: 

# with LLVM's ld you might need to pass -m explicitly, e.g. -m elf_amd64
ld -r -b binary -o bins.o file1.png folder_x/otherfile.txt
objcopy --rename-section .data=.rodata,contents,alloc,load,readonly bins.o
# generate dynamic symbol list
nm bins.o | awk 'BEGIN{print "{"}{print $3";"}END{print "};"}' > bins.symlst

This will actually create 3 symbols per file, with a filename-based symbol name and some prefix and suffixes, all pretty self explanatory: 

_binary_file1_png_end
_binary_file1_png_size
_binary_file1_png_start
_binary_folder_x_otherfile_txt_end
_binary_folder_x_otherfile_txt_size
_binary_folder_x_otherfile_txt_start
]]> PowerVR SGX on the BeagleBone Black in 2019 http://blog.0xpebbles.org/PowerVR-SGX-on-the-beaglebone-black-in-2019 17 Aug 2019 00:00:00 +0000 Since I spent way too much time on figuring this out, confused by plenty of old and outdated online sources, here is how one would get the PowerVR SGX driver running on a current Linux in 2019 - in this case a debian 9.6 with a 4.4 kernel. Other combinations might/should work also, but I have not tested them.

To summarize briefly, the difference to the previous ways of getting the SGX stuff running, is that the old GFX_SDK_* based setups are outdated. This includes omaplfb.ko kernel module based instructions, which also come from the GFX_SDK_* times. Last but not least, it looks like one has to run a TI kernel (updated to with flag --ti-kernel, see below), as the bone kernels (--bone-kernel) don't seem to have the new SGX driver stack.

I used bone-debian-9.6-console-armhf-2018-12-10-2gb.img.xz from RobertCNelson's site at https://rcn-ee.com/rootfs/2018-12-10/microsd/, as I wanted a leaner base system than what's available on https://beagleboard.org/latest-images.

To put it in a nutshell, we need to run a TI kernel, and install the matching SGX DDK, and ideally initialize it on startup. So on your system on the BBB the following should install the TI kernel (v4.4, which worked for me) - the below assumes that the default user debian of the image is used, so some sudo is needed occasionally: 

cd /opt/scripts/tools

# install TI kernel, this also installs the matching SGX drivers
git pull                                       # fetch latest revisions of kernels to upgrade to
sudo ./update_kernel.sh --ti-kernel --lts-4_4  # if branch not there anymore, make sure whatever branch is used contains a working pvrsrvkm.ko
sudo reboot

Now the SGX DDK, which we'll build from source: 

# some libs for later
sudo apt install -y libdrm-omap1 libgbm1

# get the SGX DDK for userland libraries, demos and tools needed to initialize PowerVR
# NOTE: the branch used below has to match kernel version, change if using different kernel above
cd ~
git clone -b ti-img-sgx/1.14.3699939_k4.4 git://git.ti.com/graphics/omap5-sgx-ddk-um-linux.git --depth=1
cd omap5-sgx-ddk-um-linux/
sudo env DISCIMAGE=/ TARGET_PRODUCT=ti335x make install
cd ~
rm -rf ./omap5-sgx-ddk-um-linux

# hackfix needed missing reference to libgbm.so.2
sudo ln -s /usr/lib/arm-linux-gnueabihf/libgbm.so.1 /usr/lib/arm-linux-gnueabihf/libgbm.so.2

# verify driver loaded - should show pvrsrvkm
lsmod | grep pvr

# verify, there should be some info like "[drm] Initialized" and "PVR_K: UM DDK..." infos
dmesg | grep -i -C1 'drm\|sgx\|pvr'

# set mode SGX should work in - at this point there are three installed, but
# only one that works, as the libpvrws_WAYLAND.so and libpvrGBMWSEGL.so are
# both for DRM/WAYLAND based multi-window systems and weston isn't installed.
# According to the docs there also should be libpvrDRMWSEGL_FRONT.so, a no-vsync
# and thus faster version of libpvrDRMWSEGL.so.
# Other options like pixel format can also be set to RGB565, RGB888 or ARGB8888,
# select the one that works for you (colors might be swapped for 565 or 888, depending
# on LCD wiring, see TI's AM335x silicon errata.
sudo rm /etc/powervr.ini
sudo tee -a /etc/powervr.ini <<EOF
[default]
WindowSystem=libpvrDRMWSEGL.so
DefaultPixelFormat=RGB888
EOF

# init powervr for testing, will be automated later, below
sudo /usr/bin/pvrsrvctl --start --no-module

# verify via tools shipped in the omap5-sgx-ddk:
cd /usr/bin   # b/c gles2test1 depending on that pwd for shader files in /usr/bin/
eglinfo       # should spit out a lot of things
gles1test1 x  # shows some spinning triangles on green background (arg is num_frames it should run I guess, non-integer args make it run forever)
gles2test1 x  # similar but on purple backgroun (see above for dummy arg x) - if red background check powervr.ini's pixel format we set above
cd -

# more info about SGX chip if needed
cat /proc/pvr/version

If the above installed fine, and all the test steps in the above block also worked, let's automate the SGX initialization at startup via systemd: 

# service script starting PowerVR environment
sudo tee -a /etc/systemd/system/pvr-init.service <<"TTT"
[Unit]
Description=PowerVR
After=multi-user.target
[Service]
Type=oneshot
RemainAfterExit=yes
# startup fails sometimes (too early?), retry in loop - this paired with TimeoutStartSec is a
# workaround to oneshot services refusing decent Restart= settings (at least for systemd <= 232)
ExecStart=/bin/sh -c 'while ! /etc/init.d/rc.pvr start; do sleep 5; done'
ExecStop=/etc/init.d/rc.pvr stop
TimeoutStartSec=60sec
User=root
[Install]
WantedBy=multi-user.target
TTT

sudo systemctl enable pvr-init

# done
sudo reboot
]]> FreeBSD 12 on the 51nb x210 http://blog.0xpebbles.org/FreeBSD-12-on-the-51nb-x210 16 Jun 2019 00:00:00 +0000 The x210 is a x201 thinkpad chassis with modern hardware inside, from a group of chinese hardware modders that started out doing screen upgrades, originally. Depending on the production batches they run and probably the chassis they have available the hardware might differ slightly from machine to machine. The group's blog has more info on the batches, different devices, BIOS updates, etc.. A few notes about the hardware first, before talking about FreeBSD on the one I got:

Hardware Notes

As said, the hardware depends on the batch, some special runs they sometimes do, and the chassis used (one can also buy just the board alone). E.g. the laptop I got does have a Core i5-8250U inside (there are i7 versions, too) and was in a chassis that came with a default palmrest (no touchpad, but some others seem to have been shipped with one). It has a WUXGA (1920x1200) display which is physically actually slightly bigger than the chassi's screen bezel.

Since I did want a touchpad (I guess I could've asked them directly, so my bad) I bought a used palmrest. There are two types for x201* machines that aren't compatible with each other, grouped as 25W and 35W models with different connectors, full list here (I needed a 35W one and specifically replaced FRU 60Y5414 with FRU 60Y5415). Turns out that in order to make the touchpad palmrest fit, some clipping was needed as the mainboard's m.2 port and another part won't fit otherwise. Some plastic and metal needs to be removed on both sides of the touchpad, so one before and two after pictures:

FRU 60Y5415 before modification FRU 60Y5415 after modification (left) FRU 60Y5415 after modification (right)

Running FreeBSD 12

Graphics and Screen

General

You'll need FreeBSD 12 for the graphics, I originally tried to make it work on FreeBSD 11.2, but didn't manage to get it to work.

The processor has an Intel UHD Graphics 620 integrated GPU. You'll need to install the graphics/drm-kmod port. Make sure to read the post install message! E.g. you'll need to add all users that need graphics to the 'video' group, and also put the following in /etc/rc.conf (more infos here): 

kld_list="/boot/modules/i915kms.ko"

I had some initial problems with the console, but I don't fully remember what. Either way, setting the following in /boot/loader.conf helped and also gave me a hires console: 

kern.vty=vt
kern.vt.fb.default_mode="1920x1200"

X11 / xorg.conf

The FreeBSD Graphics wiki suggests that one "should not have to prepare an xorg.conf configuration file" and that "Xorg should autodetect the driver and utilize the modesetting Xorg driver and glamor driver".

However, this did not work for me on this machine. The autodetection used the intel driver with uxa as acceleration method, which was slow and buggy. Not having read carefully the wiki that states clearly that the autodetection "should [...] utilize the modesetting Xorg driver and glamor driver", I did switch the acceleration from uxa to sna, first. This improved the situation by a lot, however some artefacts remained and X11 froze from time to time (which I was able to unfreeze by switching to the vt console and back).
Anyways, what was really needed (and works really well) was a custom xorg.conf that sets the modesetting driver with glamor as acceleration method, explicitly. So I let X -configure create an xorg.conf template for me and then specified explicitly in Section "Device": 

Driver  "modesetting"
Option  "AccelMethod" "glamor"

Screen Size

As mentioned above, the physical screen size is a tiny bit bigger than the chassi's bezel. I simply reduced the usable screen area a bit via the window manager. I use x11-wm/spectrwm which has a region feature, which allowed me to do that easily. In the end I have a usable region of 1912x1194 pixels, losing 8 pixels on the right, 3 on top and 3 on the bottom. So the needed spectrwm config entry in this specific case would be: 

region = screen[1]:1912x1194+0+3

If not having such support from the wm, one could register and use a new modeline via xrandr. I did that first, and it works, but I didn't figure out how to specifically tell it where the screen should start, it was always anchored in the top left corner.

Alternatively, one could also file off some plastic of the bezel, to make use of the full screen.

Screen Brightness

The brightness buttons don't work out of the box, and things like kldloading acpi_ibm(4) won't help - this isn't actually a thinkpad after all. Using graphics/intel-backlight does not help either, as the GPU doesn't control the brightness (in contrast to for example the x201), but the embedded controller does. Poking around the ACPI tables (acpidump -dt) made me quickly find an already existing method \_SB.PCI0.GFX0.DD1F._BCM that can be used to set \_SB.PCI0.LPCB.EC0.BKLG, which controls the brightness levels. So using sysutils/acpi_call we can run (after kldloading acpi_call first): 

acpi_call -p '\_SB.PCI0.GFX0.DD1F._BCM' -i $VAL   # 0 <= $VAL <= 15

The predefined 16 brightness levels are not ideal, 0 is still way brighter than what you would get from other laptops. The good news is that those levels can be adjusted by apparently patching the EC firmware (more info here), however I haven't done so, myself, though.

I also have not looked into hooking up the brightness buttons out of laziness, but simply wrapped the acpi_call based control above in a convenience script.

Sound

Seems to work fine out of the box using snd_hda(4). However, in order for the sound to automatically switch over to the headphone jack when something gets plugged in, I needed the following in /boot/device.hints: 

hint.hdaa.0.nid31.config="as=4 seq=0 device=Speaker"
hint.hdaa.0.nid25.config="as=4 seq=15 device=Headphones"

I looked up the nid numbers needed from the output of sysctl dev.hdaa.

The volume buttons I hooked up myself by making use of the XF86 multimedia keys XF86AudioRaiseVolume and XF86AudioLowerVolume. Since I'm using x11-wm/spectrwm, as mentioned above, the according spectrwm config entries would be something along those lines (with some bonus key combinations to control treble and bass, given we had to bind those buttons manually anyways): 

program[raise_volume]  = mixer vol +2 pcm +2
program[lower_volume]  = mixer vol -2 pcm -2
program[raise_treble]  = mixer treble +5
program[lower_treble]  = mixer treble -5
program[raise_bass]    = mixer bass +5
program[lower_bass]    = mixer bass -5
bind[raise_volume]     = XF86AudioRaiseVolume
bind[lower_volume]     = XF86AudioLowerVolume
bind[raise_treble]     = Control+XF86AudioRaiseVolume
bind[lower_treble]     = Control+XF86AudioLowerVolume
bind[raise_bass]       = Shift+XF86AudioRaiseVolume
bind[lower_bass]       = Shift+XF86AudioLowerVolume

The downside of making use of XF86 multimedia keycodes is that this now only works in X, of course.

WiFi

My machine came with a Broadcom BCM4352 802.11ac Mini PCIe WiFi card, for which there is no driver support at the time of writing. So I replaced it with another one. Contrary to stock/regular Thinkpads (and laptops in general), the BIOS the x210 comes with doesn't do any wifi card whitelisting, so swapping out the card doesn't require any extra work.

Webcam

The laptop I got also had a webcam installed (from Chicony Electronics), which shows up as a USB device and works using multimedia/webcamd. For example, to test it quickly, make sure cuse is kldloaded, then: 

service webcamd onestart $DEV  # where $DEV is the camera's USB device, e.g. ugen0.2
mplayer tv:// -tv driver=v4l:width=320:height=240:device=/dev/video0 -fps 25

The camera supports in my case the following resolutions: QVGA (320x240), VGA (640x480), SVGA (800x600), XGA (1024x768), WXGA (1280x800), SXGA (1280x1024)

Other

  • I did not test the modem, microphone jack or the external mini display port so far.
  • I did dump the contents of the flash chip using a BusPirate, which worked fine. Eventually I want to try flashing coreboot, but haven't done so, yet. No flash write protection seems to be active, so internal flashing could be used, I guess.
  • The machine is mostly silent, the fan starts when there is some decent load, but other than that it doesn't even spin at all and the machine also doesn't get hot either (around 45°C for idly use).
  • Suspend to RAM or disk doesn't seem to work, the graphics driver freezes while trying to suspend and the machine hangs at that point. Maybe a future update will fix this.
  • Thinklight and numlock work, but those work without OS support, anyways.
  • Also, as with the brightness buttons, other Fn-buttons aren't wired up, either.
  • I don't have any data to compare the x210's battery life with, but compared to my x201i it improved greatly. On a 3 year old, already well degraded 9-cell battery that didn't run the x201i for more than a good hour, this machine gets 4h out of it. I'm not sure what a new battery would yield, but given what I had before, I'm super happy.

    • UPDATE (2019-10-09): A brand new 9-cell battery with 7800mAh seems to run this machine for nearly 10h (for light use, e.g. reading stuff, writing mails, etc.). This is way more than what I actually hoped for.

    ]]> The (sorry) state of the www http://blog.0xpebbles.org/The-sorry-state-of-the-www 14 Mar 2019 00:00:00 +0000 The web today isn't the way more content-centric and naive-but-simple web we used to have in the 90s and early 2000s, but has developed a much stronger focus on tracking and trying to influence what people think instead of being a source of more or less freely and actively shared information. Additionally, a lot of sites seem to care today more about presentation than content, resulting in many shiny, but bloated and less useful sites.

    The bloat is comical sometimes; even heavily used sites that would benefit directly from leaner approaches somehow find it normal to use insane amounts of resources, leading to pointlessly increased traffic, CPU load, energy consumption and ultimately carbon footprint. e.g. a standard slack web chat session with no more than 50 users easily uses up 250-500MB of RAM (in addition to the already heavy browser requirements). Loading slack on my laptop takes longer than booting the OS. I find it hard to justify the need for this kind of bloat for a chat. We talk about exchanging a few bytes of text between people in real time, with optional message history, something we do for 30+ years on way less beefy machines.

    Also, more and more gets centralized, distributed and managed by a few big players in the field, who additionally make it hard for independent services to exist. This same centralization happens for other common, non-web services: for example, running your own mailserver today is more often than not a constant battle to not be blacklisted as a spam source, because many relays just consider @gmail.com and other big ones as the only trustworthy sender domains (which is ironic at best), no matter how clean your record is, how careful you set up everything for spam heuristics (like PTR records, DKIM, etc.).

    Shining through in all paragraphs above is a feeling that seems to underlie this all, namely it all haven gotten more and more about power. In some areas it surely always was somewhere about power, but from the end user perspective it is so much more invasive now, and the content so much more shallow by average, and the positive enthousiasm over this big pool of information of the early web is now either gone or comes with a bitter after-taste.

    Not to mention that there are so many things with the web that are broken to begin with, its tech's ever growing complexity will always be error prone, leading to all kinds of vulnerabilities, breaches and scams. Its one way links will always break eventually leading to frustration, and not to mention all the resources and energy wasted by all the useless bloat of the nowadays rapid-development web frameworks and often intentional content-hiding style.

    I don't seem to be the only one that is worried about this development so here's a collection of links looking at this from plenty of different angles:

    Another worrisome thing going on, IMHO, is the push by many of the big players for DNS-over-HTTPS (DOH). They seem to propagate the claim that this is needed because DNS is insecure. Although a valid point at the time of writing, they ignore the fact that the latter is addressed by things like dnscrypt with DNSSEC. Besides the fact that pushing operating system level services like a name resolver into an application (e.g. a browser, where you would for example set the DoH resolver in Firefox via config option network.trr.uri) is insane to begin with, they seem to have strong reasons to push for DoH, probably motivated again by power and tracking reasons. Why wouldn't they be interested in overriding the globally by default decentralized ISP provided resolvers (which most peoples' system would get via DHCP if not overridden), and instead getting a grip on the majority of all DNS lookups by sending them by default to one or a few big players? Data is money after all. I don't want to even think about the massive single point of failure such a centralization would also be.

    There are claims that this would somehow free you from those apparently evil ISPs that censor your requests. Sure, this might happen (as is the case with kinda every coffee shop or hotel uplink), but they don't mention how tasty it would be for them to resolve things for you that bypass your OS-level resolver, where you could setup per-domain blacklists to block trackers, adservers, malicious stuff, etc..

    Wouldn't it be too nice for companies like Cloudflare to receive in a centralized way a major number of name lookups, by being the default resolver in one or more of the already few major browsers? Would you trust this US company, that already encourages people to give them their certificates to play man-in-the-middle, to know more about your browsing habits? Would you trust Google with their Chrome browser having even more insight into your live (and control over what you can access), but defaulting Chrome to use their resolvers? What a coincidence that that's a company that makes money by tracking you for targeted advertising... not. A quote that reflects my worries from the comments under this DoH intro:

    Brett Glass: So, Mozilla intends to hack users' DNS, redirecting their queries away from their ISPs (which are trustworthy and with which they have a business relationship) to an untrustworthy VPN vendor - Cloudflare. Those users are not Cloudflare's customers, and so the only way Cloudflare can monetize this service is to spy on users and sell their personal information. In short, Mozilla is supporting, aiding, and abetting privacy invasion - probably in exchange for money from Cloudflare. Not only unethical but probably actionable by the FTC.

    Some links specifically about DoH and the debate around it:

    To me this feels like it's all in line with the "Death of Transit" presentation linked to above in the first bullet point list: another worrisome development mainly supported by some internet megacorps longing for even more centralization and control.

    As a closing line to avoid misinterpretation: my intro text as well as the articles in the first bullet point list are about the web as a content source, and not about the web's reinvention of the thin-client/mainframe model, which is what webapps basically are.

    UPDATE (2019-06-03): Added a detailed and interesting opinion article to the DoH article list
    UPDATE (2021-03-17): Added some more links

    ]]>