Terry Burton's blog

Creating PDF Documentation From a GitHub Wiki Using Pandoc

Fri, 20 Feb 2015 00:00:00 +0000

The following explains how the printable documentation for Barcode Writer in Pure PostScript was refreshed so that it could be generated primarily from content in the wiki.

Adding the Wiki as a Git Submodule

To allow the documentation build system to access files in the wiki, build directory and main project repository using predictable relative file names you can add the wiki repository as a git submodule to the main repository:

git submodule add ../postscriptbarcode.wiki.git wikidocs

In future you will want to checkout your repository as follows:

git clone --recursive git@github.com:bwipp/postscriptbarcode.git

Documentation Build System File Structure

We create the Pandoc-based documentation build system in the __pandoc directory of the wikidocs submodule.

wikidocs
│
│ (Documentation build system)
│
├── __pandoc
│   │
│   ├── Makefile ①
│   │
│   │ (Symlink to wiki images directory)
│   │
│   ├── images -> ../images/
│   │
│   │ (Filters for pre-processing the wiki markdown)
│   │
│   ├── __rewritewikilinks.hs ②
│   ├── __rewritepngtoeps.hs ③
│   │
│   │ (PDF specific content — mainly section headings, introductions, etc.)
│   │
│   ├── __symbology-reference.md
│   ├── __options-reference.md
│   └── __knowledge-base.md
│
│ (Normal wiki content follows)
│
├── images
│   ├── ean13-1.eps   (EPS files purpose generated for PDF documentation)
│   └── ean13-1.png
│
├── symbologies
│   └── EAN-13.md
├── options
│   └── Text-Properties.md
└── kb
      └── FAQs.md

Pandoc Arguments

To build the PDF documentation we run Pandoc with as follow:

$ pandoc

  --from=markdown_github
  --to=latex

  --filter __pandoc/__rewritewikilinks.hs ②
  --filter __pandoc/__rewritepngtoeps.hs ③

  -M title="Barcode Writer in Pure PostScript"
  -M author="https://bwipp.terryburton.co.uk"

  -V geometry:a4paper,margin=2cm

  --toc --toc-depth=2
  --chapters

  --output=__pandoc/barcodewriter.pdf

  [ followed by a list of *.md files ]

Note that this requires Pandoc version 1.12 or later for filter support.

Makefile ①

Using a makefile allows the documentation to be rebuilt and added to the main repository using the following commands:

$ cd wikidocs
$ make -f __pandoc/Makefile
$ cp __pandoc/barcodewriter.pdf ../docs

PANDOC_DIR = __pandoc
PANDOC_TEMPLATE_LATEX = $(PANDOC_DIR)/templates/template.latex
EPS_IMAGES = $(wildcard images/*.eps)
PDF_DOCS  = $(PANDOC_DIR)/barcodewriter.pdf

define MD_FILES =
  ../README.md                            \
  kb/Quick-Guide.md                       \
  ../src/README.monolithic                \
  ../src/README.resource                  \
                                          \
  $(PANDOC_DIR)/__symbology-reference.md  \
  symbologies/EAN-13.md                   \
...
  $(PANDOC_DIR)/__options-reference.md    \
  options/Text-Properties.md              \
...
  $(PANDOC_DIR)/__knowledge-base.md       \
  kb/FAQs.md                              \
...
  ../cited-by.md
endef

$(PDF_DOCS): $(MD_FILES) $(EPS_IMAGES) $(PANDOC_TEMPLATE_LATEX) $(INTRO_FILE_LATEX)
    pandoc                                      \
  -f markdown_github -t latex                   \
  -F $(PANDOC_DIR)/__rewritewikilinks.hs        \
  -F $(PANDOC_DIR)/__rewritepngtoeps.hs         \
  -M title="Barcode Writer in Pure PostScript"  \
  -M author="https://bwipp.terryburton.co.uk"    \
  -V geometry:a4paper,margin=2cm                \
  --toc --toc-depth=2                           \
  --chapters                                    \
  -o $@                                         \
  $(MD_FILES)

Filter: rewritewikilinks.hs ②

Rewrite relative links to other wiki pages to lowercase in-page style links maintaining existing in-page links and absolute website links, e.g. links to a EAN-13 wiki page would become a #ean-13 in-page link. Then ensure that each page begins with a heading named after the page such that these rewritten links map to the label for the corresponding heading thus maintaining a functioning navigation system in the final PDF document.

#!/usr/bin/env runhaskell

import Text.Pandoc.JSON
import Data.Char

main = toJSONFilter rewritewikilinks

rewritewikilinks :: Inline -> [Inline]
rewritewikilinks ( Link txt ('#':url,title) ) = [ ( Link txt ("#"++url,title) ) ]
rewritewikilinks ( Link txt ('h':'t':'t':'p':':':url,title) ) = [ ( Link txt ("http:"++url,title) ) ]
rewritewikilinks ( Link txt ('h':'t':'t':'p':'s':':':url,title) ) = [ ( Link txt ("https:"++url,title) ) ]
rewritewikilinks ( Link txt (url,title) ) = [ ( Link txt ("#" ++ map toLower url,title) ) ]
rewritewikilinks x = [x]

Filter: rewritepngtoeps.hs ③

Rewrite sources for images from .png file to the corresponding .eps file. Obviously this requires you to create the EPS files in the first place. For barcodes this greatly improves the quality of the images in the PDF output by maintaining them in vector format. This might not be as important for other types of content.

#!/usr/bin/env runhaskell

import Text.Pandoc.JSON
import System.FilePath

main = toJSONFilter rewritepngtoeps

rewritepngtoeps :: Inline -> [Inline]
rewritepngtoeps ( Image txt (url,title) ) = [ ( Image txt (rewritefile url,title) ) ]
rewritepngtoeps x = [x]

rewritefile :: String -> String
rewritefile filename = replaceExtension filename ".eps"

NAT Filtering DNS With iptables/netfilter

Tue, 17 Nov 2009 00:00:00 +0000

Update: DNS Response Policy Zones (DNS RPZ) are now a much better solution to this problem.

Recently I have been looking at approaches to containing the activity of the Conficker (Downup, Downadup, Kido) worm, in particular by focusing on blocking of initial DNS queries for its large set of rendezvous domains that it uses to update its executable.

The existing techniques that I’ve read about involve creating nameservers that are authoritative for a huge list of zones, either themselves resolving or forwarding to distinct resolvers. But as anybody that has run BIND configured with a large set of authoritative zones will know, the resource requirements for such a configuration are excessive: high memory usage, long pauses during cache cleanup, slow startup times…

So I derived an entirely different filtering technique that appears to work very well for blocking a set of over 100,000 domains, which has negligible resource requirements and which does not require reconfiguration of an existing resolver infrastructure.

The essential principle is that both UDP and TCP versions of the protocol for DNS queries are trivial to proxy in a robust manner at layer 4, i.e. using only simple NAT, without any application awareness. In fact, this is how many commodity broadband routers function - they perform NAT on DNS packets destined to their LAN-side IP address to redirect them to an ISP’s resolvers and then relay the response back to the requester.

This allows for the creation of an extremely lightweight set of DNS proxy servers that can be placed between some network whose hosts you wish to contain and your existing resolver infrastructure. These proxy servers may then be configured to perform content-based filtering of DNS requests by performing string matching at a specific offset within the DNS packets. Such a proxy server requires nothing more than low-spec hardware (or a modest virtual machine) and uses only the built in features of a vanilla Linux 2.6 kernel - no nameserver software required.

What follows assumes a sound understanding Linux networking and iptables. The firewall snippets are in “iptables-restore” format. It is intended to provide comprehensive details of the approach I am using but does not consitute anything like a complete HOWTO.

The following configures a NAT that will rewrite DNS packets destined to the DNS_PROXY address to a set of DNS_RESOLVER_{1,2,3} addresses following a round-robin policy.

echo 1 > /proc/sys/net/ipv4/ip_forward

*mangle
:PREROUTING ACCEPT [0:0]
-A PREROUTING -d [DNS_PROXY] -p udp -m udp --dport 53 -m statistic --mode nth --every 3 --packet 0  -m state --state new -j CONNMARK --set-mark 1
-A PREROUTING -d [DNS_PROXY] -p udp -m udp --dport 53 -m statistic --mode nth --every 3 --packet 1  -m state --state new -j CONNMARK --set-mark 2
-A PREROUTING -d [DNS_PROXY] -p udp -m udp --dport 53 -m statistic --mode nth --every 3 --packet 2  -m state --state new -j CONNMARK --set-mark 3
-A PREROUTING -d [DNS_PROXY] -p tcp -m tcp --dport 53 -m statistic --mode random --probability 0.33 -m state --state new -j CONNMARK --set-mark 1
-A PREROUTING -d [DNS_PROXY] -p tcp -m tcp --dport 53 -m statistic --mode random --probability 0.5  -m state --state new -j CONNMARK --set-mark 2
-A PREROUTING -d [DNS_PROXY] -p tcp -m tcp --dport 53                                               -m state --state new -j CONNMARK --set-mark 3
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING  -m connmark --mark 1 -j DNAT --to-destination [DNS_RESOLVER_1]
-A PREROUTING  -m connmark --mark 2 -j DNAT --to-destination [DNS_RESOLVER_2]
-A PREROUTING  -m connmark --mark 3 -j DNAT --to-destination [DNS_RESOLVER_3]
-A POSTROUTING -m connmark --mark 1 -j SNAT --to-source [DNS_PROXY]
-A POSTROUTING -m connmark --mark 2 -j SNAT --to-source [DNS_PROXY]
-A POSTROUTING -m connmark --mark 3 -j SNAT --to-source [DNS_PROXY]
COMMIT

We want DNS packets that are queries, rather than responses, to be sent via the DNSCHECK chain. Other DNS packets are passed unchecked whilst non-DNS protocols are blocked. We only accept DNS packets from a trusted SRC_NET to avoid being an open resolver.

*filter
:FORWARD ACCEPT [0:0]
-A FORWARD -s SRC_NET -p udp --dport 53 -m u32 --u32 "0>>22&0x3C@8>>15&0x01=0" -j DNSCHECK
-A FORWARD -s SRC_NET -p udp --dport 53 -j ACCEPT
-A FORWARD -s SRC_NET -p tcp --dport 53 -j ACCEPT
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -j LOGDROP

To reduce the number of (expensive) “string” match tests applied to each DNS query we dispatch the packet to the relevant DNSCHECKxx chain based on the first two characters of the query data, using a (cheap) “u32” match on the correct offset of the packet.

:DNSCHECK - [0:0]
-A DNSCHECK -m u32 --u32 "0>>22&0x3C@19&0xffff=0x6161" -j DNSCHECKaa
-A DNSCHECK -m u32 --u32 "0>>22&0x3C@19&0xffff=0x6162" -j DNSCHECKab
...
-A DNSCHECK -m u32 --u32 "0>>22&0x3C@19&0xffff=0x7a7a" -j DNSCHECKzz

In each DNSCHECKxx chain we perform the full string matches on the query data.

:DNSCHECKaa - [0:0]
-A DNSCHECKaa -m string --from 40 --to 53 --hex-string "|08|aaadokfn|03|net|00|"   --algo bm -j LOGDROP
-A DNSCHECKaa -m string --from 40 --to 49 --hex-string "|05|aaakp|02|cc|00|"       --algo bm -j LOGDROP
-A DNSCHECKaa -m string --from 40 --to 55 --hex-string "|0b|aaapcxqiqgg|02|ws|00|" --algo bm -j LOGDROP
...

Log and drop matches, rate limiting to prevent flooding of logfiles.

:LOGDROP - [0:0]
-A LOGDROP -m limit --limit 1/second --limit-burst 100 -j LOG
-A LOGDROP -j DROP

The following sample scripts should be useful for generating and loading the daily iptables rulesets:

gen_domains.sh

Uses Downatool2 under Wine to create date-based files for a number of days that contain a sorted list of rendezvous domains for each Conficker variant. Could be invoked as an unprivileged user from cron once per week.

build_iptables.pl

Builds a iptables firewall policy in “iptables-restore” format that implements the NAT-filter for DNS described above from a list of files containing blacklisted domains.

load_iptables.sh

Uses build_iptables.pl to create a firewall policy to filter domains for the next couple of days and then executes it. Could be invoked as root from cron each night.

Apache accesslog to syslog

Fri, 09 Nov 2007 00:00:00 +0000

Apache allows its error logs to be written to local syslog, however it does not natively support the directing of access logs to the syslog. How frustrating!

It does however allow access logs to be written to a pipe and I have seen a number of home-brew scripts that essentially redirect the Apache access log data from STDIN to syslog.

I’ve yet to see anything quite as simple as the following directive that I cooked up today:

CustomLog "||/usr/bin/logger -t apache -i -p local6.notice" combined

It pipes the access log data to the BSD logger(1) utility that is installed by default on almost any Unix system. No need for any more of those STDIN wrapper scripts!