My blog_title_here

Recent fixes to great tools - 0bin and Convergence

I've tried both of these in the past, but didn't have attention budget to make them really work for me - which finally found now, so wanted to also give crawlers a few more keywords on these nice things.

0bin - leak-proof pastebin

As I pastebin a lot of stuff all the time - basically everything multiline - because all my IM happens in ERC over IRC (with bitlbee linking xmpp and all the proprietary crap like icq, skype and twitter), and IRC doesn't handle multiline messages at all.

All sorts of important stuff ends up there - some internal credentials, contacts, non-public code, bugs, private chat logs, etc - so I always winced a bit when pasting something in fear that google might index/data-mine it and preserve forever, so I figured it'll bite me eventually, somewhat like this:

Easy and acceptable solution is to use simple client-side crypto, with link having decryption key after hashmark, which never gets sent to pastebin server and doesn't provide crawlers with any useful data. ZeroBin does that.

But original ZeroBin is php, which I don't really want to touch, and have its share of problems - from the lack of command-line client (for e.g. grep stuff log | zerobinpaste), to overly-long urls and flaky overloaded interface.

Luckily, there's more hackable python version of it - 0bin, for which I hacked together a simple zerobinpaste tool, then simplified interface to bare minimum and updated to use shorter urls (#41, #42) and put to my host - result is paste.fraggod.net - my own nice robot-proof pastebin.

URLs there aren't any longer than with regular pastebins:

http://paste.fraggod.net/paste/pLmEb0BI#Verfn+7o

Plus the links there expire reliably, and it's easy to force this expiration, having control over app backend.

Local fork should have all the not-yet-merged stuff as well as the non-upstreamable simpler white-bootstrap theme.

fatrace - poor man's auditd

Was hacking on (or rather debugging) Convergence FF plugin and it became painfully obvious that I really needed something simple to push js changes from local git clone to ~/.mozilla so that I can test them.

Usually I tend to employ simple ad-hoc for src in $(git st | awk ...); do cat $src >... hack, and done same thing in this case as well, but was forgetting to run it after small "debug printf" changes waaay too often.

At this point, I sometimes hack some ad-hoc emacs post-save hook to run the thing, but this time decided to find some simpler and more generic "run that on any changes to path" tool.

Until the last few years, the only way to do that was polling or inotify, and for some project dir it's actually quite fine, but luckily there's fanotify in kernel now, and fatrace looks like the simliest cli tool based on it.

# fatrace
sadc(977): W /var/log/sa/sa24
sadc(977): W /var/log/sa/sa24
sadc(977): W /var/log/sa/sa24
sadc(977): W /var/log/sa/sa24
qmgr(1195): O /var/spool/postfix/deferred
qmgr(1195): CO /var/spool/postfix/deferred/0
qmgr(1195): CO /var/spool/postfix/deferred/3
qmgr(1195): CO /var/spool/postfix/deferred/7
...

I never got over auditd's complexity for such simple use-cases, so was damn glad that there is a real and simpler alternative now.

Unfortunately, with power of the thing comes the need for root, so one simple bash wrapper later, my "sync changes" issue was finally resolved:

(root) ~# fatrace_pipe ~user/hatch/project
(user) project% xargs -in1 </tmp/fatrace.fifo make

Looks like a real problem-solver for a lot of real-world "what the hell happens on the fs there!?" cases as well - can't recommend the thing highly-enough for all that.

TCP Hijacking for The Greater Good

As discordian folk celebrated Jake Day yesterday, decided that I've had it with random hanging userspace state-machines, stuck forever with tcp connections that are not legitimately dead, just waiting on both sides.

And since pretty much every tool can handle transient connection failures and reconnects, decided to come up with some simple and robust-enough solution to break such links without (or rather before) patching all the apps to behave.

I've used such technique to test some twisted-things in the past, so it was easy to dig scapy-automata code for doing that, though the real trick is not to craft/send FIN or RST packet, but rather to guess TCP seq/ack numbers to stamp it with.

Alas, none of the existing tools (e.g. tcpkill) seem to do anything clever in this regard.

cutter states that

There is a feature of the TCP/IP protocol that we could use to good effect here - if a packet (other than an RST) is received on a connection that has the wrong sequence number, then the host responds by sending a corrective "ACK" packet back.

But neither the tool itself nor the technique described seem to work, and I actually failed to find (or recall) any mentions (or other uses) of such corrective behavior. Maybe it was so waaay back, dunno.

Naturally, as I can run such tool on the host where socket endpoint is, local kernel has these numbers stored, but apparently no one really cared (or had a legitimate enough use-case) to expose these to the userspace... until very recently, that is.

Patching kernel to just expose stuff via /proc looks like bit of a burden as well, though an isolated module code would probably do the job well. Weird that there doesn't seem to be one of these around already, closest one being tcp_probe.c code, which hooks into tcp_recv code-path and doesn't really get seqs without some traffic either.

One interesting idea that got my attention and didn't require a single line of extra code was proposed on the local xmpp channel - to use tcp keepalives.

Sure, they won't make kernel drop connection when it's userspace that hangs on both ends, with connection itself being perfectly healthy, but every one of these carries a seq number that can be spoofed and used to destroy that "healthy" state.

Pity these are optional and can't be just turned on for all sockets system-wide on linux (unlike some BSD systems, apparently), and nothing uses these much by choice (which can be seen in netstat --timer).

And tuning keepalives to be frequent-enough seem to be a no-brainer and shouldn't have any effect on 99% of legitimate connections at all, as they probably pass some traffic every other second, not after minutes or hours.

net.ipv4.tcp_keepalive_time = 900
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 156

(default is to send empty keepalive packet after 2 hours of idleness)

With that, tool has to run ~7 min on average to kill any tcp connection in the system, which totally acceptable, and no fragile non-portable ptrace-shellcode magic involved (at least yet, I bet it'd be much easier to do in the future).

Code and some docs for the tool/approach can be found on github.

More of the same (update 2013-08-11):

Actually, lacking some better way to send RST/FIN from a machine to itself than swapping MACs (and hoping that router is misconfigured enough to bounce packet "from itself" back) or "-j REJECT --reject-with tcp-reset" (plus a "recent" match or transient-port matching, to avoid blocking reconnect as well), countdown for a connection should be ~7 + 15 min, as only next keepalive will reliably produce RST response.

With a bit of ipset/iptables/nflog magic, it was easy to make the one-time REJECT rule, snatching seq from dropped packet via NFLOG and using that to produce RST for the other side as well.

Whole magic there goes like this:

-A conn_cutter ! -p tcp -j RETURN
-A conn_cutter -m set ! --match-set conn_cutter src,src -j RETURN
-A conn_cutter -p tcp -m recent --set --name conn_cutter --rsource
-A conn_cutter -p tcp -m recent ! --rcheck --seconds 20\
        --hitcount 2 --name conn_cutter --rsource -j NFLOG
-A conn_cutter -p tcp -m recent ! --rcheck --seconds 20\
        --hitcount 2 --name conn_cutter --rsource -j REJECT --reject-with tcp-reset

-I OUTPUT -j conn_cutter

"recent" matcher there is a bit redundant in most cases, as outgoing connections usually use transient-range tcp ports, which shouldn't match for different attempts, but some apps might bind these explicitly.

ipset turned out to be quite a neat thing to avoid iptables manipulations (to add/remove match).

It's interesting that this set of rules handles RST to both ends all by itself if packet arrives from remote first - response (e.g. ACK) from local socket will get RST but won't reach remote, and retransmit from remote will get RST because local port is legitimately closed by then.

Current code allows to optionally specify ipset name, whether to use nflog (via spin-off scapy-nflog-capture driver) or raw sockets, and doesn't do any mac-swapping, only sending RST to remote (which, again, should still be sufficient with frequent-enough keepalives).

Now, if only some decade-old undocumented code didn't explicitly disable these nice keepalives...

Fighting storage bitrot and decay

Everyone is probably aware that bits do flip here and there in the supposedly rock-solid, predictable and deterministic hardware, but somehow every single data-management layer assumes that it's not its responsibility to fix or even detect these flukes.

Bitrot in RAM is a known source of bugs, but short of ECC, dunno what one can do without huge impact on performance.

Consider typical linux storage stack on a commodity hardware:

• You have closed-box proprietary hdd brick at the bottom, with no way to tell what it does to protect your data - aside from vendor marketing pitches, that is.

• Then you have well-tested and robust linux driver for some ICH storage controller.

  I wouldn't bet that it will corrupt anything at this point, but it doesn't do much else to the data but pass around whatever it gets from the flaky device either.

• Linux blkdev layer above, presenting /dev/sdX. No checks, just simple mapping.

• device-mapper.

  Here things get more interesting.

  I tend to use lvm wherever possible, but it's just a convenience layer (or a set of nice tools to setup mappings) on top of dm, no checks of any kind, but at least it doesn't make things much worse either - lvm metadata is fairly redundant and easy to backup/recover.

  dm-crypt gives no noticeable performance overhead, exists either above or under lvm in the stack, and is nice hygiene against accidental leaks (selling or leasing hw, theft, bugs, etc), but lacking authenticated encryption modes it doesn't do anything to detect bit-flips.

  Worse, it amplifies the issue.

  In the most common CBC mode one flipped bit in the ciphertext will affect a few other bits of data until the end of the dm block.

  Current dm-crypt default (since the latest cryptsetup-1.6.X, iirc) is XTS block encryption mode, which somewhat limits the damage, but dm-crypt has little support for changing modes on-the-fly, so tough luck.
  But hey, there is dm-verity, which sounds like exactly what I want, except it's read-only, damn.

  Read-only nature is heavily ingrained in its "hash tree" model of integrity protection - it is hashes-of-hashes all the way up to the root hash, which you specify on mount, immutable by design.

  Block-layer integrity protection is a bit weird anyway - lots of unnecessary work potential there with free space (can probably be somewhat solved by TRIM), data that's already journaled/checksummed by fs and just plain transient block changes which aren't exposed for long and one might not care about at all.

• Filesystem layer above does the right thing sometimes.

  COW fs'es like btrfs and zfs have checksums and scrubbing, so seem to be a good options.

  btrfs was slow as hell on rotating plates last time I checked, but zfs port might be worth a try, though if a single cow fs works fine on all kinds of scenarios where I use ext4 (mid-sized files), xfs (glusterfs backend) and reiserfs (hard-linked backups, caches, tiny-file sub trees), then I'd really be amazed.

  Other fs'es plain suck at this. No care for that sort of thing at all.

• Above-fs syscall-hooks kernel layers.

  IMA/EVM sound great, but are also for immutable security ("integrity") purposes ;(

  In fact, this layer is heavily populated by security stuff like LSM's, which I can't imagine being sanely used for bitrot-detection purposes.

  Security tools are generally oriented towards detecting any changes, intentional tampering included, and are bound to produce a lot of false-positives instead of legitimate and actionable alerts.

  Plus, upon detecting some sort of failure, these tools generally don't care about the data anymore acting as a Denial-of-Service attack on you, which is survivable (everything can be circumvented), but fighting your own tools doesn't sound too great.

• Userspace.

  There is tripwire, but it's also a security tool, unsuitable for the task.

  Some rare discussions of the problem pop up here and there, but alas, I failed to salvage anything useable from these, aside from ideas and links to subject-relevant papers.

Scanning github, bitbucket and xmpp popped up bitrot script and a proof-of-concept md-checksums md layer, which apparently haven't even made it to lkml.

So, naturally, following long-standing "... then do it yourself" motto, introducing fs-bitrot-scrubber tool for all the scrubbing needs.

It should be fairly well-described in the readme, but the gist is that it's just a simple userspace script to checksum file contents and check changes there over time, taking all the signs of legitimate file modifications and the fact that it isn't the only thing that needs i/o in the system into account.

Main goal is not to provide any sort of redundancy or backups, but rather notify of the issue before all the old backups (or some cluster-fs mirrors in my case) that can be used to fix it are rotated out of existance or overidden.

Don't suppose I'll see such decay phenomena often (if ever), but I don't like having the odds, especially with an easy "most cases" fix within grasp.

If I'd keep lot of important stuff compressed (think what will happen if a single bit is flipped in the middle of few-gigabytes .xz file) or naively (without storage specifics and corruption in mind) encrypted in cbc mode (or something else to the same effect), I'd be worried about the issue so much more.

Wish there'd be something common out-of-the-box in the linux world, but I guess it's just not the time yet (hell, there's not even one clear term in the techie slang for it!) - with still increasing hdd storage sizes and much more vulnerable ssd's, some more low-level solution should materialize eventually.

Here's me hoping to raise awareness, if only by a tiny bit.

github project link

Secure cloud backups with Tahoe-LAFS

Tahoe-LAFS is a great capability-based secure distributed storage system, where you basically do "tahoe put somefile" and get capability string like "URI:CHK:iqfgzp3ouul7tqtvgn54u3ejee:...u2lgztmbkdiuwzuqcufq:1:1:680" in return.

With remote storage providers, tahoe node works as a client, so all crypto being client-side, actual cloud provider is clueless about the stuff you store, which I find to be quite important thing, especially if you stripe data across many of these leaky and/or plain evil things.

Finally got around to connecting a third backend (box.net) to tahoe today, so wanted to share a few links on the subject:

• https://github.com/mk-fg/tahoe-lafs-public-clouds

  Public cloud drivers for tahoe-lafs.

• https://github.com/mk-fg/lafs-backup-tool

  Tool to intelligently (compression, deduplication, rate-limiting, filtering, metadata, etc) backup stuff to tahoe.

• https://github.com/LeastAuthority/tahoe-lafs

  Upstream repo with more enterprisey cloud backend drivers (s3, openstack, googlestorage, msazure).

• https://tahoe-lafs.org/trac/tahoe-lafs/browser/git/docs/specifications/backends/raic.rst

  Redundant Array of Independent Clouds concept.

• http://www.sickness.it/crazycloudexperiment.txt

  A way to link all the clouds together without having any special drivers.

  As I run tahoe nodes on a headless linux machines, running proprietary GUI clients there doesn't sound too appealing, even if they exist for certain services.

Headless Skype to IRC gateway part 4 - skyped bikeshed

As suspected before, ended up rewriting skyped glue daemon.

But as I'm currently purely a remote worker (not sure if I qualify for "freelancer", being just a drone), and skype is being quite critical for comms in this field, just working thing that silently drops errors and messages is not good enough.

Regardless of whether the thing will be useful upstream, it should finally put a final dot into skype setup story for me, as the whole setup seem to be robust and reliable enough for my purposes now.

Unless vmiklos will find it useful enough to merge, I'll probably maintain the script in this bitlbee fork, rebasing it on top of stable upstream bitlbee.

codetag + tmsu: Tag all the Things (and Go!)

Happens a lot to me, as over the years, a lot of such ad-hoc yet reusable code gets written, and I can usually remember enough implementation details (e.g. which modules were used there, how the methods/classes were called and such), but going "grep" over the source dir takes a shitload of time.

Some things make it faster - ack or pss tools can scan only relevant things (like e.g. "grep ... **/*.py" will do in zsh), but these also run for minutes, as even simple "find" does - there're several django source trees in appengine sdk, php projects with 4k+ files inside, maybe even whole linux kernel source tree or two...

Traversing all these each time on regular fs to find something that can be rewritten in a few minutes will never be an option for me, but luckily there're cool post-fs projects like tmsu, which allow to transcend single-hierarchy-index limitation of a traditional unix fs in much more elegant and useful way than gazillion of symlinks and dentries.

tmsu allows to attach any tags to any files, then query these files back using a set of tags, which it does really fast using sqlite db and clever indexes there.

So, just tagging all the "*.py" files with "lang:py" will allow to:

% time tmsu files lang:py | grep myclass
tmsu files lang:py  0.08s user 0.01s system 98% cpu 0.094 total
grep --color=auto myclass  0.01s user 0.00s system 10% cpu 0.093 total

That's 0.1s instead of several minutes for all the python code in the development area on this machine.

I've added taggers for things that are immediately useful for me to tag files by - implementation language, code hosting (github, bitbucket, local project, as I sometimes remember that snippet was in some public tool), scm type (git, hg, bzr, svn), but it adding a new one is just a metter of writing a "Tagger" function, which, given the path and config, returns a list of string tags, plus they're only used if explicitly enabled in config.

Being a proper compiled language, Go allows to make the thing into a single static binary, which is quite neat, as I realized that I now have a tool to tag all the things everywhere - media files on servers' remote-fs'es, like music and movies, hundreds of configuration files by the app they belong to (think tmsu files daemon:apache to find/grep all the horrible ".htaccess" things and it's "*.conf" includes), distfiles by the os package name, etc... can be useful.

So, to paraphrase well-known meme, Tag All The Things! ;)

github link

Headless Skype to IRC gateway part 3 - bitlbee + skyped

As per previous entry, with mock-desktop setup of Xvfb, fluxbox, x11vnc and skype in place, the only thing left is to use skype interfaces (e.g. dbus) to hook it up with existing IRC setup and maybe insulate skype process from the rest of the system.

Last bit is even easier than usual, since all the 32-bit libs skype needs are collected in one path, so no need to allow it to scan whatever system paths. Decided to go with the usual simplistic apparmor-way here - apparmor.profile, don't see much reason to be more paranoid here.

pcm.!default {
  type null
}
ctl.!default {
  type null
}

(libasound can be pointed to a local config by ALSA_CONFIG_PATH env var)

That "null" module is actually a dynamically-loaded .so, but alsa prints just a single line about it being missing instead of an endless stream of complaints for missing hw, so the thing works, by accident.

Luckily, bitlbee has support for skype, thanks to vmiklos, with sane way to run bitlbee and skype setup on different hosts (as it actually is in my case) through "skyped" daemon talking to skype and bitlbee connecting to its tcp (tls-wrapped) socket.

Using skyped shipped with bitlbee (which is a bit newer than on bitlbee-skype github) somewhat worked, with no ability to reconnect to it (hangs after handling first connection), ~1/4 chance of connection from bitlbee failing, it's persistence in starting skype (even though it's completely unnecessary in my case - systemd can do it way better) and such.

Traceback (most recent call last):
  File "./skyped", line 209, in listener
    ssl_version=ssl.PROTOCOL_TLSv1)
  File "/usr/lib64/python2.7/ssl.py", line 381, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib64/python2.7/ssl.py", line 143, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 104] Connection reset by peer

Functionality-wise, pretty much all I needed is there - one-to-one chats, bookmarked channels (as irc channels!), file transfers (just set "accept all" for these) with notifications about them, user info, contact list (add/remove with allow/deny queries),

But the most important thing by far is that it works at all, saving me plenty of work to code whatever skype-control interface over irc, though I'm very tempted to rewrite "skyped" component, which is still a lot easier with bitlbee plugin on the other end.

Units and configs for the whole final setup can be found on github.

Headless Skype to IRC gateway part 2 - SkypeKit

Thought it should be (hardly) worth a notice that Skype (well, Microsoft now) offers a thing called SkypeKit.

To get it, one have to jump through a dozen of hoops, including long registration form, $5 "tax for your interest in out platform" and wait for indefinite amount of time for invite to the privileged circle of skype hackers.

Here's part of the blurb one have to agree to:

By registering with Skype Developer, you will have access to confidential information and documentation relating to the SkypeKit program that has not been publicly released ("Confidential Information") and you agree not to disclose, publish or disseminate the Confidential Information to any third party (including by posting on any developer forum); and to take reasonable measures to prevent the unauthorised use, disclosure, publication or dissemination of the Confidential Information.

Skype to IRC gateway on a headless server as a systemd user session daemon

Skype is a necessary evil for me, but just for text messages, and it's quite annoying that its closed nature makes it hard to integrate it into existing IM/chat infrastructure (for which I use ERC + ZNC + bitlbee + ejabberd).

So, finally got around to pushing the thing off my laptop machine.

Despite being quite a black-box product, skype has a surprisingly useful API, allowing to do pretty much everything desktop client allows to, which is accessible via several means, one of them being dbus. Wish that API was accessible on one of their servers, but no such luck, I guess. Third-party proxies are actually available, but I don't think +1 point of trust/failure is necessary here.

Since they stopped providing amd64 binaries (and still no word of sources, of course) and all the local non-laptop machines around are amd64, additional quirk is either enabling multibuild and pulling it everything up to and including Qt and WebKit to the poor headless server or just put what skype needs there built on 32-bit machine.

Not too enthusiastic about building lots of desktop crap on atom-based mini-ITX server, decided to go with the latter option, and dependency libs turn out to be fairly lean:

% ldd /opt/skype/skype | awk '$3 {print $3}' |
        xargs ls -lH | awk '{sum+=$5} END {print sum}'
49533468

Naturally, 50M is not an issue for a reasonably modern amounts of RAM.

But, of course, skype runs on X server, so Xvfb (cousing of X, drawing to memory instead of some GPU hardware):

# cave resolve -zx1 xorg-server x11vnc fluxbox

Concrete example above is for source-based exherbo, I think binary distros like debian might package Xvfb binary separately from X (in some "xvfb" package). fluxbox is there to have easy time interacting with skype-created windows.

Note - no heavy DE stuff is needed here, and as I was installing it on a machine hosting cairo-based graphite web frontend, barely any packages are actually needed here, aside from a bunch of X protocol headers and the things specified.

So, to run Xvfb with VNC I've found a bunch of simple shell scripts, which were guaranteed to not provide a lot of things a normal desktop session does, miss stray pids, create multiple instances for all the things involved, loose output, no xdg session, etc.

In general (and incomplete) case, something like this should be done:

export DISPLAY=:0
Xvfb $DISPLAY -screen 0 800x600x16 &
x11vnc -display $DISPLAY -nopw -listen localhost &
fluxbox &
skype &
wait

So, to not reinvent the same square wheel, decided to go with trusty systemd --user, as it's used as a system init anyway.

skype-desktop.service:

[Service]
User=skype
PAMName=login
Type=notify
Environment=DISPLAY=:1
Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=%h/tmp/session_bus_socket
ExecStart=/usr/lib/systemd/systemd --user

[Install]
WantedBy=multi-user.target

Aside from a few quirks like hardcoding dbus socket, that already fixes a lot of XDG_* related env-stuff, proper start/stop cleanup (no process escapes from that cgroup), monitoring (state transitions for services are echoed on irc to me), logging (all output will end up in queryable journal and syslog) and such, so highly recommend not going the "simple" bash-way here.

Complimentary session units generally look like this (Xvfb.service):

[Service]
SyslogIdentifier=%p
ExecStart=/usr/bin/Xvfb $DISPLAY -screen 0 800x600x16

And with systemct start skype-desktop, nice (but depressingly empty) fluxbox desktop is now accessible over ssh+vnc (don't trust vnc enough to run it on non-localhost, plus should be rarely needed anyway):

% ssh -L 5900:localhost:5900 user@host &
% vncclient localhost

Getting skype to run on the target host was a bit more difficult than I've expected though - local x86 machine has -march=native in CFLAGS and core-i3 cpu, so just copying binaries/libs resulted in a predictable:

[271817.608818] traps: ld-linux.so.2[7169]
        trap invalid opcode ip:f77dad60 sp:ffb91860 error:0 in ld-linux.so.2[f77c6000+20000]

Fortunately, there're always generic-arch binary distros, so had to spin up a qemu with ubuntu livecd iso, install skype there and run the same collect-all-the-deps script.

Basically, what's needed for skype to run is it's own data/media files ("/opt/skype", "/usr/share/skype"), binary ("/usr/lib/skype", "/opt/skype/skype") and all the so's it's linked against.

Yay!

So, to the API-to-IRC scripts then... probably in the next entry, as I get to these myself. Also following might be revised apparmor profile for such setup and maybe a script to isolate the whole thing even further into namespaces (which is interesting thing to try, but not sure how it might be useful yet with LSM already in place).

All the interesting stuff for the whole endeavor can be found in the ad-hoc repo I've created for it: https://github.com/mk-fg/skype-space