My blog_title_here

X-Y plots of d3 scales and counter-intuitive domain/range effect

As I was working on a small d3-heavy project (my weird firefox homepage), I did use d3 scales for things like opacity of the item, depending on its relevance, and found these a bit counter-intuitive, but with no readily-available demo (i.e. X-Y graphs of scales with same fixed domain/range) on how they actually work.

Basically, I needed this:

I'll be first to admit that I'm no data scientist and not particulary good at math, but from what memories on the subject I have, intuition tells me that e.g. "d3.scale.pow().exponent(4)" should rise waaaay faster from the very start than "d3.scale.log()", but with fixed domain + range values, that's exactly the opposite of truth!

Here, it's obvious that if you want to pick something that mildly favors higher X values, you'd pick pow(2), and not sqrt.

Feel like such chart should be in the docs, but don't feel qualified enough to add it, and maybe it's counter-intuitive just for me, as I don't dabble with data visualizations much and/or might be too much of a visually inclined person.

In case someone needs the script to do the plotting (it's really trivial though): scales_graph.zip

My Firefox Homepage

Wanted to have some sort of "homepage with my fav stuff, arranged as I want to" in firefox for a while, and finally got resolve to do something about it - just finished a (first version of) script to generate the thing - firefox-homepage-generator.

Default "grid of page screenshots" never worked for me, and while there are other projects that do other layouts for different stuff, they just aren't flexible enough to do whatever horrible thing I want.

In this particular case, I wanted to experiment with chaotic tag cloud of bookmarks (so they won't ever be in the same place), relations graph for these tags and random picks for "links to read" from backlog.

Result is a dynamic d3 + d3.layout.cloud (interactive example of this layout) page without much style:

Started using FF bookmarks again in a meaningful way only recently, so not much stuff there yet, but it does seem to help a lot, especially with these handy awesome bar tricks.

Not entirely sure how useful the cloud visualization or actually having a homepage would be, but it's a fun experiment and a nice place to collect any useful web-surfing-related stuff I might think of in the future.

Repo link: firefox-homepage-generator

Shadowrun: Dragonfall

Conky eye candy clocks and meters

So my laptop broke anyway, but on the bright side - I've got fairly large (certainly by my display standards) desktop fullhd screen now.

While restoring OS there, decided to update ~/.conkyrc (see conky), as it was kinda small for this larger screen, so why not put some eye-candy there, while at it?

Leftmost radial meters show (inner-to-outer) clock with hands and rings right next to them, blue-ish cpu arcs (right-bottom, outer one is load summary, inner ones are per-core), used (non-cache) memory/swap (left), network traffic (top-right, green/red arcs for up/down) and / and /home df arcs (outer top).

On the right it's good ol' binary clock.

Whole script to draw the things can be found in de-setup repo on gh along with full conkyrc I currently use.

Software hacks to fix broken hardware - laptop fan

Had a fan in a laptop dying for a few weeks now, but international mail being universally bad (and me too hopeful about dying fan's lifetime), replacement from ebay is still on its looong way.

Meanwhile, thing started screeching like mad, causing strong vibration in the plastic and stopping/restarting every few seconds with an audible thunk.

Things not looking good, and me being too lazy to work hard enough to be able to afford new laptop, had to do something to postpone this one's imminent death.

Cleaning the dust and hairs out of fan's propeller and heatsink and changing thermal paste did make the thing a bit cooler, but given that it's fairly slim Acer S3 ultrabook, no local repair shop was able to offer any immediate replacement for the fan, so no clean hw fix in reach yet.

Interesting thing about broken fans though, is that they seem to start vibrating madly out of control only beyond certain speed, so one option was to slow the thing down, while keeping cpu cool somehow.

cpupower tool that comes with linux kernel can nicely downclock this i5 cpu to 800 MHz, but that's not really enough to keep fan from spinning madly - some default BIOS code seem to be putting it to 100% at 50C.

Besides, from what I've seen, it seem to be quite counter-productive, making everything (e.g. opening page in FF) much longer, keeping cpu at 100% of that lower rate all the time, which seem to heat it up slower, sure, but to the same or even higher level for the same task (e.g. opening that web page), with side effect being also wasting time.

As acers3fand script linked from the wiki was for a bit different purpose, wrote my own (also lighter and more self-contained) script - fan_control to only put more than ~50% of power to it after it goes beyond 60C and warns if it heats up way more without putting the fan into "wailing death" mode ever, with max being at about 75% power, also reaching for cpupower hack before that.

Such manual control opens up a possibility of cpu overheating though, or otherwise doesn't help much when you run cpu-intensive stuff, and I kinda don't want to worry about some cronjob, stuck dev script or hung DE app killing the machine while I'm away, so one additional hack I could think of is to just throttle CPU bandwidth enough so that:

• short tasks complete at top performance, without delays.
• long cpu-intensive stuff gets throttled to a point where it can't generate enough heat and cpu stays at some 60C with slow fan speed.
• some known-to-be-intensive tasks like compilation get their own especially low limits.

So kinda like cpupower trick, but more fine-grained and without fixed presets one can slow things down to (as lowest bar there doesn't cut it).

Kernel Control Groups (cgroups) turned out to have the right thing for that - "cpu" resource controller there has cfs_quote_us/cfs_period_us knobs to control cpu bandwidth for threads within a specific cgroup.

For distinct limits on daemons started from cli, there's "systemd-run" tool these days, and for more proper interactive wrapping, I've had pet cgroup-tools scripts for a while now (to limit cpu priority of heavier bg stuff like builds though).

With that last tweak, situation seem to be under control - no stray app can really kill the cpu and fan doesn't have to do all the hard work to prevent it either, seemingly solving that hardware fail with software measures for now.

Now, if only replacement fan got here before I drop off the nets even with these hacks.

FAT32 driver in python

Wrote a driver for still common FAT32 recently, while solving the issue with shuffle on cheap "usb stick with microsd slot" mp3 player.

It's kinda amazing how crappy firmware in these things can be.

Guess one should know better than to get such crap with 1-line display, gapful playback, weak battery, rewind at non-accelerating ~3x speed, no ability to pick tracks while playing and plenty of other annoying "features", but the main issue I've had with the thing by far is missing shuffle functionality - it only plays tracks in static order in which they were uploaded (i.e. how they're stored on fs).

Seems like whoever built the thing made it deliberately hard to shuffle the tracks offline - just one sort by name would've made things a lot easier, and it's clear that device reads the full dir listing from the time it spends opening dirs with lots of files.

Most obvious way to do such "offline shuffle", given how the thing orders files, is to re-upload tracks in different order, which is way too slow and wears out flash ram.

Second obvious for me was to dig into FAT32 and just reorder entries there, which is what the script does.

It's based off example of a simpler fat16 parser in construct module repo and processes all the necessary metadata structures like PDRs, FATs (cluster maps) and directory tables with vfat long-name entries inside.

Given that directory table on FAT32 is just an array (with vfat entries linked to dentries after them though), it's kinda easy just to shuffle entries there and write data back to clusters from where it was read.

One less obvious solution to shuffle, coming from understanding how vfat lfn entries work, is that one can actually force fs driver to reorder them by randomizing filename length, as it'll be forced to move longer entries to the end of the directory table.

But that idea came a bit too late, and such parser can be useful for extending FAT32 to whatever custom fs (with e.g. FUSE or 9p interface) or implementing some of the more complex hacks.

It's interestng that fat dentries can (and apparently known to) store unix-like modes and uid/gid instead of some other less-useful attrs, but linux driver doesn't seem to make use of it.

OS'es also don't allow symlinks or hardlinks on fat, while technically it's possible, as long as you keep these read-only - just create dentries that point to the same cluster.

Should probably work for both files and dirs and allow to create multiple hierarchies of the same files, like several dirs where same tracks are shuffled with different seed, alongside dirs where they're separated by artist/album/genre or whatever other tags.

It's very fast and cheap to create these, as each is basically about "(name_length + 32B) * file_count" in size, which is like just 8 KiB for dir structure holding 100+ files.

So plan is to extend this small hack to use mutagen to auto-generate such hierarchies in the future, or maybe hook it directly into beets as an export plugin, combined with transcoding, webui and convenient music-db there.

Also, can finally tick off "write proper on-disk fs driver" from "things to do in life" list ;)

Encrypted root on a remote vds

Most advice wrt encryption on a remote hosts (VPS, VDS) don't seem to involve full-disk encryption as such, but is rather limited to encrypting /var and /home, so that machine will boot from non-crypted / and you'll be able to ssh to it, decrypt these parts manually, then start services that use data there.

That seem to be in contrast with what's generally used on local machines - make LUKS container right on top of physical disk device, except for /boot (if it's not on USB key) and don't let that encryption layer bother you anymore.

Two policies seem to differ in that former one is opt-in - you have to actively think which data to put onto encrypted part (e.g. /etc/ssl has private keys? move to /var, shred from /etc), while the latter is opt-out - everything is encrypted, period.

So, in spirit of that opt-out way, I thought it'd be a drag to go double-think wrt which data should be stored where and it'd be better to just go ahead and put everything possible to encrypted container for a remote host as well, leaving only /boot with kernel and initramfs in the clear.

Searching around for someone to tie the two things up, found a bit incomplete and non-packaged solutions like this RH enhancement proposal and a set of hacky scripts and instructions in dracut-crypt-wait repo on bitbucket.

I thought first way was rather more elegant and clever, allowing dracut to figure out which device to decrypt and start cryptsetup with all the necessary, configured and documented parameters, also still allowing to type passphrase into console - best of both worlds, so went along with that one, creating dracut-crypt-sshd project.

As README there explains, using it is as easy as adding it into dracut.conf (or passing to dracut on command line) and adding networking to grub.cfg, e.g.:

menuentry "My Linux" {
        linux /vmlinuz ro root=LABEL=root
                rd.luks.uuid=7a476ea0 rd.lvm.vg=lvmcrypt rd.neednet=1
                ip=88.195.61.177::88.195.61.161:255.255.255.224:myhost:enp0s9:off
        initrd /dracut.xz
}

("ip=dhcp" might be simpler way to go, but doesn't yield default route in my case)

And there, you'll have sshd on that IP port 2222 (configurable), with pre-generated (during dracut build) keys, which might be a good idea to put into "known_hosts" for that ip/port somewhere. "authorized_keys" is taken from /root/.ssh by default, but also configurable via dracut.conf, if necessary.

Apart from sshd, that module includes two tools for interaction with console - console_peek and console_auth (derived from auth.c in the bugzilla link above).

Logging in to that sshd then yields sequence like this:

[214] Aug 08 13:29:54 lastlog_perform_login: Couldn't stat /var/log/lastlog: No such file or directory
[214] Aug 08 13:29:54 lastlog_openseek: /var/log/lastlog is not a file or directory!

# console_peek
[    1.711778] Write protecting the kernel text: 4208k
[    1.711875] Write protecting the kernel read-only data: 1116k
[    1.735488] dracut: dracut-031
[    1.756132] systemd-udevd[137]: starting version 206
[    1.760022] tsc: Refined TSC clocksource calibration: 2199.749 MHz
[    1.760109] Switching to clocksource tsc
[    1.809905] systemd-udevd[145]: renamed network interface eth0 to enp0s9
[    1.974202] 8139too 0000:00:09.0 enp0s9: link up, 100Mbps, full-duplex, lpa 0x45E1
[    1.983151] dracut: sshd port: 2222
[    1.983254] dracut: sshd key fingerprint: 2048 0e:14:...:36:f9  root@congo (RSA)
[    1.983392] dracut: sshd key bubblebabble: 2048 xikak-...-poxix  root@congo (RSA)
[185] Aug 08 13:29:29 Failed reading '-', disabling DSS
[186] Aug 08 13:29:29 Running in background
[    2.093869] dracut: luksOpen /dev/sda3 luks-...
Enter passphrase for /dev/sda3:
[213] Aug 08 13:29:50 Child connection from 188.226.62.174:46309
[213] Aug 08 13:29:54 Pubkey auth succeeded for 'root' with key md5 0b:97:bb:...

# console_auth
Passphrase:

#

Probably obligatory disclaimer for such post should mention that crypto above won't save you from malicious hoster or whatever three-letter-agency that will coerce it into cooperation, should it take interest in your poor machine - it'll just extract keys from RAM image (especially if it's a virtualized VPS) or backdoor kernel/initramfs and force a reboot.

Threat model here is more trivial - be able to turn off and decomission host without fear of disks/images then falling into some other party's hands, which might also happen if hoster eventually goes bust or sells/scraps disks due to age or bad blocks.

Also, even minor inconvenience like forcing to extract keys like outlined above might be helpful in case of quite well-known "we came fishing to a datacenter, shut everything down, give us all the hardware in these racks" tactic employed by some agencies.

Absolute security is a myth, but these measures are fairly trivial and practical to be employed casually to cut off at least some number of basic threats.

So, yay for dracut, the amazingly cool and hackable initramfs project, which made it that easy.

Code link: https://github.com/mk-fg/dracut-crypt-sshd

cjdns per-IP (i.e. per-peer) traffic accounting

I've been using Hyperboria darknet for about a month now, and after late influx of russian users there (after this article) got plently of peers, so node is forwarding a bit of network traffic.

Being a dorknet-proper, of course, you can't see what kind of traffic it is or to whom it goes (though cjdns doesn't have anonymity as a goal), but I thought it'd be nice to at least know when my internet lags due to someone launching DoS flood or abusing torrents.

Over the Internet (called "clearnet" here), cjdns peers using udp, but linux conntrack seem to be good enough to track these "connections" just as if they were stateful tcp flows.

Simple-ish traffic accounting on vanilla linux usually boils down to ulogd2, which can use packet-capturing interfaces (raw sockets via libpcap, netfilter ULOG and NFLOG targets), but it's kinda heavy-handed here - traffic is opaque, only endpoints matter, so another one of its interfaces seem to be better option - conntrack tables/events.

Handy conntrack-tools (or /proc/net/{ip,nf}_conntrack) cat track all the connections, including simple udp-based ones (like cjdns uses), producing entries like:

udp 17 179 \
        src=110.133.5.117 dst=188.226.51.71 sport=52728 dport=8131 \
        src=188.226.51.71 dst=110.133.5.117 sport=8131 dport=52728 \
        [ASSURED] mark=16 use=1

First trick is to enable the packet/byte counters there, which is a simple, but default-off sysctl knob:

# sysctl -w net.netfilter.nf_conntrack_acct=1

That will add "bytes=" and "packets=" values there for both directions.

Of course, polling the table is a good way to introduce extra hangs into system (/proc files are basically hooks that tend to lock stuff to get consistent reads) and loose stuff in-between polls, so luckily there's an event-based netlink interface and ulogd2 daemon to monitor that.

One easy way to pick both incoming and outgoing udp flows in ulogd2 is to add connmarks to these:

-A INPUT -p udp --dport $cjdns_port -j CONNMARK --set-xmark 0x10/0x10
-A OUTPUT -p udp --sport $cjdns_port -j CONNMARK --set-xmark 0x10/0x10

Then setup filtering by these in ulogd.conf:

...

stack=log:NFCT,mark:MARK,ip2str:IP2STR,print:PRINTFLOW,out:GPRINT

[log]
accept_proto_filter=udp

[mark]
mark=0x10
mask=0x10

[out]
file="/var/log/ulogd2/cjdns.log"

Should produce parseable log of all the traffic flows with IPs and such.

Fairly simple script can then be used to push this data to graphite, munin, ganglia, cacti or whatever time-series graphing/processing tool. Linked script is for graphite "carbon" interface.

Firefox - breaking free of webdevs' tyranny

Wanted to share three kinda-big-deal fixes I've added to my firefox:

• Patch to remove sticky-on-top focus-grabbing "Do you want to activate plugins on this page?" popup.
• Patch to prevent plugins (e.g. Abode Flash) from ever grabbing firefox hotkeys like "Ctrl + w" (close tab) or F5, forcing to do click outside e.g. YouTube video window to get back to ff.
• Easy "toggle js" fix for JavaScript on pages grabbing controls like keyboard and mouse (e.g. overriding F5 to retweet instead of reload page, preventing copy-paste if forms and on pages, etc).

Lately, firefox seem to give more-and-more control into the hands of web developers, who seem to be hell-bent on abusing that to make browsing UX a living hell.

FF bug-reports about Flash grabbing all the focus date back to 2001 and are unresolved still.

Sites override Up/Down, Space, PgUp/PgDown, F5, Ctrl+T/W I've no idea why - guess some JS developers just don't use keyboard at all, which is somewhat understandable, combined with the spread of tablet-devices these days.

Overriding clicks in forms to prevent pasting email/password seem to be completely ignoring valid (or so I think) use-case of using some storage app for these.

And native "click-to-play" switch seem to be hilariously unusable in FF, giving cheerful "Hey, there's flash here! Let me pester you with this on every page load!" popups.

All are known, neither one seem to be going away anytime soon, so onwards to the fixes.

var prefs = Components.classes['@mozilla.org/preferences-service;1']
    .getService(Components.interfaces.nsIPrefBranch),
  state = prefs.getBoolPref('javascript.enabled');
prefs.setBoolPref('javascript.enabled', !state);

That's the whole thing, bound to something like Ctrl+\ (the one above Enter here), makes a nice "Turbo and Get Off My JS" key. Fairly sure there are addons that allow to toggle prefs ("javascript.enabled" above) via keys without needing any code, but I have this one.

Damn glad there are open-source (and uglifyjs-like) browsers like that, hope proprietary google-ware won't take over the world in the nearest future.

Mentioned patches are available in (and integrated with-) the firefox-nightly exheres in my repo, forked off awesome sardemff7-pending firefox-scm.exheres-0 / mozilla-app.exlib work.

Recent fixes to great tools - 0bin and Convergence

I've tried both of these in the past, but didn't have attention budget to make them really work for me - which finally found now, so wanted to also give crawlers a few more keywords on these nice things.

0bin - leak-proof pastebin

As I pastebin a lot of stuff all the time - basically everything multiline - because all my IM happens in ERC over IRC (with bitlbee linking xmpp and all the proprietary crap like icq, skype and twitter), and IRC doesn't handle multiline messages at all.

All sorts of important stuff ends up there - some internal credentials, contacts, non-public code, bugs, private chat logs, etc - so I always winced a bit when pasting something in fear that google might index/data-mine it and preserve forever, so I figured it'll bite me eventually, somewhat like this:

Easy and acceptable solution is to use simple client-side crypto, with link having decryption key after hashmark, which never gets sent to pastebin server and doesn't provide crawlers with any useful data. ZeroBin does that.

But original ZeroBin is php, which I don't really want to touch, and have its share of problems - from the lack of command-line client (for e.g. grep stuff log | zerobinpaste), to overly-long urls and flaky overloaded interface.

Luckily, there's more hackable python version of it - 0bin, for which I hacked together a simple zerobinpaste tool, then simplified interface to bare minimum and updated to use shorter urls (#41, #42) and put to my host - result is paste.fraggod.net - my own nice robot-proof pastebin.

URLs there aren't any longer than with regular pastebins:

http://paste.fraggod.net/paste/pLmEb0BI#Verfn+7o

Plus the links there expire reliably, and it's easy to force this expiration, having control over app backend.

Local fork should have all the not-yet-merged stuff as well as the non-upstreamable simpler white-bootstrap theme.