My blog_title_here

PyParsing vs Yapps

As I've been decompiling dynamic E config in the past anyway, wanted to back it up to git repo along with the rest of them.

e.cfg looks like this:

group "E_Config" struct {
  group "xkb.used_options" list {
    group "E_Config_XKB_Option" struct {
      value "name" string: "grp:caps_toggle";
    }
  }
  group "xkb.used_layouts" list {
    group "E_Config_XKB_Layout" struct {
      value "name" string: "us";
...

Simple way to make it "canonical" is just to order groups/values there alphabetically, blanking-out some transient ones.

That needs a parser, and while regexps aren't really suited to that kind of thing, pyparsing should work:

number = pp.Regex(r'[+-]?\d+(\.\d+)?')
string = pp.QuotedString('"') | pp.QuotedString("'")
value_type = pp.Regex(r'\w+:')
group_type = pp.Regex(r'struct|list')

value = number | string
block_value = pp.Keyword('value')\
  + string + value_type + value + pp.Literal(';')

block = pp.Forward()
block_group = pp.Keyword('group') + string\
  + group_type + pp.Literal('{') + pp.OneOrMore(block) + pp.Literal('}')
block << (block_group | block_value)

config = pp.StringStart() + block + pp.StringEnd()

Fun fact: this parser doesn't work.

Bails with some error in the middle of the large (~8k lines) real-world config, while working for all the smaller pet samples.

I guess some buffer size must be tweaked (kinda unusual for python module though), maybe I made a mistake there, or something like that.

So, yapps2-based parser:

parser eet_cfg:
  ignore: r'[ \t\r\n]+'
  token END: r'$'
  token N: r'[+\-]?[\d.]+'
  token S: r'"([^"\\]*(\\.[^"\\]*)*)"'
  token VT: r'\w+:'
  token GT: r'struct|list'

  rule config: block END {{ return block }}

  rule block: block_group {{ return block_group }}
    | block_value {{ return block_value }}
  rule block_group:
    'group' S GT r'\{' {{ contents = list() }}
    ( block {{ contents.append(block) }} )*
    r'\}' {{ return Group(S, GT, contents) }}

  rule value: S {{ return S }} | N {{ return N }}
  rule block_value: 'value' S VT value ';' {{ return Value(S, VT, value) }}

Less verbose (even with more processing logic here) and works.

Embedded in a python code (doing the actual sorting), it all looks like this (might be useful to work with E configs, btw).

yapps2 actually generates quite readable code from it, and it was just simpler (and apparently more bugproof) to write grammar rules in it.

ymmv, but it's a bit of a shame that pyparsing seem to be the about the only developed parser-generator of such kind for python these days.

Had to package yapps2 runtime to install it properly, applying some community patches (from debian package) in process and replacing some scary cli code from 2003. Here's a fork.

Migrating configuration / settings to E17 (enlightenment) 0.17.0 from older E versions

It's a documented feature that 0.17.0 release (even if late pre-release version was used before) throws existing configuration out of the window.

E is a bit special (at least among wm's - fairly sure some de's do similar things as well) in that it keeps its settings on disk compiled and compressed (with eet) - but it's much easier to work with than it might sound like at first.

So, to get the bits of config migrated, one just has to pull the old (pre-zero) config out, then start zero-release e to generate new config, decompile both of these, pull compatible bits from old into the new one, then compile it and put back into "~/.e/e/config"

Before zero update, config can be found in "~/.e/e/config/standard/e.cfg"

If release version was started already and dropped the config, then old one should be "~/.e/e/config/standard/e.1.cfg" (or any different number instead of "1" there, just mentally substitute it in examples below).

Note that "standard" there is a profile name, if it might be called differently, check "~/.e/e/config/profile.cfg" (profile name should be readable there, or use "eet -x ~/.e/e/config/profile.cfg config").

"eet -d ~/.e/e/config/standard/e.cfg config" should produce perfectly readable version of the config to stdout.

Below is how I went about the whole process.

Make a repository to track changes (will help if the process might take more merge-test iterations than one):

% mkdir e_config_migration
% cd e_config_migration
% git init

Before zero update:

% cp ~/.e/e/config/standard/e.cfg e_pre_zero
% eet -d e_pre_zero config > e_pre_zero.cfg

Start E-release (wipes the config, produces a "default" new one there).

% cp ~/.e/e/config/standard/e.cfg e_zero
% eet -d e_zero config > e_zero.cfg
% git add e_*
% git commit -m "Initial pre/post configurations"
% emacs e_pre_zero.cfg e_zero.cfg

Then copy all the settings that were used in any way to e_zero.cfg.

I copied pretty much all the sections with relevant stuff, checking that the keys in them are the same - and they were, but I've used 0.17.0alpha8 before going for release, so if not, I'd just try "filling the blanks", or, failing that, just using old settings as a "what has to be setup through settings-panel" reference.

To be more specific - "xkb" options/layouts (have 2 of them setup), shelves/gadgets (didn't have these, and was lazy to click-remove existing ones), "remembers" (huge section, copied all of it, worked!), all "bindings" (pain to setup these).

After all these sections, there's a flat list of "value" things, which turned out to contain quite a lot of hard-to-find-in-menus parameters, so here's what I did:

• copy that list (~200 lines) from old config to some file - say, "values.old", then from a new one to e.g. "values.new".
• sort -u values.old > values.old.sorted; sort -u values.new > values.new.sorted
• diff -uw values.{old,new}.sorted

After all the changes:

% eet -e e_zero config e_zero.cfg 1
% git commit -a -m Merged-1
% cp e_zero ~/.e/e/config/standard/e.cfg
% startx

New config worked for me for all the changes I've made - wasn't sure if I can copy *that* much from the start, but it turned out that almost no reconfiguration was necessary.

Caveat is, of course, that you should know what you're doing here, and be ready to handle issues / rollback, if any, that's why putting all these changes in git might be quite helpful.

Terms of Service - Didn't Read

Right now I was working on python-skydrive module and further integration of MS SkyDrive into tahoe-lafs as a cloud backend, to keep the stuff you really care about safe.

And even if you don't trust SkyDrive to keep stuff safe, you still have to register your app with these guys, especially if it's an open module, because "You are solely and entirely responsible for all uses of Live Connect occurring under your Client ID." and it's unlikely that a generic python interface author will vouch for all it's uses like that.

What do "register app" mean? Agreeing to yet another "Terms of Service", of course!

Do anyone ever reads these?

I hardly care enough to read, but there's an app for exactly that, and it's relatively well-known by now.

http://www.indiegogo.com/terms-of-service-didnt-read

A new toy to play with - TI Launchpad with MSP430 MCU

A friend gave me this thing to play with (and eventually adapt to his purposes).

What's interesting here is that TI seem to give these things out for free.

Seriously, a box with a debug/programmer board and two microcontroller chips (which are basically your programmable computer with RAM, non-volatile flash memory, lots of interfaces, temp sensor, watchdog, etc that can be powered from 2 AA cells), to any part of the world with FedEx for a beer's worth - $5.

Guess it's time to put a computer into every doorknob indeed.

Unhosted remoteStorage idea

• Works with multiple clients, i.e. everyone has it's own state.
• Server doesn't have to store any data for possible-infinite number of clients, not even session or login data.
• Same pages still can be served to all clients, some will just hide unwanted content.
• Previous point leads to pages being very cache-friendly.
• No need to "recognize" client in any way, which is commonly acheived with authentication.
• No interation of "write" kind with the server means much less potential for abuse (DDoS, spam, other kinds of exploits).

Flip side of that rosy picture is that localStorage only works in one browser (or possibly several synced instances), which is quite a drag, because one advantage of a web-based reader is that it can be accessed from anywhere, not just single platform, where you might as well install specialized app.

Recently, however, remoteStorage project from unhosted group has caught my attention.

But aside for short-term "problem solved" thing, it's really nice to see such movements out there, even though whole stack of market incentives (which heavily favors control over data, centralization and monopolies) is against them.

Proper(-ish) way to start long-running systemd service on udev event (device hotplug)

Update 2015-01-12: There's a follow-up post with a different way to do that, enabled by "systemd-escape" tool available in more recent systemd versions.

RUN
...
Starting daemons or other long running processes is not appropriate for
udev; the forked processes, detached or not, will be unconditionally killed
after the event handling has finished.

I think it's for the best - less accumulating cruft and unmanageable pids forked from udevd, but unfortunately it also breaks existing udev rule-files, the ones which use RUN+="..." to do just that.

One of the most obvious breakage for me was the smartcard failing, so decided to fix that. Documentation on the whole migration process is somewhat lacking (hence this post), even though docs on all the individual pieces are there (which are actually awesome).

In my case, when device (usb smartcard token) get plugged, ifdhandler process should be started via openct-control (OpenCT sc middleware), which then creates unix socket through which openct libraries (used in turn by OpenSC PKCS#11 or PCSClite) can access the hardware.

So, basically I've had something like this (there are more rules for different hw, of course, but for the sake of clarity...):

SUBSYSTEM!="usb", GOTO="openct_rules_end"
ACTION!="add", GOTO="openct_rules_end"
PROGRAM="/bin/sleep 0.1"
...
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device",\
  ENV{ID_VENDOR_ID}=="0529", ENV{ID_MODEL_ID}=="0600",\
  GROUP="usb",\
  RUN+="/usr/sbin/openct-control attach usb:$env{PRODUCT} usb $env{DEVNAME}"
...
LABEL="openct_rules_end"

Instead of RUN here, ENV{SYSTEMD_WANTS} can be used to start a properly-handled service, but note that some hardware parameters are passed from udev properties and in general systemd unit can't reference these.

I.e. if just ENV{SYSTEMD_WANTS}="openct-handler.service" (or more generic smartcard.target) is started, it won't know which device to pass to "openct-control attach" command.

One way might be storing these parameters in some dir, where they'll be picked by some path unit, a bit more hacky way would be scanning usb bus in the handler, and yet another one (which I decided to go along with) is to use systemd unit-file templating to pass these parameters.

openct-handler@.service:

[Unit]
Requires=openct.service

[Service]
Type=forking
GuessMainPID=false
ExecStart=/bin/sh -c "exec openct-control attach %I"

Note that it requires openct.service, which is basically does "openct-control init" once per boot to setup paths and whatnot:

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/openct-control init
ExecStop=/usr/sbin/openct-control shutdown

[Install]
WantedBy=multi-user.target

Finally, udev rules file for the device:

SUBSYSTEM=="usb", ACTION="add", ENV{DEVTYPE}=="usb_device", \
  ENV{ID_VENDOR_ID}=="0529", ENV{ID_MODEL_ID}=="0600", \
  GROUP="usb", TAG+="systemd", \
  ENV{SYSTEMD_WANTS}="openct-handler@\
    usb:$env{ID_VENDOR_ID}-$env{ID_MODEL_ID}-$env{ID_REVISION}\
    \x20usb\x20-dev-bus-usb-$env{BUSNUM}-$env{DEVNUM}.service"

(I highly doubt newline escaping in ENV{SYSTEMD_WANTS} above will work - added them just for readability, so pls strip these in your mind to a single line without spaces)

Systemd escaping in the rule above is described in systemd.unit(5) and produces a name - and start a service - like this one:

openct-handler@usb:0529-0600-0100\x20usb\x20-dev-bus-usb-002-003.service

Which then invokes:

sh -c "exec openct-control attach\
  usb:0529/0600/0100 usb /dev/bus/usb/002/003"

And it forks ifdhandler process, which works with smartcard from then on.

ifdhandler seem to be able to detect unplugging events and exits gracefully, but otherwise BindTo= unit directive can be used to stop the service when udev detects that device is unplugged.

Note that it might be more obvious to just do RUN+="systemctl start whatever.service", but it's a worse way to do it, because you don't bind that service to a device in any way, don't produce the "whatever.device" unit and there are lot of complications due to systemctl being a tool for the user, not the API proper.

Late adventures with time-series data collection and representation

When something is wrong and you look at the system, most often you'll see that... well, it works. There's some cpu, disk, ram usage, some number of requests per second on different services, some stuff piling up, something in short supply here and there...

And there's just no way of telling what's wrong without answers to the questions like "so, what's the usual load average here?", "is the disk always loaded with requests 80% of time?", "is it much more requests than usual?", etc, otherwise you might be off to some wild chase just to find out that load has always been that high, or solve the mystery of some unoptimized code that's been there for ages, without doing anything about the problem in question.

Historical data is the answer, and having used rrdtool with stuff like (customized) cacti and snmpd (with some my hacks on top) in the past, I was overjoyed when I stumbled upon a graphite project at some point.

Initially, I started using it with just collectd (and still using it), but there's a need for something to convert metric names to a graphite hierarcy.

After looking over quite a few solutions to collecd-carbon bridge, decided to use bucky, with a few fixes of my own and quite large translation config.

Some of the problems I've noticed with such collectd setup:

• Disk I/O metrics are godawful or just doesn't work - collected metrics of read/write either for processes of device are either zeroes, have weird values detached from reality (judging by actual problems and tools like atop and sysstat provide) or just useless.
• Lots of metrics for network and memory (vmem, slab) and from various plugins have naming, inconsistent with linux /proc or documentation names.
• Some useful metrics that are in, say, sysstat doesn't seem to work with collectd, like sensor data, nfsv4, some paging and socket counters.
• Some metrics need non-trivial post-processing to be useful - disk utilization % time is one good example.
• Python plugins leak memory on every returned value. Some plugins (ping, for example) make collectd segfault several times a day.
• One of the most useful info is the metrics from per-service cgroup hierarchies, created by systemd - there you can compare resource usage of various user-space components, totally pinpointing exactly what caused the spikes on all the other graphs at some time.
• Second most useful info by far is produced from logs and while collectd has a damn powerful tail plugin, I still found it to be too limited or just too complicated to use, while simple log-tailing code does the better job and is actually simpler due to more powerful language than collectd configuration. Same problem with table plugin and /proc.
• There's still a need for lagre post-processing chunk of code and pushing the values to carbon.

So off with collectd for all the custom metrics.

Wrote a simple "while True: collect_and_send() && sleep(till_deadline);" loop in python, along with the cgroup data collectors (there are even proper "block io" and "syscall io" per-service values!), log tailer and sysstat data processor (mainly for disk and network metrics which have batshit-crazy values in collectd plugins).

Pity, gmond looked like a nice, solid and resilent thing otherwise.

I still like the idea to pipe graphite metrics through AMQP (like rocksteady does), routing them there not only to graphite, but also to some proper threshold-monitoring daemon like shinken (basically nagios, but distributed and more powerful), with alerts, escalations, trending and flapping detection, etc, but most of the existing solutions all seem to use graphite and whisper directly, which seem kinda wasteful.

Looking forward, I'm actually deciding between replacing collectd completely for a few most basic metrics it now collects, pulling them from sysstat or just /proc directly or maybe integrating my collectors back into collectd as plugins, extending collectd-carbon as needed and using collectd threshold monitoring and matches/filters to generate and export events to nagios/shinken... somehow first option seem to be more effort-effective, even in the long run, but then maybe I should just work more with collectd upstream, not hack around it.

Phasing out fossil completely

Also, I'm torn at this point between the thoughts along the lines "selection of modern DVCS spoil us" against "damn, why they there is no clear popular + works-for-everything thing", but it's probably normal, as I have (or had) similar thoughts about lot of technologies.

On github as well now

Following another hiatus from a day job, I finally have enough spare time to read some of the internets and do something about them.

So I thought to gradually push all this stuff out to github and/or bitbucket (still need to learn or at least look at hg for that!). github being the most obvious and easiest choice, just created a few repos there and started the migration. More to come.

Effort should be somewhat justified by the fact that such repos will be geo-distributed backups, shareable links and I can learn more SCM internals by the way.

For now - me on github.

Running stuff like firefox, flash and skype with apparmor

Should've done it a long time ago, actually. I was totally sure it'd be much harder task, but then recently I've had some spare time and decided to do something about this binary crap, and looking for possible solutions stumbled upon apparmor.

Case in point - skype apparmor profile, which doesn't allow it to access anything but ~/.Skype, /opt/skype and a few other system-wide things:

#include <tunables/global>
/usr/bin/skype {
  #include <abstractions/base>
  #include <abstractions/user-tmp>
  #include <abstractions/pulse>
  #include <abstractions/nameservice>
  #include <abstractions/ssl_certs>
  #include <abstractions/fonts>
  #include <abstractions/X>
  #include <abstractions/freedesktop.org>
  #include <abstractions/kde>
  #include <abstractions/site/base>
  #include <abstractions/site/de>

  /usr/bin/skype mr,
  /opt/skype/skype pix,
  /opt/skype/** mr,
  /usr/share/fonts/X11/** m,

  @{PROC}/*/net/arp r,
  @{PROC}/sys/kernel/ostype r,
  @{PROC}/sys/kernel/osrelease r,

  /dev/ r,
  /dev/tty rw,
  /dev/pts/* rw,
  /dev/video* mrw,

  @{HOME}/.Skype/ rw,
  @{HOME}/.Skype/** krw,

  deny @{HOME}/.mozilla/ r, # no idea what it tries to get there
  deny @{PROC}/[0-9]*/fd/ r,
  deny @{PROC}/[0-9]*/task/ r,
  deny @{PROC}/[0-9]*/task/** r,
}

"deny" lines here are just to supress audit warnings about this paths, everything is denied by default, unless explicitly allowed.

Compared to "default" linux DAC-only "as user" confinement, where it has access to all your documents, activities, smartcard, gpg keys and processes, ssh keys and sessions, etc - it's a huge improvement.