My blog_title_here

Info feeds

Thanks to feedjack, I'm able to keep in sync with 120 feeds (many of them, like slashdot or reddit, being an aggregates as well), as of today. Quite a lot of stuff I couldn't even imagine handling a year ago, and a good aggregator definitely helps, keeping all the info just one click away.

And every workstation-based (desktop) aggregator I've seen is a fail:

• RSSOwl. Really nice interface and very powerful. That said, it eats more ram than a firefox!!! Hogs CPU till the whole system stutters, and eats more of it than every other app I use combined (yes, including firefox). Just keeping it in the background costs 20-30% of dualcore cpu. Changing "show new" to "show all" kills the system ;)
• liferea. Horribly slow, interface hangs on any action (like fetching feed "in the background"), hogs cpu just as RSSOwl and not quite as feature-packed.
• Claws-mail's RSSyl. Quite nice resource-wise and very responsive, unlike dedicated software (beats me why). Pity it's also very limited interface-wise and can't reliably keep track of many of feeds by itself, constantly loosing a few if closed non-properly (most likely it's a claws-mail fault, since it affects stuff like nntp as well).
• Emacs' gnus and newsticker. Good for a feed or two, epic fail in every way with more dozen of them.
• Various terminal-based readers. Simply intolerable.

Maybe someday I'll get around to use something like Google Reader, but it's still one hell of a mess, and it's no worse than similar web-based services out there. So much for the cloud services. *sigh*

Distributed fault-tolerant fs take 2: MooseFS

Ok, almost one month of glusterfs was too much for me to handle. That was an epic fail ;)

Well, maybe I'm just out of luck or too brain-dead for it, whatever.

So, moving on, I've tried (although briefly) ceph.

Being in mainline kernel, and not just the staging part, I'd expected it to be much less work-in-progress, but as it is, it's very raw, to the point that x86_64 monitor daemon just crashes upon receiving data from plain x86. Interface is a bunch of simple shell scripts, fairly opaque operation, and the whole thing is built on such crap as boost.

Managed to get it running with two nodes, but it feels like the end of the world - one more kick and it all falls to pieces. Confirmed by the reports all over the mailing list and #ceph.

In-kernel and seemingly fast is a good mix though, so I may get back to it eventually, but now I'd rather prefer to settle on something that actually works.

Next thing in my sight was tahoe-lafs, but it still lacks normal posix-fs interface layer, sftp interface being totally unusable on 1.8.0c3 - no permissions, cp -R fails w/ I/O error, displayed data in inconsistent even with locally-made changes, and so on. A pity, whole system design looks very cool, with raid5-like "parity" instead of plain chunk replication, and it's python!

Thus I ended up with MooseFS.

Replication? Piece a cake, and it's configured on per-tree basis, so important or compact stuff can have one replication "goal" and some heavy trash in the neighbor path have no replication at all. No chance of anything like this with gluster and it's not even documented for ceph.

Performance is totally I/O and network bound (which is totally not-the-case with tahoe, for instance), so no complaints here as well.

One more amazing thing is how simple and transparent it is:

fraggod@anathema:~% mfsgetgoal tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso
tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso: 2

fraggod@anathema:~% mfsfileinfo tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso
tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso:
 chunk 0: 000000000000CE78_00000001 / (id:52856 ver:1)
  copy 1: 192.168.0.8:9422
  copy 2: 192.168.0.11:9422
 chunk 1: 000000000000CE79_00000001 / (id:52857 ver:1)
  copy 1: 192.168.0.10:9422
  copy 2: 192.168.0.11:9422
 chunk 2: 000000000000CE7A_00000001 / (id:52858 ver:1)
  copy 1: 192.168.0.10:9422
  copy 2: 192.168.0.11:9422
 chunk 3: 000000000000CE7B_00000001 / (id:52859 ver:1)
  copy 1: 192.168.0.8:9422
  copy 2: 192.168.0.10:9422
 chunk 4: 000000000000CE7C_00000001 / (id:52860 ver:1)
  copy 1: 192.168.0.10:9422
  copy 2: 192.168.0.11:9422

fraggod@anathema:~% mfsdirinfo tmp/db/softCore/_nix/os
tmp/db/softCore/_nix/os:
 inodes: 12
 directories: 1
 files: 11
 chunks: 175
 length: 11532174263
 size: 11533462528
 realsize: 23066925056

MooseFS has yet to pass the trial of time on my makeshift "cluster", yet none of the other setups went (even remotely) as smooth as this one so far, thus I feel pretty optimistic about it.

Home-brewed NAS gluster with sensible replication

The Setup

Ok, what I have is:

• Node1
  • physical storage (raid1) "disk11", 300G, old and fairly "valuable" data (well, of course it's not "valuable", since I can just re-download it all from p2p, but I'd hate to do that)
  • physical disk "disk12", 150G, same stuff as disk11
• Node2
  • physical disk "disk2", 1.5T, blank, to-be-filled

What I want is one single network storage, with db1 (disk11 + disk12) data available from any node (replicated) and new stuff which won't fit onto this storage should be writen to db2 (what's left of disk2).

With glusterfs there are several ways to do this:

Scenario 1: fully network-aware client.

That's actually the simplest scenario - glusterfsd.vol files on "nodes" should just export local disks and client configuration ties it all together.

Pros:

• Fault tolerance. Client is fully aware of the storage hierarchy, so if one node with db1 is down, it will just use the other one.
• If the bandwidth is better than disk i/o, reading from db1 can be potentially faster (dunno if glusterfs allows that, actually), but that's not the case, since "client" is one of the laptops and it's a slow wifi link.

Cons:

• Write performance is bad - client has to push data to both nodes, and that's a

  big minus with my link.

Scenario 2: server-side replication.

Here, "nodes" export local disks for each other and gather local+remote db1 into cluster/replicate and then export this already-replicated volume. Client just ties db2 and one of the replicated-db1 together via nufa or distribute layer.

Pros:

• Better performance.

Cons:

• Single point of failure, not only for db2 (which is natural, since it's not replicated), but for db1 as well.

Scenario 3: server-side replication + fully-aware client.

db1 replicas are synced by "nodes" and client mounts all three volumes (2 x db1, 1 x db2) with either cluster/unify layer and nufa scheduler (listing both db1 replicas in "local-volume-name") or cluster/nufa.

That's the answer to obvious question I've asked myself after implementing scenario 2: "why not get rid of this single_point_of_failure just by using not single, but both replicated-db1 volumes in nufa?"

In this case, if node1 goes down, client won't even notice it! And if that happens to node2, files from db2 just disappear from hierarchy, but db1 will still remain fully-accessible.

But there is a problem: cluster/nufa has no support for multiple local-volume-name specs. cluster/unify has this support, but requires it's ugly "namespace-volume" hack. The solution would be to aggregate both db1's into a distribute layer and use it as a single volume alongside db2.

With aforementioned physical layout this seem to be just the best all-around case.

Pros:

• Best performance and network utilization.

Cons:

• None?

Implementation

So, scenarios 2 and 3 in terms of glusterfs, with the omission of different performance, lock layers and a few options, for the sake of clarity:

node1 glusterfsd.vol:

## db1: replicated node1/node2
volume local-db1
    type storage/posix
    option directory /srv/storage/db1
end-volume
# No client-caches here, because ops should already come aggregated
# from the client, and the link between servers is much fatter than the client's
volume node2-db1
    type protocol/client
    option remote-host node2
    option remote-subvolume local-db1
end-volume
volume composite-db1
    type cluster/replicate
    subvolumes local-db1 node2-db1
end-volume
## db: linear (nufa) db1 + db2
## export: local-db1 (for peers), composite-db1 (for clients)
volume export
    type protocol/server
    subvolumes local-db1 composite-db1
end-volume

node2 glusterfsd.vol:

## db1: replicated node1/node2
volume local-db1
    type storage/posix
    option directory /srv/storage/db1
end-volume
# No client-caches here, because ops should already come aggregated
# from the client, and the link between servers is much fatter than the client's
volume node1-db1
    type protocol/client
    option remote-host node1
    option remote-subvolume local-db1
end-volume
volume composite-db1
    type cluster/replicate
    subvolumes local-db1 node1-db1
end-volume
## db2: node2
volume db2
    type storage/posix
    option directory /srv/storage/db2
end-volume
## db: linear (nufa) db1 + db2
## export: local-db1 (for peers), composite-db1 (for clients)
volume export
    type protocol/server
    subvolumes local-db1 composite-db1
end-volume

client (replicated to both nodes):

volume node1-db1
    type protocol/client
    option remote-host node1
    option remote-subvolume composite-db1
end-volume
volume node2-db1
    type protocol/client
    option remote-host node2
    option remote-subvolume composite-db1
end-volume
volume db1
    type cluster/distribute
    option remote-subvolume node1-db1 node2-db1
end-volume
volume db2
    type protocol/client
    option remote-host node2
    option remote-subvolume db2
end-volume
volume db
    type cluster/nufa
    option local-volume-name db1
    subvolumes db1 db2
end-volume

Actually there's one more scenario I thought of for non-local clients - same as 2, but pushing nufa into glusterfsd.vol on "nodes", thus making client mount single unified volume on a single host via single port in a single connection.

Not that I really need this one, since all I need to mount from external networks is just music 99.9% of time, and NFS + FS-Cache offer more advantages there, although I might resort to it in the future, when music won't fit to db1 anymore (doubt that'll happen anytime soon).

P.S.

Configs are fine, but the most important thing for setting up glusterfs are these lines:

node# /usr/sbin/glusterfsd --debug --volfile=/etc/glusterfs/glusterfsd.vol
client# /usr/sbin/glusterfs --debug --volfile-server=somenode /mnt/tmp

No IPSec on-a-stick for me ;(

Guess being a long user of stuff like OpenSSH, iproute2 and VDE rots your brain - you start thinking that building any sort of tunnel is a bliss. Well, it's not. At least not "any sort".

I chose openswan because it looks like more extensive implementation than the rest, and is supported by folks like Red Hat, plus it is fairly up to date and looks constantly developed. Another cherry in it was apparent smartcard support via nss now and opensc in the past.

Comparing to ssh or pretty much any tunneling I've encountered to this point, it's still quite a remarkably epic fail. Guess I'll waste a bit more time on this crap, since success seems so close, but it's quite amazing how crappy such projects can still be these days. Of course, at least it's free, right?

Drop-in ccrypt replacement for bournal

Anyway, encryption there is done by the means of ccrypt utility, which is sorta CLI for openssl. I don't get the rationale behind using it instead of openssl directly (like "openssl enc ..."), and there are actually even better options, like gnupg, which won't need a special logic to keep separate stream-cipher password, like it's done in bournal.

Getting rid of dead bittorrent trackers for rtorrent by scrubbing .torrent files

If you're downloading stuff off the 'net via bt like me, then TPB is probably quite a familiar place to you.

So the easiest generic solution (which will, of course, work for rtorrent) I've found is just to process the .torrent files before feeding them to the leecher app. Since I'm downloading these via firefox exclusively, and there I use FlashGot (not the standard "open with" interface since I also use it to download large files on remote machine w/o firefox, and afaik it's just not the way "open with" works) to drop them into an torrent bin via script, it's actually a matter of updating the link-receiving script.

So, to the script itself...

# URL checker
def trak_check(trak):
    if not trak: return False
    try: ip = gethostbyname(urlparse(trak).netloc.split(':', 1)[0])
    except gaierror: return True # prehaps it will resolve later on
    else: return ip not in ('127.0.0.1', '0.0.0.0')

# Actual processing
torrent = bdecode(torrent)
for tier in list(torrent['announce-list']):
    for trak in list(tier):
        if not trak_check(trak):
            tier.remove(trak)
            # print >>sys.stderr, 'Dropped:', trak
    if not tier: torrent['announce-list'].remove(tier)
# print >>sys.stderr, 'Result:', torrent['announce-list']
if not trak_check(torrent['announce']):
    torrent['announce'] = torrent['announce-list'][0][0]
torrent = bencode(torrent)

I've wanted to make it as light as possible so no logging or optparse/argparse stuff I tend cram everywhere I can, and the extra and heavy imports like urllib/urllib2 are conditional as well. The only dependency is python (>=2.6) itself.

Basic use-case is one of these:

% brecode.py < /path/to/file.torrent > /path/to/proccessed.torrent
% brecode.py http://tpb.org/torrents/myfile.torrent > /path/to/proccessed.torrent
% brecode.py http://tpb.org/torrents/myfile.torrent\
    -r http://tpb.org/ -c "...some cookies..." -d /path/to/torrents-bin/

All the extra headers like cookies and referer are optional, so is the destination path (dir, basename is generated from URL). My use-case in FlashGot is this: "[URL] -r [REFERER] -c [COOKIE] -d /mnt/p2p/bt/torrents"

And there's the script itself.

Music collection updates feed via musicbrainz and last.fm

Naturally, subscribing to something like an unfiltered flow of new music releases isn't an option, but no music site other than last.fm out there knows the acts in my collection to track updates for those, and last.fm doesn't seem to have the functionality I need - just to list new studio releases from the artists I listened to beyond some reasonable threshold, or I just haven't been able to find it.

First one seemed easier, but not as entertaining as the second, plus I have virtually no knowledge to pick a site which will be best-suited for that (and I'd hate to pick a first thing from the google), and I'd probably have to post-process what this site feeds me anyway. I've decided to stick with the second way.

Object-db to a web content is a typical task for a web framework, so I chose to use django as a basic template for the task.

The next task is to compare two lists to drop the stuff I already have (local albums list) from the fetched list.

After that comes db storage, and there's not much to implement but a simple ORM-model definition with a few unique keys and the django will take care of the rest.

The last part is the data representation.

One more thing I've thought of is the caching - no need to strain free databases with redundant queries, so the only non-cached data from these are the lists of the releases which should be updated from time to time, the rest can be kept in a single "seen" set of id's, so it'd be immediately obvious if the release was processed and queried before and is of no more interest now.

To summarize: the tools are django, python-musicbrainz2 and pylast; last.fm and musicbrainz - the data sources (although I might extend this list); direct result - this feed.

Exherbo / paludis fossil syncer

Repository, syncer and some instructions are here.

Thought I'd give google some keywords, should someone be looking for the same thing, although I'd probably try to push it into paludis and/or "unavailable" repo, when (and if) I'll get a bit more solid grasp on exherbo concepts.

LUKS + dm-crypt rootfs without password via smartcard

I don't believe in fairly common practice of "trying out" something new on a VM - it just don't work for me, probably because I see it as a stupid and posintless thing on some subconscious level, so I've decided to put it onto one of my two laptops, which kinda needed a good cleanup anyway.

While at it, I thought it'd be a good idea to finally dump that stupid practice of entering fs-password on boot, yet I did like the idea of encrypted fs, especially in case of laptop, so I've needed to devise reasonably secure yet paswordless boot method.

I use in-kernel LUKS-enabled dm-crypt (with the help of cryptsetup tool), and I need some initrd (or init-fs) for LVM-root anyway.

Boot sequence comes down to these:

• Mount pseudofs like proc/sys, get encrypted partition dev and real-rootfs signature (for findfs tool, like label or uuid) from cmdline.
• Init openct, find smartcard in /sys by hardcoded product id and attach it to openct.
• Mount persistent key-material storage (same /boot in my case).
• Read "old" key, replace it with a hashed version, aka "new key".
• Sign old key using smartcard, open fs with the resulting key.
• Drop this key from LUKS storage, add a signed "new" key to it.
• Kill openct processes, effectively severing link with smartcard.
• Detect and activate LVM volume groups.
• Find (findfs) and mount rootfs among currently-available partitions.
• Umount proc/sys, pivot_root, chroot.
• Here comes the target OS' init.

Took me some time to assemble and test this stuff, although it was fun playing with linux+busybox mini-OS. Makes me somewhat wonder about what takes several GiBs of space in a full-fledged OS when BB contains pretty much everything in less than one MiB ;)

And it's probably a good idea to put some early check of /boot partition (hashes, mounts, whatever) into booted OS init-scripts to see if it was not altered in any significant way. Not really a guarantee that something nasty weren't done to it (and then cleaned up, for example) plus there's no proof that actual OS was booted up from it and the kernel isn't tainted in some malicious way, but should be enough against some lame tampering or pranks, should these ever happen.

Anyway, here's the repo with all the initrd stuff, should anyone need it.

Thoughts on VCS, supporting documentation and Fossil

Git is a great help in these tasks, but what I feel lacking there is a first - common timeline (spanning both into the past and the future) for all these data series, and second - documentation.

Solution to the first one I've yet to find.

Documentation for anything more than local implementation details and it's history is virtually non-existant and most times it takes a lot of effort and time to retrace the original line of thought, reasoning and purpose behind the stuff I've done (and why I've done it like that) in the past, often with the considerable gaps and eventual re-invention of the wheels and pitfalls I've already done, due to faulty biological memory.

So, today I've decided to scour over the available project and task management software to find something that ties the vcs repositories and their logs with the future tickets and some sort of expanded notes, where needed.

It just looks like a completely wrong approach for my task, yet I thought that I can probably tolerate that if there are no better options and then I've stumbled upon Fossil VCS.

Fossil's overall design principles and concepts (plus this) are well-documented on it's site (which is a just a fossil repo itself), and the catch-points for me were:

• Tickets and wiki, of course. Can be edited locally, synced, distributed, have local settings and appearance, based on tcl-ish domain-specific language.
• Distributed nature, yet rational position of authors on centralization and synchronization topic.
• All-in-one-static-binary approach! Installing hundreds of git binaries to every freebsd-to-debian-based system, was a pain, plus I've ended up with 1.4-1.7 version span and some features (like "add -p") depend on a whole lot of stuff there, like perl and damn lot of it's modules. Unix-way is cool, but that's really more portable and distributed-way-friendly.
• Repository in a single package, and not just a binary blob, but a freely-browsable sqlite db. It certainly is a hell lot more convenient than path with over nine thousand blobs with sha1-names, even if the actual artifact-storage here is built basically the same way. And the performance should be actually better than the fs - with just index-selects BTree-based sqlite is as fast as filesystem, but keeping different indexes on fs is by sym-/hardlinking, and that's a pain that is never done right on fs.
• As simple as possible internal blobs' format.
• Actual symbolics and terminology. Git is a faceless tool, Fossil have some sort of a style, and that's nice ;)

Yet there are some things I don't like about it:

• HTTP-only sync. In what kind of twisted world that can be better than ssh+pam or direct access? Can be fixed with a wrapper, I guess, but really, wtf...
• SQLite container around generic artifact storage. Artifacts are pure data with a single sha1sum-key for it, and that is simple, solid and easy to work with anytime, but wrapped into sqlite db it suddenly depends on this db format, libs, command-line tool or language bindings, etc. All the other tables can be rebuilt just from these blobs, so they should be as accessible as possible, but I guess that'd violate whole single-file design concept and would require a lot of separate management code, a pity.

But that's nothing more than a few hours' tour of the docs and basic hello-world tests, guess it all will look different after I'll use it for a while, which I'm intend to do right now. In the worst case it's just a distributed issue tracker + wiki with cli interface and great versioning support in one-file package (including webserver) which is more than I can say about trac, anyway.