Dec 09, 2010
It's been a while since I augmented libnotify / notification-daemon stack to better
suit my (maybe not-so-) humble needs, and it certainly was an improvement, but
there's no limit to perfection and since then I felt the urge to upgrade it
every now and then.
One early fix was to let messages with priority=critical through without
delays and aggregation. I've learned it the hard way when my laptop shut down
because of drained battery without me registering any notification about it.
Other good candidates for high priority seem to be real-time messages like
emms track updates and network
connectivity loss events which either too important to ignore or just don't
make much sense after delay. Implementation here is straightforward - just
check urgency level and pass these unhindered to notification-daemon.
Another important feature which seem to be missing in reference daemon is the
ability to just cleanup the screen of notifications. Sometimes you just need
to dismiss them to see the content beneath, or you just read them and don't
want them drawing any more attention.
The only available interface for that seem to be CloseNotification method,
which can only close notification message using it's id, hence only useful
from the application that created the note. Kinda makes sense to avoid apps
stepping on each others toes, but since id's in question are sequential, it
won't be much of a problem to an app to abuse this mechanism anyway.
Proxy script, sitting in the middle of dbus communication as it is, don't have
to guess these ids, as can just keep track of them.
So, to clean up the occasional notification-mess I extended the
CloseNotification method to accept 0 as a "special" id, closing all the
currently-displayed notifications.
Binding it to a key is just a matter of (a bit inelegant, but powerful)
dbus-send tool invocation:
% dbus-send --type=method_call\
--dest=org.freedesktop.Notifications\
/org/freedesktop/Notifications\
org.freedesktop.Notifications.CloseNotification uint32:0
Expanding the idea of occasional distraction-free needs, I found the idea of
the ability to "plug" the notification system - collecting the notifications
into the same digest behind the scenes (yet passing urgent ones, if this
behavior is enabled) - when necessary quite appealing, so I just added a flag
akin to "fullscreen" check, forcing notification aggregation regardless of
rate when it's set.
Of course, some means of control over this flag was necessary, so another
extension of the interface was to add "Set" method to control
notification-proxy options. Method was also useful to occasionally toggle
special "urgent" messages treatment, so I empowered it to do so as well by
making it accept a key-value array of parameters to apply.
And since now there is a plug, I also found handy to have a complimentary
"Flush" method to dump last digested notifications.
Same handy dbus-send tool comes to rescue again, when these need to be toggled
or set via cli:
% dbus-send --type=method_call\
--dest=org.freedesktop.Notifications\
/org/freedesktop/Notifications\
org.freedesktop.Notifications.Set\
dict:string:boolean:plug_toggle,true
In contrast to cleanup, I occasionally found myself monitoring low-traffic IRC
conversations entirely through notification boxes - no point switching the
apps if you can read the whole lines right there, but there was a catch of
course - you have to immediately switch attention from whatever you're doing
to a notification box to be able to read it before it times out and
disappears, which of course is a quite inconvenient.
Easy solution is to just override "timeout" value in notification boxes to
make them stay as long as you need to, so one more flag for the "Set" method
to handle plus one-liner check and there it is.
Now it's possible to read them with minimum distraction from the current
activity and dismiss via mentioned above extended CloseNotification method.
As if the above was not enough, sometimes I found myself willing to read and
react to the stuff from one set of sources, while temporarily ignoring the
traffic from the others, like when you're working at some hack, discussing it
(and the current implications / situation) in parallel over jabber or irc,
while heated discussion (but interesting none the less) starts in another
channel.
Shutting down the offending channel in
ERC, leaving
BNC to monitor the
conversation or just supress notifications with some ERC command would
probably be the right way to handle that, yet it's not always that simple,
especially since every notification-enabled app then would have to implement
some way of doing that, which of course is not the case at all.
Remedy is in the customizable filters for notifications, which can be a simple
set of regex'es, dumped into some specific dot-file, but even as I started to
implement the idea, I could think of several different validation scenarios
like "match summary against several regexes", "match message body", "match
simple regex with a list of exceptions" or even some counting and more complex
logic for them.
Idea of inventing yet another perlish (poorly-designed, minimal, ambiguous,
write-only) DSL for filtering rules didn't struck me as an exactly bright one,
so I thought for looking for some lib implementation of clearly-defined and
thought-through syntax for such needs, yet found nothing designed purely for
such filtering task (could be one of the reasons why every tool and daemon
hard-codes it's own DSL for that *sigh*).
On that note I thought of some generic yet easily extensible syntax for such
rules, and came to realization that simple SICP-like subset of scheme/lisp
with regex support would be exactly what I need.
Luckily, there are plenty implementations of such embedded languages in
python, and since I needed a really simple and customizabe one, I've decided
to stick with extended 90-line "
lis.py",
described by Peter Norvig
here and
extended
here. Out goes unnecessary file-handling,
plus regexes and some minor fixes and the result is "make it into whatever you
need" language.
Just added a stat and mtime check on a dotfile, reading and compiling the
matcher-function from it on any change. Contents may look like this:
(define-macro define-matcher (lambda
(name comp last rev-args)
`(define ,name (lambda args
(if (= (length args) 1) ,last
(let ((atom (car args)) (args (cdr args)))
(,comp
(~ ,@(if rev-args '((car args) atom) '(atom (car args))))
(apply ,name (cons atom (cdr args))))))))))
(define-matcher ~all and #t #f)
(define-matcher all~ and #t #t)
(define-matcher ~any or #f #f)
(define-matcher any~ or #f #t)
(lambda (summary body)
(not (and
(~ "^erc: #\S+" summary)
(~ "^\*\*\* #\S+ (was created on|modes:) " body))
(all~ summary "^erc: #pulseaudio$" "^mail:")))
Which kinda shows what can you do with it, making your own syntax as you go
along (note that stuff like "and" is also a macro, just defined on a higher
level).
Even with weird macros I find it much more comprehensible than rsync filters,
apache/lighttpd rewrite magic or pretty much any pseudo-simple magic set of
string-matching rules I had to work with.
I considered using python itself to the same end, but found that it's syntax
is both more verbose and less flexible/extensible for such goal, plus it
allows to do far too much for a simple filtering script which can potentially
be evaluated by process with elevated privileges, hence would need some sort
of sandboxing anyway.
In my case all this stuff is bound to convenient key shortcuts via
fluxbox wm:
# Notification-proxy control
Print :Exec dbus-send --type=method_call\
--dest=org.freedesktop.Notifications\
/org/freedesktop/Notifications org.freedesktop.Notifications.Set\
dict:string:boolean:plug_toggle,true
Shift Print :Exec dbus-send --type=method_call\
--dest=org.freedesktop.Notifications\
/org/freedesktop/Notifications org.freedesktop.Notifications.Set\
dict:string:boolean:cleanup_toggle,true
Pause :Exec dbus-send --type=method_call\
--dest=org.freedesktop.Notifications\
/org/freedesktop/Notifications\
org.freedesktop.Notifications.CloseNotification\
uint32:0
Shift Pause :Exec dbus-send --type=method_call\
--dest=org.freedesktop.Notifications\
/org/freedesktop/Notifications\
org.freedesktop.Notifications.Flush
Pretty sure there's more room for improvement in this aspect, so I'd have to
extend the system once again, which is fun all by itself.
Resulting (and maybe further extended) script is here, now linked against a
bit revised lis.py scheme implementation.
Dec 07, 2010
Info on the subject in the internets is a bit scarce, so here goes my
case. Keep in mind however that it's not a stress-benchmark of any kind and
actually rather degenerate use-case, since loads aren't pushing hardware to
any limits.
Guess I can say that it's quite remarkable in a way that it's really
unremarkable - I just kinda forgot it's there, which is probably the best thing
one can expect from a filesystem.
My setup is 4 physical nodes at most, with 1.3 TiB of data in fairly large
files (3361/52862 dirs/files, calculated average is about 250 MiB for a file).
Spanned fs hosts a storage for distfiles, media content, vm images and pretty
much anything that comes in the compressed form and worth keeping for further
usage. Hence the access to files is highly sequential in most cases (as in
reading gzip, listening to mp3, watching a movie, etc).
OS on the nodes is gentoo/exherbo/fedora mix and is a subject to constant
software updates, breakages and rebuilds. Naturally, mfs is proven to be quite
resilent in these conditions, since it doesn't depend on boost or other
volatile crap and just consists of several binaries and configuration files,
which work fine even with defaults right out of the box.
One node is a "master", eating 100 MiB RAM (115 VSZ) on the few-month average
(according to atop logs). Others have metalogger slaves which cost virtually
nothing (<3 MiB VSZ), so it's not a big deal to keep metadata fully-replicated
just in case.
Chunkservers have 500 GiB - 3 TiB space on btrfs. These usually hang on 10 MiB
RAM, occasional 50-100 MiB in VSZ, though it's not swapped-out, just unused.
Cpu usage for each is negligible, even though mfsmaster + mfsmount +
mfschunkserver node is Atom D510 on miniITX board.
mfsmount maintains persistent connection to master and on-demand to
chunkservers.
It doesn't seem to mind if some of them are down though, so I guess it's
perfectly possible to upload files via mfsmount to one (the only accessible)
node and they'll be replicated to others from there (more details on that
below), although I'm unsure what will happen when you'll try to retrieve
chunks, stored exclusively on inaccessible nodes (guess
it's easy enough to test, anyway).
I use only one mfsmount on the same machine as master, and re-export (mostly
for reading) it over NFS, SFTP, WebDAV and plain HTTP to other machines.
Re-export is there because that way I don't need access to all machines in
cluster, which can be in a separate network (i.e. if I access fs from work),
plus stuff like NFS comes out of the box (no need for separate client) and
have a nice FS-Cache support, which saves a lot of bandwidth, webdav/sftp
works for ms-os machines as well and server-based replication saves more
precious bandwidth all by itself.
FS bandwidth in my case in constant ~1 MiB read 24/7 plus any on-demand
reading on speeds, which are usually slower than any single hdd (over slower
network links like 100 Mbps LAN and WiFi), and using only a few threads as
well, so I'm afraid I can't give any real-world stress results here.
On a local bulk-copy operations to/from mfs mount though, disk always seem to
be a bottleneck, with all other parameters far below any possible limitations,
but in my case it's a simple "wd green" low-speed/noise high-capacity disks or
seagate/hitachi disks with AAM threshold set to lowest level via "hdparm -M"
(works well for sound, but
I never really cared about how it affects speed to check).
Chunkservers' storage consists of idexed (AA/AABCD...) paths, according
to chunk names, which can be easily retreived from master. They rely on
fs scanning to determine which chunks they have, so I've been able to
successfully merge two nodes into one w/o storing the chunks on
different filesystems/paths (which is also perfectly possible).
Chunkservers talk to each other on p2p-basis (doesn't imply that they don't need
connection to master, but bandwidth there doesn't seem to be an issue at all) to
maintain requested replication goal and auto-balance disk space between
themselves, so the free percentage tries to be equal on all nodes (w/o
compromising the goal, of course), so with goal=2 and 4 nodes I have 30% space
usage on backend-fs on both 500 GiB node and 3 TiB one.
Balancing seem to be managed by every chunkserver in background (not quite sure
if I've seen it in any docs, but there's a "chunk testing" process, which seem
to imply that, and can be tuned btw), according to info about chunk and other
currently-available nodes' space utilization from master.
Hence, adding/removing nodes is a bliss - just turn it on/off, no configuration
changes for other nodes are necessary - master sees the change (new/lost
connection) and all the chunkservers start relocating/getting the chunks to
restore the balance and maintain the requested goal. In a few hours everything
will be balanced again.
Whole approach seem superior to dumb round-robin of the chunks on creation or
rehashing and relocating every one of them on single node failure, and
suggests that it might be easy to implement custom replication and balancing
scheme just by rsync'ing chunks between nodes as necessary (i.e. to make most
of small ssd buffer, putting most-demanded files' chunks there).
And indeed I've utilized that feature twice to merge different nodes and
filesystems, although the latter is not really necessary, since chunkserver
can work with several storage paths on different filesystems, but it's just
seem irrational to keep several btrfs trees these days, as they can even span
to multiple devices.
But the best part, enabling me not to look further for alternatives, is the
simple fact that I've yet to see any problem in the stability department - it
still just works. mfsmount never refused to give or receive a file, node
daemons never crashed or came back up with a weird inconsistency (which I
don't think is easy to produce with such simple setup/design, anyway).
Connection between nodes has failed quite often - sometimes my
NIC/switch/cables went to 30% packet loss for no apparent reason, sometimes
I've messed up openswan and ipsec or some other network setup, shut down and
hard-rebooted the nodes as necessary, but such failures were always
unnoticeable here, without any need to restart anything on the mfs level -
chunkservers just reconnect, forget obsolete chunks and keep going about their
business.
Well, there *was* one exception: one time I've managed to hard-reboot a
master machine and noticed that mfsmaster has failed to start.
Problem was missing metadata.mfs file in /var/lib, which I believe is created
on mfsmaster stop and checkpointed every hour to .back file, so, knowing there
was no changes to fs in the last few minutes, I just removed the .back suffix
and everything started just fine.
Doing it The Right Way would've involved stopping any of the metalogger nodes
(or signaling it somehow) and retreiving this file from there, or just
starting master on that node, updating the mfsmaster ns entry, since they're
identical.
Of course, it's just a commodity hardware and lighter loads, but it's still way
above other stuff I've tried here in virtually every aspect, so thumbs up for
moose.
Nov 12, 2010
As of today, I'm unemployed once again.
Guess now I'll have time to debug and report a btrfs-systemd crash, read all the
feeds, fix some long-standing issues on my home servers, update an antique web
setup, write a few watch-notify scripts there, deploy/update a configuration
management systems, update/finish/publish a few of my spare-time projects, start
playing with a lot of new ideas, check out networking tools like connman, wicd,
nm and a bunch of other cool-stuff oss projects, write a few hooks for plotting
and graphing stuff in real-time, adapt emacs mail/rss tools, update other elisp
stuff, emacs itself, a few symbian-to pc event hooks, check out gnustep
environment, ltu lang articles, pybrain and a few other simple machine-learning
implementations, some lua-ai for spring, play a lot of games I've missed in past
few years, read a few dozen books I've already uploaded but never had a time to,
study linear and geometric algebra... maybe find a new job, even, before I
starve?
Nah, nobody in the world have that much time... ;)
Nov 05, 2010
It's been more than a week since I've migrated from sysvinit and gentoo'ish
baselayout scripts to
systemd with it's units, and aside
from few initial todos it's been surprisingly easy.
Nice guide for migration (which actually tipped me into trying systemd)
can
be found here, in this post I'd
rather summarize my experiences.
Most distributions seem to take "the legacy" way of migration, starting all
the old initscripts from systemd just as sysinit did before that.
It makes some sense, since all the actions necessary to start the service are
already written there, but most of them are no longer necessary with systemd -
you don't need pidfiles, daemonization, killing code, LSB headers and most
checks for other stuff... which kinda leaves nothing at all for 95% of
software I've encountered!
I haven't really tried to adapt fedora or debian init for systemd (since my
setup runs
exherbo), so I may be missing some
crucial points here, but it looks like even in these systems initscripts,
written in simple unaugmented *sh, are unnecessary evil, each one doing the
same thing in it's own crappy way.
With exherbo (or gentoo, for that matter), which has a bit more advanced init
system, it's even harder to find some sense in keeping these
scripts. Baselayout
allows some cool stuff beyond simple LSB headers, but
does so in it's own way, typical initscript here looks like this:
#!/sbin/runscript
depend() {
use logger
need clock hostname
provide cron
}
start() {
ebegin "Starting ${SVCNAME}"
start-stop-daemon --start --pidfile ${FCRON_PIDFILE}\
--exec /usr/sbin/fcron -- -c ${FCRON_CONF}
eend $?
}
stop() {
ebegin "Stopping ${SVCNAME}"
start-stop-daemon --stop --pidfile ${FCRON_PIDFILE}
eend $?
}
...with $SVCNAME taken from the script name and other vars from complimentary
"/etc/conf.d/someservice" file (with sensible defaults in initscript itself).
Such script already allows nice and logged output (with e* commands) and
clearly-defined, relatively uncluttered sections for startup and shutdown. You
don't have to parse commandline arguments (although it's perfectly possible),
since baselayout scripts will do that, and every daemon is accounted for via
"start-stop-daemon" wrapper - it has a few simple ways to check their status via
passed --pidfile or --exec lines, plus it handles forking (if necessary), IO
redirection, dropping privileges and stuff like that.
All these feats lead to much more consistent init and control over services'
state:
root@damnation:~# rc-status -a
Runlevel: shutdown
killprocs [ stopped ]
savecache [ stopped ]
mount-ro [ stopped ]
Runlevel: single
Runlevel: nonetwork
local [ started ]
Runlevel: cryptinit
rsyslog [ started ]
ip6tables [ started ]
...
twistd [ started ]
local [ started ]
Runlevel: sysinit
dmesg [ started ]
udev [ started ]
devfs [ started ]
Runlevel: boot
hwclock [ started ]
lvm [ started ]
...
wdd [ started ]
keymaps [ started ]
Runlevel: default
rsyslog [ started ]
ip6tables [ started ]
...
twistd [ started ]
local [ started ]
Dynamic Runlevel: hotplugged
Dynamic Runlevel: needed
sysfs [ started ]
rpc.pipefs [ started ]
...
rpcbind [ started ]
rpc.idmapd [ started ]
Dynamic Runlevel: manual
One nice colored list of everything that should be running, is running, failed
to start, crashed and whatever. One look and you know if unscheduled reboot
has any surprises for you. Weird that such long-lived and supported distros as
debian and fedora make these simple tasks so much harder (chkconfig --list?
You can keep it! ;)
Furthermore, it provides as many custom and named runlevels as you want, as a
way to flip the state of the whole system with a painless one-liner.
Now, systemd provides all of these features, in a cleaner nicer form and much
more, but that makes migration from one to the other actually harder.
Systemd is developed/tested mainly on and for fedora, so abscence of LSB
headers in these scripts is a problem (no dependency information), and
presence of other headers (which start another scripts w/o systemd help or
permission) is even more serious problem.
start-stop-daemon interference is also redundant and actually harmful and so
is e* (and other special bl-commands and wrappers), and they won't work w/o
baselayout framework.
Thus, it makes sense for systemd on exherbo to be totally independent of
baselayout and it's scripts, and having a separate package option to install
systemd and baselayout-specific init stuff:
root@sacrilege:~# cave show -f acpid
* sys-power/acpid
::arbor 2.0.6-r2* {:0}
::installed 2.0.6-r2 {:0}
sys-power/acpid-2.0.6-r2:0::installed
Description
acpid is designed to notify user-space programs of ACPI events. It will
will attempt to connect to the Linux kernel via the input layer and
netlink. When an ACPI event is received from one of these sources, acpid
will examine a list of rules, and execute the rules that match the event.
Homepage http://tedfelix.com/linux/acpid-netlink.html
Summary A configurable ACPI policy daemon for Linux
From repositories arbor
Installed time Thu Oct 21 23:11:55 YEKST 2010
Installed using paludis-0.55.0-git-0.54.2-44-g203a470
Licences GPL-2
Options (-baselayout) (systemd) build_options: -trace
sys-power/acpid-2.0.6-r2:0::arbor
Homepage http://tedfelix.com/linux/acpid-netlink.html
Summary A configurable ACPI policy daemon for Linux
Description
acpid is designed to notify user-space programs of ACPI events. It will
will attempt to connect to the Linux kernel via the input layer and
netlink. When an ACPI event is received from one of these sources, acpid
will examine a list of rules, and execute the rules that match the event.
Options -baselayout systemd
build_options: -recommended_tests split strip jobs -trace -preserve_work
Overridden Masks
Supported platforms ~amd64 ~x86
So, basically, the migration to systemd consists of enabling the option and
flipping the "eclectic init" switch:
root@sacrilege:~# eclectic init list
Available providers for init:
[1] systemd *
[2] sysvinit
Of course, in reality things are little more complicated, and breaking init is
quite undesirable prospect, so I took advantage of virtualization capabilities
of cpu on my new laptop and made a complete virtual replica of the system.
- A dedicated lv for whole setup.
- luksFormat it, so it'd represent an encrypted "raw" partition.
- pvcreate / vgcreate / lvcreate / mkfs on top of it, identical (although much
smaller) to original system.
- A script to mount all these and rsync the "original" system to this replica,
with a few post-sync hooks to make some vm-specific changes - different vg
name, no extra devices for media content, simpler passwords.
I have this script here, list
of "exclusions" for rsync is actually taken from backup scripts, since it's
designed to omit various heavy and non-critical paths like caches, spools and
debugging info, plus there's not much point syncing most /home contents. All
in all, whole setup is about 2-3G and rsync makes a fast job of updating it.
vm (qemu-kvm) startup is right there in the
script and uses exactly the same
kernel/initrd as the host machine, although I skip encryption part (via kernel
cmdline) for faster bootup.
And the first launch gave quite a mixed result: systemd fired a bunch of basic
stuff at once, then hanged for about a minute before presenting a getty. After
login, it turned out that none of the filesystems in /etc/fstab got mounted.
Systemd handles mounts in quite a clever (and fully documented) way - from
each device in fstab it creates a "XXX.device" unit, "fsck@XXX.service", and
either "XXX.mount" or "XXX.automount" from mountpoints (depending on optional
"comment=" mount opts). All the autogenerated "XXX.mount" units without
explicit "noauto" option will get started on boot.
And they do get started, hence that hang. Each .mount, naturally, depends on
corresponding .device unit (with fsck in between), and these are considered
started when udev issues an event.
In my case, even after exherbo-specific lvm2.service, which does vgscan and
vgchange -ay stuff, these events are never generated, so .device units hang
for 60 seconds and systemd marks them as "failed" as well as dependent .mount
units.
It looks like my local problem, since I actually activate and use these in
initrd, so I just worked around it by adding "ExecStart=-/sbin/udevadm trigger
--subsystem-match=block --sysname-match=dm-*" line to lvm2.service. That
generated the event in parallel to still-waiting .device units, so they got
started, then fsck, then just mounted.
While this may look a bit like a problem, it's quite surprising how transparent
and easy-to-debug whole process is, regardless of it's massively-parallel
nature - all the information is available via "systemctl" and it's show/status
commands, all the services are organized (and monitored) in systemd-cgls tree,
and can be easily debugged with systemd monitoring and console/dmesg-logging
features:
root@sacrilege:~# systemd-cgls
├ 2 [kthreadd]
├ 3 [ksoftirqd/0]
├ 6 [migration/0]
├ 7 [migration/1]
├ 9 [ksoftirqd/1]
├ 10 [kworker/0:1]
...
├ 2688 [kworker/0:2]
├ 2700 [kworker/u:0]
├ 2728 [kworker/u:2]
├ 2729 [kworker/u:4]
├ user
│ └ fraggod
│ └ no-session
│ ├ 1444 /bin/sh /usr/bin/startx
│ ├ 1462 xinit /home/fraggod/.xinitrc -- /etc/X11/xinit/xserverrc :0 -auth /home/fraggod/.serveraut...
...
│ ├ 2407 ssh root@anathema -Y
│ └ 2751 systemd-cgls
└ systemd-1
├ 1 /sbin/init
├ var-src.mount
├ var-tmp.mount
├ ipsec.service
│ ├ 1059 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --nocrsend no --str...
│ ├ 1060 logger -s -p daemon.error -t ipsec__plutorun
│ ├ 1061 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --nocrsend no --str...
│ ├ 1062 /bin/sh /usr/lib/ipsec/_plutoload --wait no --post
│ ├ 1064 /usr/libexec/ipsec/pluto --nofork --secretsfile /etc/ipsec.secrets --ipsecdir /etc/ipsec.d -...
│ ├ 1069 pluto helper # 0
│ ├ 1070 pluto helper # 1
│ ├ 1071 pluto helper # 2
│ └ 1223 _pluto_adns
├ sys-kernel-debug.mount
├ var-cache-fscache.mount
├ net@.service
├ rpcidmapd.service
│ └ 899 /usr/sbin/rpc.idmapd -f
├ rpcstatd.service
│ └ 892 /sbin/rpc.statd -F
├ rpcbind.service
│ └ 890 /sbin/rpcbind -d
├ wpa_supplicant.service
│ └ 889 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -u -Dwext -iwlan0
├ cachefilesd.service
│ └ 883 /sbin/cachefilesd -n
├ dbus.service
│ └ 784 /usr/bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
├ acpid.service
│ └ 775 /usr/sbin/acpid -f
├ openct.service
│ └ 786 /usr/sbin/ifdhandler -H -p etoken64 usb /dev/bus/usb/002/003
├ ntpd.service
│ └ 772 /usr/sbin/ntpd -u ntp:ntp -n -g -p /var/run/ntpd.pid
├ bluetooth.service
│ ├ 771 /usr/sbin/bluetoothd -n
│ └ 1469 [khidpd_046db008]
├ syslog.service
│ └ 768 /usr/sbin/rsyslogd -n -c5 -6
├ getty@.service
│ ├ tty1
│ │ └ 1451 /sbin/agetty 38400 tty1
│ ├ tty3
│ │ └ 766 /sbin/agetty 38400 tty3
│ ├ tty6
│ │ └ 765 /sbin/agetty 38400 tty6
│ ├ tty5
│ │ └ 763 /sbin/agetty 38400 tty5
│ ├ tty4
│ │ └ 762 /sbin/agetty 38400 tty4
│ └ tty2
│ └ 761 /sbin/agetty 38400 tty2
├ postfix.service
│ ├ 872 /usr/lib/postfix/master
│ ├ 877 qmgr -l -t fifo -u
│ └ 2631 pickup -l -t fifo -u
├ fcron.service
│ └ 755 /usr/sbin/fcron -f
├ var-cache.mount
├ var-run.mount
├ var-lock.mount
├ var-db-paludis.mount
├ home-fraggod-.spring.mount
├ etc-core.mount
├ var.mount
├ home.mount
├ boot.mount
├ fsck@.service
├ dev-mapper-prime\x2dswap.swap
├ dev-mqueue.mount
├ dev-hugepages.mount
├ udev.service
│ ├ 240 /sbin/udevd
│ ├ 639 /sbin/udevd
│ └ 640 /sbin/udevd
├ systemd-logger.service
│ └ 228 //lib/systemd/systemd-logger
└ tmp.mount
root@sacrilege:~# systemctl status ipsec.service
ipsec.service - IPSec (openswan)
Loaded: loaded (/etc/systemd/system/ipsec.service)
Active: active (running) since Fri, 05 Nov 2010 15:16:54 +0500; 2h 16min ago
Process: 981 (/usr/sbin/ipsec setup start, code=exited, status=0/SUCCESS)
Process: 974 (/bin/sleep 10, code=exited, status=0/SUCCESS)
CGroup: name=systemd:/systemd-1/ipsec.service
├ 1059 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --noc...
├ 1060 logger -s -p daemon.error -t ipsec__plutorun
├ 1061 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --noc...
├ 1062 /bin/sh /usr/lib/ipsec/_plutoload --wait no --post
├ 1064 /usr/libexec/ipsec/pluto --nofork --secretsfile /etc/ipsec.secrets --ipsecdir ...
├ 1069 pluto helper # 0
├ 1070 pluto helper # 1
├ 1071 pluto helper # 2
└ 1223 _pluto_adns
It's not just hacking at some opaque *sh hacks (like debian init or even
interactive-mode baselayout) and takes so little effort to the point that it's
really enjoyable process.
But making it mount and start all the default (and available) stuff is not the
end of it, because there are plenty of services not yet adapted to systemd.
I actually expected some (relatively) hard work here, because there are quite
a few initscripts in /etc/init.d, even on a desktop machine, but once again, I
was in for a nice surprise, since systemd just makes all the work go away. All
you need to do is to decide on the ordering (or copy it from baselayout
scripts) and put an appropriate "Type=" and "ExecStart=" lines in .service
file. That's all there is, really.
After that, of course, complete bootup-shutdown test on a vm is in order, and
everything "just works" as it is supposed to.
Bootup on a real hardware is exactly the same as vm, no surprises here.
"udevadm trigger" seem to be necessary as well, proving validity of vm model.
Systemd boot time is way faster than sysvinit, as it is supposed to, although I
don't really care, since reboot is seldom necessary here.
As a summary, I'd recommend everyone to give systemd a try, or at least get
familiar with it's rationale
and features (plus this three-part
blog series: one, two, three). My units aren't perfect (and I'll probably update
network-related stuff to use ConnMan), but if you're
lazy, grab them here. Also, here is a
repo with units for archlinux, which I loosely used as a
reference point along with /lib/systemd contents.
Sep 12, 2010
Thanks to feedjack, I'm able to keep
in sync with 120 feeds (many of them, like slashdot or
reddit, being an aggregates as well), as of
today. Quite a lot of stuff I couldn't even imagine handling a year ago, and a
good aggregator definitely helps, keeping all the info just one click away.
And every workstation-based (desktop) aggregator I've seen is a fail:
- RSSOwl. Really nice interface and very
powerful. That said, it eats more ram than a firefox!!! Hogs CPU till the
whole system stutters, and eats more of it than every other app I use combined
(yes, including firefox). Just keeping it in the background costs 20-30% of
dualcore cpu. Changing "show new" to "show all" kills the system ;)
- liferea. Horribly slow, interface hangs on any
action (like fetching feed "in the background"), hogs cpu just as RSSOwl and
not quite as feature-packed.
- Claws-mail's RSSyl. Quite nice
resource-wise and very responsive, unlike dedicated software (beats me
why). Pity it's also very limited interface-wise and can't reliably keep track
of many of feeds by itself, constantly loosing a few if closed non-properly
(most likely it's a claws-mail fault, since it affects stuff like nntp as
well).
- Emacs' gnus and newsticker. Good for a feed or two, epic fail in every way
with more dozen of them.
- Various terminal-based readers. Simply intolerable.
Server-based aggregator on the other hand is a bliss - any hoards of stuff as
you want it, filtered, processed, categorized and re-exported to any format
(same rss, but not a hundred of them, for any other reader works as well) and
I don't give a damn about how many CPU-hours it spends doing so (yet it tend
to be very few, since processing and storage is done via production-grade
database and modules, not some crappy ad-hoc wheel re-invention).
And it's simple as a doorknob, so any extra functionality can be added with no
effort.
Maybe someday I'll get around to use something like Google Reader, but it's
still one hell of a mess, and it's no worse than similar web-based services out
there. So much for the cloud services. *sigh*
Sep 09, 2010
Ok, almost one month of glusterfs
was too much for me to handle. That was an epic fail ;)
Random errors on start (yeah, just restart nodes a few times and it'll be
fine) and during operation (same ESTALE, EIOs for whole mount, half of files
just vanishing) seem to be a norm for it. I mean, that's with a perfectly sane
and calm conditions - everything works, links stable.
A bit complicated configurations like server-side replication seem to be the
cause of these, sometimes to the point when the stuff just gives ESTALE in
100% cases right from the start w/o any reason I can comprehend. And adding a
third node to the system just made things worse and configuration files are
even more scary.
Well, maybe I'm just out of luck or too brain-dead for it, whatever.
So, moving on, I've tried (although briefly) ceph.
Being in mainline kernel, and not just the staging part, I'd expected it to be
much less work-in-progress, but as it is, it's very raw, to the point that
x86_64 monitor daemon just crashes upon receiving data from plain x86. Interface
is a bunch of simple shell scripts, fairly opaque operation, and the whole thing
is built on such crap as boost.
Managed to get it running with two nodes, but it feels like the end of the
world - one more kick and it all falls to pieces. Confirmed by the reports all
over the mailing list and #ceph.
In-kernel and seemingly fast is a good mix though, so I may get back to it
eventually, but now I'd rather prefer to settle on something that actually
works.
Next thing in my sight was tahoe-lafs, but it still
lacks normal posix-fs interface layer, sftp interface being totally unusable on
1.8.0c3 - no permissions, cp -R fails w/ I/O error, displayed data in
inconsistent even with locally-made changes, and so on. A pity, whole system
design looks very cool, with raid5-like "parity" instead of plain chunk
replication, and it's python!
Thus I ended up with MooseFS.
First thing to note here is incredibly simple and yet infinitely more powerful
interface that probably sold me the fs right from the start.
None of this configuration layers hell of gluster, just a line in hosts (so
there's no need to tweak configs at all!) plus a few about what to export
(subtrees-to-nets, nfs style) and where to put chunks (any fs as a simple
backend key-value storage), and that's about it.
Replication? Piece a cake, and it's configured on per-tree basis, so important
or compact stuff can have one replication "goal" and some heavy trash in the
neighbor path have no replication at all. No chance of anything like this with
gluster and it's not even documented for ceph.
Performance is totally I/O and network bound (which is totally not-the-case with
tahoe, for instance), so no complaints here as well.
One more amazing thing is how simple and transparent it is:
fraggod@anathema:~% mfsgetgoal tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso
tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso: 2
fraggod@anathema:~% mfsfileinfo tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso
tmp/db/softCore/_nix/os/systemrescuecd-x86-1.5.8.iso:
chunk 0: 000000000000CE78_00000001 / (id:52856 ver:1)
copy 1: 192.168.0.8:9422
copy 2: 192.168.0.11:9422
chunk 1: 000000000000CE79_00000001 / (id:52857 ver:1)
copy 1: 192.168.0.10:9422
copy 2: 192.168.0.11:9422
chunk 2: 000000000000CE7A_00000001 / (id:52858 ver:1)
copy 1: 192.168.0.10:9422
copy 2: 192.168.0.11:9422
chunk 3: 000000000000CE7B_00000001 / (id:52859 ver:1)
copy 1: 192.168.0.8:9422
copy 2: 192.168.0.10:9422
chunk 4: 000000000000CE7C_00000001 / (id:52860 ver:1)
copy 1: 192.168.0.10:9422
copy 2: 192.168.0.11:9422
fraggod@anathema:~% mfsdirinfo tmp/db/softCore/_nix/os
tmp/db/softCore/_nix/os:
inodes: 12
directories: 1
files: 11
chunks: 175
length: 11532174263
size: 11533462528
realsize: 23066925056
And if that's not enough, there's even a cow snaphots, trash bin with a
customizable grace period and a special attributes for file caching and
ownership, all totally documented along with the architectural details in
manpages and on the project site.
Code is plain C, no shitload of deps like boost and lib*whatevermagic*, and
it's really lite. Whole thing feels like a simple and solid design, not some
polished turd of a *certified professionals*.
Yes, it's not truly-scalable, as there's a master host (with optional
metalogger failover backups) with fs metadata, but there's no chance it'll be
a bottleneck in my setup and comparing to a "no-way" bottlenecks of other
stuff, I'd rather stick with this one.
MooseFS has yet to pass the trial of time on my makeshift "cluster", yet none of
the other setups went (even remotely) as smooth as this one so far, thus I feel
pretty optimistic about it.
Aug 15, 2010
Hardware
I'd say "every sufficiently advanced user is indistinguishable from a
sysadmin" (yes, it's a play on famous Arthur C Clarke's quote), and it doesn't
take much of "advanced" to come to a home-server idea.
And I bet the main purpose for most of these aren't playground, p2p client or
some http/ftp server - it's just a storage. Serving and updating the stored
stuff is kinda secondary.
And I guess it's some sort of nature law that any storage runs outta free
space sooner or later. And when this happens, just buying more space seem to
be a better option than cleanup because a) "hey, there's dirt-cheap 2TB
harddisks out there!" b) you just get used to having all that stuff at
packet's reach.
Going down this road I found myself out of connectors on the motherboard
(which is fairly spartan
D510MO miniITX)
and the slots for an extension cards (the only PCI is used by dual-port nic).
So I hooked up two harddisks via usb, but either the usb-sata controllers or
usb's on the motherboard were faulty and controllers just hang with both leds
on, vanishing from the system. Not that it's a problem - just mdraid'ed them
into raid1 and when one fails like that, I just have to replug it and start
raid recovery, never losing access to the data itself.
Then, to extend the storage capacity a bit further (and to provide a backup to
that media content) I just bought +1 miniITX unit.
Now, I could've mouned two NFS'es from both units, but this approach has
several disadvantages:
- Two mounts instead of one. Ok, not a big deal by itself.
- I'd have to manage free space on these by hand, shuffling subtrees between
them.
- I need replication for some subtrees, and that complicates the previous point
a bit further.
- Some sort of HA would be nice, so I'd be able to shutdown one replica and
switch to using another automatically.
The obvious improvement would be to use some distributed network filesystem,
and pondering on the possibilities I've decided to stick with the glusterfs
due to it's flexible yet very simple "layer cake" configuration model.
Oh, and the most important reason to set this whole thing up - it's just fun ;)
The Setup
Ok, what I have is:
- Node1
- physical storage (raid1) "disk11", 300G, old and fairly "valuable" data
(well, of course it's not "valuable", since I can just re-download it all
from p2p, but I'd hate to do that)
- physical disk "disk12", 150G, same stuff as disk11
- Node2
- physical disk "disk2", 1.5T, blank, to-be-filled
What I want is one single network storage, with db1 (disk11 + disk12) data
available from any node (replicated) and new stuff which won't fit onto this
storage should be writen to db2 (what's left of disk2).
With glusterfs there are several ways to do this:
Scenario 1: fully network-aware client.
That's actually the simplest scenario - glusterfsd.vol files on "nodes" should
just export local disks and client configuration ties it all together.
Pros:
- Fault tolerance. Client is fully aware of the storage hierarchy, so if one
node with db1 is down, it will just use the other one.
- If the bandwidth is better than disk i/o, reading from db1 can be potentially
faster (dunno if glusterfs allows that, actually), but that's not the case,
since "client" is one of the laptops and it's a slow wifi link.
Cons:
- Write performance is bad - client has to push data to both nodes, and that's a
- big minus with my link.
Scenario 2: server-side replication.
Here, "nodes" export local disks for each other and gather local+remote db1 into
cluster/replicate and then export this already-replicated volume. Client just
ties db2 and one of the replicated-db1 together via nufa or distribute layer.
Pros:
Cons:
- Single point of failure, not only for db2 (which is natural, since it's not
replicated), but for db1 as well.
Scenario 3: server-side replication + fully-aware client.
db1 replicas are synced by "nodes" and client mounts all three volumes (2 x db1,
1 x db2) with either cluster/unify layer and nufa scheduler (listing both db1
replicas in "local-volume-name") or cluster/nufa.
That's the answer to obvious question I've asked myself after implementing
scenario 2: "why not get rid of this single_point_of_failure just by using
not single, but both replicated-db1 volumes in nufa?"
In this case, if node1 goes down, client won't even notice it! And if that
happens to node2, files from db2 just disappear from hierarchy, but db1 will
still remain fully-accessible.
But there is a problem: cluster/nufa has no support for multiple
local-volume-name specs. cluster/unify has this support, but requires it's ugly
"namespace-volume" hack. The solution would be to aggregate both db1's into a
distribute layer and use it as a single volume alongside db2.
With aforementioned physical layout this seem to be just the best all-around
case.
Pros:
- Best performance and network utilization.
Cons:
Implementation
So, scenarios 2 and 3 in terms of glusterfs, with the omission of different
performance, lock layers and a few options, for the sake of clarity:
node1 glusterfsd.vol:
## db1: replicated node1/node2
volume local-db1
type storage/posix
option directory /srv/storage/db1
end-volume
# No client-caches here, because ops should already come aggregated
# from the client, and the link between servers is much fatter than the client's
volume node2-db1
type protocol/client
option remote-host node2
option remote-subvolume local-db1
end-volume
volume composite-db1
type cluster/replicate
subvolumes local-db1 node2-db1
end-volume
## db: linear (nufa) db1 + db2
## export: local-db1 (for peers), composite-db1 (for clients)
volume export
type protocol/server
subvolumes local-db1 composite-db1
end-volume
node2 glusterfsd.vol:
## db1: replicated node1/node2
volume local-db1
type storage/posix
option directory /srv/storage/db1
end-volume
# No client-caches here, because ops should already come aggregated
# from the client, and the link between servers is much fatter than the client's
volume node1-db1
type protocol/client
option remote-host node1
option remote-subvolume local-db1
end-volume
volume composite-db1
type cluster/replicate
subvolumes local-db1 node1-db1
end-volume
## db2: node2
volume db2
type storage/posix
option directory /srv/storage/db2
end-volume
## db: linear (nufa) db1 + db2
## export: local-db1 (for peers), composite-db1 (for clients)
volume export
type protocol/server
subvolumes local-db1 composite-db1
end-volume
client (replicated to both nodes):
volume node1-db1
type protocol/client
option remote-host node1
option remote-subvolume composite-db1
end-volume
volume node2-db1
type protocol/client
option remote-host node2
option remote-subvolume composite-db1
end-volume
volume db1
type cluster/distribute
option remote-subvolume node1-db1 node2-db1
end-volume
volume db2
type protocol/client
option remote-host node2
option remote-subvolume db2
end-volume
volume db
type cluster/nufa
option local-volume-name db1
subvolumes db1 db2
end-volume
Actually there's one more scenario I thought of for non-local clients - same
as 2, but pushing nufa into glusterfsd.vol on "nodes", thus making client
mount single unified volume on a single host via single port in a single
connection.
Not that I really need this one, since all I need to mount from external
networks is just music 99.9% of time, and
NFS + FS-Cache
offer more advantages there, although I might resort to it in the future, when
music won't fit to db1 anymore (doubt that'll happen anytime soon).
P.S.
Configs are fine, but the most important thing for setting up glusterfs are
these lines:
node# /usr/sbin/glusterfsd --debug --volfile=/etc/glusterfs/glusterfsd.vol
client# /usr/sbin/glusterfs --debug --volfile-server=somenode /mnt/tmp
Jun 14, 2010
Guess being a long user of stuff like OpenSSH,
iproute2
and VDE rots your brain - you start thinking
that building any sort of tunnel is a bliss. Well, it's not. At least not "any
sort".
This day I've dedicated to set up some basic IPSec tunnel and at first that
seemed an easy task - it's long ago in kernel (
kame
implementation, and it's not the only one for linux), native for IPv6 (which I
use in a local network), has quite a lot of publicity (and guides), it's open
(and is quite simple, even with IKE magic) and there are at least three major
userspace implementations:
openswan,
ipsec-tools (racoon, kame) and
Isakmpd. Hell, it's even supported on Windows. What's more to ask for?
Well, prehaps I made a bad decision starting with openswan and "native" kame
NETKEY, but my experience wasn't quite a nice one.
I chose openswan because it looks like more extensive implementation than the
rest, and is supported by folks like Red Hat, plus it is fairly up to date and
looks constantly developed. Another cherry in it was apparent smartcard support
via nss now and opensc in the past.
Latest version of it in ebuild form (which isn't quite enough for me anyway,
since I use exheres these days) is 2.6.23. That's more than half a year old,
and even that one is masked in gentoo due to apparent bugs and the ebuild is
obviously blind-bumped from some previous version, since it doesn't take
things like opensc->nss move (finalized in 2.6.23) into account.
Okay, hacking my own ebuild and exheres for it was fun enough, at least I've
got a firm grasp of what it's capable of, but seeing pure-Makefile build
system and hard-coded paths in such a package was a bit unexpected. Took me
some time to deal with include paths, then lib paths, then it turned out to
had an
open bug which prevents
it's build on linux (wtf!?), and then it crashes on install phase due to some
ever-crappy XML stuff.
At least the docs are good enough (even though it's not easy to build them),
so I set up an nss db, linked smartcard to it, and got a... segfault? Right
on ipsec showhostkey? Right, there's
this bug
in 2.6.26, although in my case it's probably another one, since the patch
doesn't fixes the problem. Great!
Ok, gdb showed that it's something like get-nss-password failing (although it
should be quite a generic interface, delegated from nss), even with
nsspassword in place and nss itself working perfectly.
Scratch that, simple nss-generated keys (not even certificates) as described
in the
most basic tutorial, and now it's pluto
daemon crashing with just a "Jun 14 15:40:25 daemon.err<27>
ipsec__plutorun[-]: /usr/lib/ipsec/_plutorun: line 244: 6229 Aborted ..."
line in syslog as soon as both ends off tunnel are up.
Oh, and of course it messes up the connection between hosts in question, so it
wouldn't be too easy to ssh between them and debug the problem.
Comparing to ssh or pretty much any tunneling I've encountered to this point,
it's still quite a remarkably epic fail. Guess I'll waste a bit more time on
this crap, since success seems so close, but it's quite amazing how crappy such
projects can still be these days. Of course, at least it's free, right?
Jun 13, 2010
There's one great app -
bournal ("when
nobody cares what you have to say!"). Essentialy it's a bash script, providing
a simple interface to edit and encrypt journal entries.
Idea behind it is quite opposite of blogging - keep your thoughts as far away
from everyone as possible. I've used the app for quite a while, ever since
I've noticed it among freshmeat release announcements. It's useful to keep
some thoughts or secrets (like keys or passwords) somewhere, aside from the
head, even if you'd never read these again.
Anyway, encryption there is done by the means of ccrypt utility, which is sorta CLI for openssl. I don't get the rationale behind using it instead
of openssl directly (like "openssl enc ..."), and there are actually even better
options, like gnupg, which won't need a special logic
to keep separate stream-cipher password, like it's done in bournal.
So today, as I needed bournal on exherbo laptop, I've faced the need to get
ccrypt binary just for that purpose again. Worse yet, I have to recall and
enter a password I've used there, and I don't actually need it to just encrypt
an entry... as if assymetric encryption, gpg-agent, smartcards and all the
other cool santa helpers don't exist yet.
I've decided to hack up my "ccrypt" which will use all-too-familiar gpg and
won't ask me for any passwords my agent or scd already know, and in an hour or
so, I've succeeded.
And
here goes - ccrypt, relying
only on "gpg -e -r $EMAIL" and "gpg -d". EMAIL should be in the env, btw.
It actually works as ccencrypt, ccdecrypt, ccat as well, and can do recursive
ops just like vanilla ccrypt, which is enough for bournal.
Jun 05, 2010
If you're downloading stuff off the 'net via bt like me, then TPB is probably quite a familiar place to you.
Ever since the '09 trial there were a problems with TPB trackers
(tracker.thepiratebay.org) - the name gets inserted into every .torrent yet it
points to 127.0.0.1.
Lately, TPB offshot, tracker.openbittorrent.com suffers from the same problem
and actually there's a lot of other trackers in .torrent files that point
either at 0.0.0.0 or 127.0.0.1 these days.
As I use
rtorrent, I have an issue with
this - rtorrent seem pretty dumb when it comes to tracker filtering so it
queries all of them on a round-robin basis, without checking where it's name
points to or if it's down for the whole app uptime, and queries take quite a
lot of time to timeout, so that means at least two-minute delay in starting
download (as there's TPB trackers first), plus it breaks a lot of other things
like manual tracker-cycling ("t" key), since it seem to prefer only top-tier
trackers and these are 100% down.
Now the problem with http to localhost can be solved with the firewall, of
course, although it's an ugly solution, and 0.0.0.0 seem to fail pretty fast
by itself, but stateless udp is still a problem.
Another way to tackle the issue is probably just to use a client that is
capable of filtering the trackers by ip address, but that probably means some
heavy-duty stuff like azureus or vanilla bittorrent which I've found pretty
buggy and also surprisingly heavy in the past.
So the easiest generic solution (which will, of course, work for rtorrent) I've
found is just to process the .torrent files before feeding them to the leecher
app. Since I'm downloading these via firefox exclusively, and there I use
FlashGot (not the standard "open with" interface since
I also use it to download large files on remote machine w/o firefox, and afaik
it's just not the way "open with" works) to drop them into an torrent bin via
script, it's actually a matter of updating the link-receiving script.
Bencode is not a mystery, plus it's
pure-py implementation is actually the reference one, since it's a part of
original python bittorrent client, so all I basically had to do is to rip
bencode.py from it and paste the relevant part into the script.
The right way might've been to add dependency on the whole bittorrent client,
but it's an overkill for such a simple task plus the api itself seem to be
purely internal and probably a subject to change with client releases anyway.
So, to the script itself...
# URL checker
def trak_check(trak):
if not trak: return False
try: ip = gethostbyname(urlparse(trak).netloc.split(':', 1)[0])
except gaierror: return True # prehaps it will resolve later on
else: return ip not in ('127.0.0.1', '0.0.0.0')
# Actual processing
torrent = bdecode(torrent)
for tier in list(torrent['announce-list']):
for trak in list(tier):
if not trak_check(trak):
tier.remove(trak)
# print >>sys.stderr, 'Dropped:', trak
if not tier: torrent['announce-list'].remove(tier)
# print >>sys.stderr, 'Result:', torrent['announce-list']
if not trak_check(torrent['announce']):
torrent['announce'] = torrent['announce-list'][0][0]
torrent = bencode(torrent)
That, plus the simple "fetch-dump" part, if needed.
No magic of any kind, just a plain "announce-list" and "announce" urls check,
dropping each only if it resolves to that bogus placeholder IPs.
I've wanted to make it as light as possible so no logging or optparse/argparse
stuff I tend cram everywhere I can, and the extra and heavy imports like
urllib/urllib2 are conditional as well. The only dependency is python (>=2.6)
itself.
Basic use-case is one of these:
% brecode.py < /path/to/file.torrent > /path/to/proccessed.torrent
% brecode.py http://tpb.org/torrents/myfile.torrent > /path/to/proccessed.torrent
% brecode.py http://tpb.org/torrents/myfile.torrent\
-r http://tpb.org/ -c "...some cookies..." -d /path/to/torrents-bin/
All the extra headers like cookies and referer are optional, so is the
destination path (dir, basename is generated from URL). My use-case in FlashGot
is this: "[URL] -r [REFERER] -c [COOKIE] -d /mnt/p2p/bt/torrents"
And there's the script itself.
Quick, dirty and inconclusive testing showed almost 100 KB/s -> 600 KB/s
increase on several different (two successive tests on the same file even with
clean session are obviously wrong) popular and unrelated .torrent files.
That's pretty inspiring. Guess now I can waste even more time on the TV-era
crap than before, oh joy ;)