Dec 23, 2014

A few recent emacs features - remote and file colors

I've been using emacs for a while now, and always on a lookout for a features that'd be nice to have there. Accumulated quite a number of these in my emacs-setup repo as a result.

Most of these features start from ideas in other editors or tools (e.g. music players, irc clients, etc - emacs seem to be best for a lot of stuff), or a simplistic proof-of-concept implementation of something similar.
I usually add these to my emacs due to sheer fun of coding in lisps, compared to pretty much any other lang family I know of.

Recently added two of these, and wanted to share/log the ideas here, in case someone else might find these useful.

"Remote" based on emacsclient tool

As I use laptop and desktop machines for coding and procrastination interchangeably, can have e.g. irc client (ERC - seriously the best irc client I've seen by far) running on either of these.

But even with ZNC bouncer setup (and easy log-reading tools for it), it's still a lot of hassle to connect to same irc from another machiine and catch-up on chan history there.

Or sometimes there are unsaved buffer changes, or whatever other stuff happening, or just stuff you want to do in a remote emacs instance, which would be easy if you could just go turn on the monitor, shrug-off screen blanking, sometimes disable screen-lock, then switch to emacs and press a few hotkeys there... yeah, it doesn't look that easy even when I'm at home and close to the thing.

emacs has "emacsclient" thing, that allows you to eval whatever elisp code on a remote emacs instance, but it's impossible to use for such simple tasks without some convenient wrappers.

And these remote invocation wrappers is what this idea is all about.

Consider terminal dump below, running in an ssh or over tcp to remote emacs server (and I'd strongly suggest having (server-start) right in ~/.emacs, though maybe not on tcp socket for security reasons):

% ece b
* 2014-12-23.a_few_recent_emacs_features_-_remote_and_file_colors.rst
  fg_remote.el
  rpc.py
* fg_erc.el
  utils.py
  #twitter_bitlbee
  #blazer
  #exherbo
  ...

% ece b remote
*ERROR*: Failed to uniquely match buffer by `remote', matches:
2014-12-23.a_few_recent_emacs_features_-_remote_and_file_colors.rst,
fg_remote.el

--- whoops... lemme try again

% ece b fg_rem
...(contents of the buffer, matched by unique name part)...

% ece erc
004 #twitter_bitlbee
004 #blazer
002 #bordercamp

--- Showing last (unchecked) irc activity, same as erc-track-mode does (but nicer)

% ece erc twitter | t
[13:36:45]<fijall> hint - you can use gc.garbage in Python to store any sort of ...
[14:57:59]<veorq> Going shopping downtown. Pray for me.
[15:48:59]<mitsuhiko> I like how if you google for "London Bridge" you get ...
[17:15:15]<simonw> TIL the Devonshire word "frawsy" (or is it "frawzy"?) - a ...
[17:17:04] *** -------------------- ***
[17:24:01]<veorq> RT @collinrm: Any opinions on VeraCrypt?
[17:33:31]<veorq> insightful comment by @jmgosney about the Ars Technica hack ...
[17:35:36]<veorq> .@jmgosney as you must know "iterating" a hash is in theory ...
[17:51:50]<veorq> woops #31c3 via @joernchen ...
~erc/#twitter_bitlbee%

--- "t" above is an alias for "tail" that I use in all shells, lines snipped jic

% ece h
Available commands:
  buffer (aliases: b, buff)
  buffer-names
  erc
  erc-mark
  get-socket-type
  help (aliases: h, rtfm, wat)
  log
  switch-sockets

% ece h erc-mark
(fg-remote-erc-mark PATTERN)

Put /mark to a specified ERC chan and reset its activity track.

--- Whole "help" thing is auto-generated, see "fg-remote-help" in fg_remote.el

And so on - anything is trivial to implement as elisp few-liner. For instance, missing "buffer-save" command will be:

(defun fg-remote-buffer-save (pattern)
  "Saves specified bufffer, matched via `fg-get-useful-buffer'."
  (with-current-buffer (fg-get-useful-buffer pattern) (save-buffer)))
(defalias 'fg-remote-bs 'fg-remote-buffer-save)
Both "bufffer-save" command and its "bs" alias will instantly appear in "help" and be available for calling via emacs client.
Hell, you can "implement" this stuff from terminal and eval on a remote emacs (i.e. just pass code above to emacsclient -e), extending its API in an ad-hoc fashion right there.

"ece" script above is a thin wrapper around "emacsclient" to avoid typing that long binary name and "-e" flag with a set of parentheses every time, can be found in the root of emacs-setup repo.

So it's easier to procrastinate in bed whole morning with a laptop than ever.
Yup, that's the real point of the whole thing.

Unique per-file buffer colors

Stumbled upon this idea in a deliberate-software blog entry recently.

There, author suggests making static per-code-project colors, but I thought - why not have slight (and automatic) per-file-path color alterations for buffer background?

Doing that makes file buffers (or any non-file ones too) recognizable, i.e. you don't need to look at the path or code inside anymore to instantly know that it's that exact file you want (or don't want) to edit - eye/brain picks it up automatically.

emacs' color.el already has all the cool stuff for colors - tools for conversion to/from L*a*b* colorspace (humane "perceptual" numbers), CIEDE2000 color diffs (JUST LOOK AT THIS THING), and so on - easy to use these for the task.

Result is "fg-color-tweak" function that I now use for slight changes to buffer bg, based on md5 hash of the file path and reliably-contrast irc nicknames (based also on the hash, used way worse and unreliable "simple" thing for this in the past):

(fg-color-tweak COLOR &optional SEED MIN-SHIFT MAX-SHIFT (CLAMP-RGB-AFTER 20)
  (LAB-RANGES ...))

Adjust COLOR based on (md5 of-) SEED and MIN-SHIFT / MAX-SHIFT lists.

COLOR can be provided as a three-value (0-1 float)
R G B list, or a string suitable for `color-name-to-rgb'.

MIN-SHIFT / MAX-SHIFT can be:
 * three-value list (numbers) of min/max offset on L*a*b* in either direction
 * one number - min/max cie-de2000 distance
 * four-value list of offsets and distance, combining both options above
 * nil for no-limit

SEED can be number, string or nil.
Empty string or nil passed as SEED will return the original color.

CLAMP-RGB-AFTER defines how many attempts to make in picking
L*a*b* color with random offset that translates to non-imaginary sRGB color.
When that number is reached, last color will be `color-clamp'ed to fit into sRGB.

Returns color plus/minus offset as a hex string.
Resulting color offset should be uniformly distributed between min/max shift limits.

It's a bit complicated under the hood, parsing all the options and limits, making sure resulting color is not "imaginary" L*a*b* one and converts to RGB without clamping (if possible), while maintaining requested min/max distances, doing several hashing rounds if necessary, with fallbacks... etc.

Actual end-result is simple though - deterministic and instantly-recognizable color-coding for anything you can think of - just pass the attribute to base coding on and desired min/max contrast levels, get back the hex color to use, apply it.

Should you use something like that, I highly suggest taking a moment to look at L*a*b* and HSL color spaces, to understand how colors can be easily tweaked along certain parameters.
For example, passing '(0 a b) as min/max-shift to the function above will produce color variants with the same "lightness", which is super-useful to control, making sure you won't ever get out-of-whack colors for e.g. light/dark backgrounds.

To summarize...

Coding lispy stuff is super-fun, just for the sake of it ;)

Actually, speaking of fun, I can't recommend installing magnars' s.el and dash.el right now highly enough, unless you have these already.
They make coding elisp stuff so much more fun and trivial, to a degree that'd be hard to describe, so please at least try coding somethig with these.

All the stuff mentioned above is in (also linked here already) emacs-setup repo.

Cheers!

Jun 15, 2014

Running isolated Steam instance with its own UID and session

Finally got around to installing Steam platform to a desktop linux machine.
Been using Win7 instance here for games before, but as another fan in my laptop died, have been too lazy to reboot into dedicated games-os here.

Given that Steam is a closed-source proprietary DRM platform for mass software distribution, it seem to be either an ideal malware spread vector or just a recipie for disaster, so of course not keen on giving it any access in a non-dedicated os.

I also feel a bit guilty on giving the thing any extra PR, as it's the worst kind of always-on DRM crap in principle, and already pretty much monopolized PC Gaming market.
These days even many game critics push for filtering and essentially abuse of that immense leverage - not a good sign at all.
To its credit, of course, Steam is nice and convenient to use, as such things (e.g. google, fb, droids, apple, etc) tend to be.

So, isolation:

  • To avoid having Steam and any games anywhere near $HOME, giving it separate UID is a good way to go.

  • That should also allow for it to run in a separate desktop session - i.e. have its own cgroup, to easily contain, control and set limits for games:

    % loginctl user-status steam
    steam (1001)
      Since: Sun 2014-06-15 18:40:34 YEKT; 31s ago
      State: active
      Sessions: *7
        Unit: user-1001.slice
              └─session-7.scope
                ├─7821 sshd: steam [priv]
                ├─7829 sshd: steam@notty
                ├─7830 -zsh
                ├─7831 bash /usr/bin/steam
                ├─7841 bash /home/steam/.local/share/Steam/steam.sh
                ├─7842 tee /tmp/dumps/steam_stdout.txt
                ├─7917 /home/steam/.local/share/Steam/ubuntu12_32/steam
                ├─7942 dbus-launch --autolaunch=e52019f6d7b9427697a152348e9f84ad ...
                └─7943 /usr/bin/dbus-daemon --fork --print-pid 5 ...
    
  • AppArmor should allow to further isolate processes from having any access beyond what's absolutely necessary for them to run, warn when these try to do strange things and allow to just restrict these from doing outright stupid things.

  • Given separate UID and cgroup, network access from all Steam apps can be easily controlled via e.g. iptables, to avoid Steam and games scanning and abusing other things in LAN, for example.


Creating steam user should be as simple as useradd steam, but then switching to that UID from within a running DE should still allow it to access same X server and start systemd session for it, plus not have any extra env, permissions, dbus access, fd's and such from the main session.

By far the easiest way to do that I've found is to just ssh steam@localhost, putting proper pubkey into ~steam/.ssh/authorized_keys first, of course.
That should ensure that nothing leaks from DE but whatever ssh passes, and it's rather paranoid security-oriented tool, so can be trusted with that .
Steam comes with a bootstrap script (e.g. /usr/bin/steam) to install itself, which also starts the thing when it's installed, so Steam AppArmor profile (github link) is for that.
It should allow to both bootstrap and install stuff as well as run it, yet don't allow steam to poke too much into other shared dirs or processes.

To allow access to X, xhost or ~/.Xauthority cookie can be used along with some extra env in e.g. ~/.zshrc:

export DISPLAY=':1.0'

In similar to ssh fashion, I've used pulseaudio network streaming to main DE sound daemon on localhost for sound (also in ~/.zshrc):

export PULSE_SERVER='{e52019f6d7b9427697a152348e9f84ad}tcp6:malediction:4713'
export PULSE_COOKIE="$HOME"/.pulse-cookie

(I have pulse network streaming setup anyway, for sharing sound from desktop to laptop - to e.g. play videos on a big screen there yet hear sound from laptop's headphones)

Running Steam will also start its own dbus session (maybe it's pulse client lib doing that, didn't check), but it doesn't seem to be used for anything, so there seem to be no need to share it with main DE.


That should allow to start Steam after ssh'ing to steam@localhost, but process can be made much easier (and more foolproof) with e.g. ~/bin/steam as:

#!/bin/bash

cmd=$1
shift

steam_wait_exit() {
  for n in {0..10}; do
    pgrep -U steam -x steam >/dev/null || return 0
    sleep 0.1
  done
  return 1
}

case "$cmd" in
  '')
    ssh steam@localhost <<EOF
source .zshrc
exec steam "$@"
EOF
    loginctl user-status steam ;;

  s*) loginctl user-status steam ;;

  k*)
    steam_exited=
    pgrep -U steam -x steam >/dev/null
    [[ $? -ne 0 ]] && steam_exited=t
    [[ -z "$steam_exited" ]] && {
      ssh steam@localhost <<EOF
source .zshrc
exec steam -shutdown
EOF
      steam_wait_exit
      [[ $? -eq 0 ]] && steam_exited=t
    }
    sudo loginctl kill-user steam
    [[ -z "$steam_exited" ]] && {
      steam_wait_exit || sudo loginctl -s KILL kill-user steam
    } ;;

  *) echo >&2 "Usage: $(basename "$0") [ status | kill ]"
esac

Now just steam in the main DE will run the thing in its own $HOME.

For further convenience, there's steam status and steam kill to easily monitor or shutdown running Steam session from the terminal.

Note the complicated shutdown thing - Steam doesn't react to INT or TERM signals cleanly, passing these to the running games instead, and should be terminated via its own cli option (and the rest can then be killed-off too).


With this setup, iptables rules for outgoing connections can use user-slice cgroup match (in 3.14 at least) or -m owner --uid-owner steam matches for socket owner uid.

The only non-WAN things Steam connects to here are DNS servers and aforementioned pulseaudio socket on localhost, the rest can be safely firewalled.


Finally, running KSP there on Exherbo, I quickly discovered that sound libs and plugins - alsa and pulse - in ubuntu "runtime" steam bootstrap setups don't work well - either there's no sound or game fails to load at all.

Easy fix is to copy the runtime it uses (32-bit one for me) and cleanup alien stuff from there for what's already present in the system, i.e.:

% cp -R .steam/bin32/steam-runtime my-runtime
% find my-runtime -type f\
  \( -path '*asound*' -o -path '*alsa*' -o -path '*pulse*' \) -delete

And then add something like this to ~steam/.zshrc:

steam() { STEAM_RUNTIME="$HOME"/my-runtime command steam "$@"; }

That should keep all of the know-working Ubuntu libs that steam bootsrap gets away from the rest of the system (where stuff like Mono just isn't needed, and others will cause trouble) while allowing to remove any of them from the runtime to use same thing in the system.

And yay - Kerbal Space Program seem to work here way faster than on Win7.

KSP and Steam on Linux

May 19, 2014

Displaying any lm_sensors data (temperature, fan speeds, voltage, etc) in conky

Conky sure has a ton of sensor-related hw-monitoring options, but it still doesn't seem to be enough to represent even just the temperatures from this "sensors" output:

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage:      +1.39 V  (min =  +0.80 V, max =  +1.60 V)
+3.3V Voltage:      +3.36 V  (min =  +2.97 V, max =  +3.63 V)
+5V Voltage:        +5.08 V  (min =  +4.50 V, max =  +5.50 V)
+12V Voltage:      +12.21 V  (min = +10.20 V, max = +13.80 V)
CPU Fan Speed:     2008 RPM  (min =  600 RPM, max = 7200 RPM)
Chassis Fan Speed:    0 RPM  (min =  600 RPM, max = 7200 RPM)
Power Fan Speed:      0 RPM  (min =  600 RPM, max = 7200 RPM)
CPU Temperature:    +42.0°C  (high = +60.0°C, crit = +95.0°C)
MB Temperature:     +43.0°C  (high = +45.0°C, crit = +75.0°C)

k10temp-pci-00c3
Adapter: PCI adapter
temp1:        +30.6°C  (high = +70.0°C)
                       (crit = +90.0°C, hyst = +88.0°C)

radeon-pci-0400
Adapter: PCI adapter
temp1:        +51.0°C

Given the summertime, and faulty noisy cooling fans, decided that it'd be nice to be able to have an idea about what kind of temperatures hw operates on under all sorts of routine tasks.

Conky is extensible via lua, which - among other awesome things there are - allows to code caches for expensive operations (and not just repeat them every other second) and parse output of whatever tools efficiently (i.e. without forking five extra binaries plus perl).

Output of "sensors" though not only is kinda expensive to get, but also hardly parseable, likely unstable, and tool doesn't seem to have any "machine data" option.

lm_sensors includes a libsensors, which still doesn't seem possible to call from conky-lua directly (would need some kind of ffi), but easy to write the wrapper around - i.e. this sens.c 50-liner, to dump info in a useful way:

atk0110-0-0__in0_input 1.392000
atk0110-0-0__in0_min 0.800000
atk0110-0-0__in0_max 1.600000
atk0110-0-0__in1_input 3.360000
...
atk0110-0-0__in3_max 13.800000
atk0110-0-0__fan1_input 2002.000000
atk0110-0-0__fan1_min 600.000000
atk0110-0-0__fan1_max 7200.000000
atk0110-0-0__fan2_input 0.000000
...
atk0110-0-0__fan3_max 7200.000000
atk0110-0-0__temp1_input 42.000000
atk0110-0-0__temp1_max 60.000000
atk0110-0-0__temp1_crit 95.000000
atk0110-0-0__temp2_input 43.000000
atk0110-0-0__temp2_max 45.000000
atk0110-0-0__temp2_crit 75.000000
k10temp-0-c3__temp1_input 31.500000
k10temp-0-c3__temp1_max 70.000000
k10temp-0-c3__temp1_crit 90.000000
k10temp-0-c3__temp1_crit_hyst 88.000000
radeon-0-400__temp1_input 51.000000

It's all lm_sensors seem to know about hw in a simple key-value form.

Still not keen on running that on every conky tick, hence the lua cache:

sensors = {
  values=nil,
  cmd="sens",
  ts_read_i=120, ts_read=0,
}

function conky_sens_read(name, precision)
  local ts = os.time()
  if os.difftime(ts, sensors.ts_read) > sensors.ts_read_i then
    local sh = io.popen(sensors.cmd, 'r')
    sensors.values = {}
    for p in string.gmatch(sh:read('*a'), '(%S+ %S+)\n') do
      local n = string.find(p, ' ')
      sensors.values[string.sub(p, 0, n-1)] = string.sub(p, n)
    end
    sh:close()
    sensors.ts_read = ts
  end

  if sensors.values[name] then
    local fmt = string.format('%%.%sf', precision or 0)
    return string.format(fmt, sensors.values[name])
  end
  return ''
end

Which can run the actual "sens" command every 120s, which is perfectly fine with me, since I don't consider conky to be an "early warning" system, and more of an "have an idea of what's the norm here" one.

Config-wise, it'd be just cpu temp: ${lua sens_read atk0110-0-0__temp1_input}C, or a more fancy template version with a flashing warning and hidden for missing sensors:

template3 ${color lightgrey}${if_empty ${lua sens_read \2}}${else}\
${if_match ${lua sens_read \2} > \3}${color red}\1: ${lua sens_read \2}C${blink !!!}\
${else}\1: ${color}${lua sens_read \2}C${endif}${endif}

It can then be used simply as ${template3 cpu atk0110-0-0__temp1_input 60} or ${template3 gpu radeon-0-400__temp1_input 80}, with 60 and 80 being manually-specified thresholds beyond which indicator turns red and has blinking "!!!" to get more attention.

Overall result in my case is something like this:

conky sensors display

sens.c (plus Makefile with gcc -Wall -lsensors for it) and my conky config where it's utilized can be all found in de-setup repo on github (or my git mirror, ofc).

May 12, 2014

My Firefox Homepage

Wanted to have some sort of "homepage with my fav stuff, arranged as I want to" in firefox for a while, and finally got resolve to do something about it - just finished a (first version of) script to generate the thing - firefox-homepage-generator.

Default "grid of page screenshots" never worked for me, and while there are other projects that do other layouts for different stuff, they just aren't flexible enough to do whatever horrible thing I want.

In this particular case, I wanted to experiment with chaotic tag cloud of bookmarks (so they won't ever be in the same place), relations graph for these tags and random picks for "links to read" from backlog.

Result is a dynamic d3 + d3.layout.cloud (interactive example of this layout) page without much style:

homepage screenshot
"Mark of Chaos" button in the corner can fly/re-pack tags around.
Clicking tag shows bookmarks tagged as such and fades all other tags out in proportion to how they're related to the clicked one (i.e. how many links share the tag with others).

Started using FF bookmarks again in a meaningful way only recently, so not much stuff there yet, but it does seem to help a lot, especially with these handy awesome bar tricks.

Not entirely sure how useful the cloud visualization or actually having a homepage would be, but it's a fun experiment and a nice place to collect any useful web-surfing-related stuff I might think of in the future.

Repo link: firefox-homepage-generator

Nov 05, 2013

Conky eye candy clocks and meters

So my laptop broke anyway, but on the bright side - I've got fairly large (certainly by my display standards) desktop fullhd screen now.

While restoring OS there, decided to update ~/.conkyrc (see conky), as it was kinda small for this larger screen, so why not put some eye-candy there, while at it?

conky screenshot

Leftmost radial meters show (inner-to-outer) clock with hands and rings right next to them, blue-ish cpu arcs (right-bottom, outer one is load summary, inner ones are per-core), used (non-cache) memory/swap (left), network traffic (top-right, green/red arcs for up/down) and / and /home df arcs (outer top).

On the right it's good ol' binary clock.

All drawings are lua script, all text and graphs below is conky's magic.
Rings are adapted from "Clock Rings" script here, just added background planes and binary clock, because why not...

Whole script to draw the things can be found in de-setup repo on gh along with full conkyrc I currently use.

Nov 01, 2013

Software hacks to fix broken hardware - laptop fan

Had a fan in a laptop dying for a few weeks now, but international mail being universally bad (and me too hopeful about dying fan's lifetime), replacement from ebay is still on its looong way.

Meanwhile, thing started screeching like mad, causing strong vibration in the plastic and stopping/restarting every few seconds with an audible thunk.

Things not looking good, and me being too lazy to work hard enough to be able to afford new laptop, had to do something to postpone this one's imminent death.

Cleaning the dust and hairs out of fan's propeller and heatsink and changing thermal paste did make the thing a bit cooler, but given that it's fairly slim Acer S3 ultrabook, no local repair shop was able to offer any immediate replacement for the fan, so no clean hw fix in reach yet.

Interesting thing about broken fans though, is that they seem to start vibrating madly out of control only beyond certain speed, so one option was to slow the thing down, while keeping cpu cool somehow.

cpupower tool that comes with linux kernel can nicely downclock this i5 cpu to 800 MHz, but that's not really enough to keep fan from spinning madly - some default BIOS code seem to be putting it to 100% at 50C.

Besides, from what I've seen, it seem to be quite counter-productive, making everything (e.g. opening page in FF) much longer, keeping cpu at 100% of that lower rate all the time, which seem to heat it up slower, sure, but to the same or even higher level for the same task (e.g. opening that web page), with side effect being also wasting time.

Luckily, found out that fan on Acer laptops can be controlled using /dev/ports registers, as described on linlap wiki page.
50C doesn't seem to be high for these CPUs at all, and one previous laptop worked fine on 80C all the time, so making threshold for killing the fan higher seem to be a good idea - it's not like there's much to loose anyway.

As acers3fand script linked from the wiki was for a bit different purpose, wrote my own (also lighter and more self-contained) script - fan_control to only put more than ~50% of power to it after it goes beyond 60C and warns if it heats up way more without putting the fan into "wailing death" mode ever, with max being at about 75% power, also reaching for cpupower hack before that.

Such manual control opens up a possibility of cpu overheating though, or otherwise doesn't help much when you run cpu-intensive stuff, and I kinda don't want to worry about some cronjob, stuck dev script or hung DE app killing the machine while I'm away, so one additional hack I could think of is to just throttle CPU bandwidth enough so that:

  • short tasks complete at top performance, without delays.
  • long cpu-intensive stuff gets throttled to a point where it can't generate enough heat and cpu stays at some 60C with slow fan speed.
  • some known-to-be-intensive tasks like compilation get their own especially low limits.

So kinda like cpupower trick, but more fine-grained and without fixed presets one can slow things down to (as lowest bar there doesn't cut it).

Kernel Control Groups (cgroups) turned out to have the right thing for that - "cpu" resource controller there has cfs_quote_us/cfs_period_us knobs to control cpu bandwidth for threads within a specific cgroup.

New enough systemd has the concept of "slices" to control resources for a groups of services, which are applied automatically for all DE stuff as "user.slice" and its "user-<name>.slice" subslices, so all that had to be done is to echo the right values (which don't cause overheating or fan-fail) to that rc's /sys knobs.
Similar generic limitations are easy to apply to other services there by grouping them with Slice= option.

For distinct limits on daemons started from cli, there's "systemd-run" tool these days, and for more proper interactive wrapping, I've had pet cgroup-tools scripts for a while now (to limit cpu priority of heavier bg stuff like builds though).

With that last tweak, situation seem to be under control - no stray app can really kill the cpu and fan doesn't have to do all the hard work to prevent it either, seemingly solving that hardware fail with software measures for now.

Keeping mobile i5 cpu around 50 degrees apparently needs it to spin only barely, yet seem to allow all the desktop stuff to function without noticeable slowdowns or difference.
Makes me wonder why Intel did allow that low-power ARM things fly past it...

Now, if only replacement fan got here before I drop off the nets even with these hacks.

Jun 06, 2013

Firefox - breaking free of webdevs' tyranny

Wanted to share three kinda-big-deal fixes I've added to my firefox:

  • Patch to remove sticky-on-top focus-grabbing "Do you want to activate plugins on this page?" popup.
  • Patch to prevent plugins (e.g. Abode Flash) from ever grabbing firefox hotkeys like "Ctrl + w" (close tab) or F5, forcing to do click outside e.g. YouTube video window to get back to ff.
  • Easy "toggle js" fix for JavaScript on pages grabbing controls like keyboard and mouse (e.g. overriding F5 to retweet instead of reload page, preventing copy-paste if forms and on pages, etc).

Lately, firefox seem to give more-and-more control into the hands of web developers, who seem to be hell-bent on abusing that to make browsing UX a living hell.

FF bug-reports about Flash grabbing all the focus date back to 2001 and are unresolved still.

Sites override Up/Down, Space, PgUp/PgDown, F5, Ctrl+T/W I've no idea why - guess some JS developers just don't use keyboard at all, which is somewhat understandable, combined with the spread of tablet-devices these days.

Overriding clicks in forms to prevent pasting email/password seem to be completely ignoring valid (or so I think) use-case of using some storage app for these.

And native "click-to-play" switch seem to be hilariously unusable in FF, giving cheerful "Hey, there's flash here! Let me pester you with this on every page load!" popups.

All are known, neither one seem to be going away anytime soon, so onwards to the fixes.

Removing "Do you want to activate plugins" thing seem to be straightforward js one-liner patch, as it's implemented in "browser/base/content/browser-plugins.js" - whole fix is adding this._notificationDisplayedOnce = true; to break the check there.
"notificationDisplayedOnce" thing is used to not popup that thing on the same page within the same browing session afaict.
With this patch applied (more up-to-date github link: no_plugins_popup.patch) it will never pester user again, ever \o/
Patch for plugin focus is clever - all one has to do is to switch focus to browser window (from embedded flash widget) before keypress gets processed and ff will handle it correctly.
Hackish plugin + ad-hoc perl script solution (to avoid patching/rebuilding ff) can be found here.
My hat goes to Alexander Rødseth however, who hacked the patch attached to ff-bug-78414 - this one is a real problem-solver, though a bit (not terribly - just context lines got shuffled around since) out-of-date.
More up-to-date (for current 21-ish stable ff from hg) fix is here: ff_loose_plugin_keygrab.patch (more future-proof github link).
JS-click/key-jacking issue seem to require some JS event firewalling, and sometimes (e.g. JS games or some weird-design sites) can be useful.
So my solution was simply to bind JS-toggle key, which allows not only to disable all that crap, but also speed some "load-shit-as-you-go" or JS-BTC-mining (or so it feels) sites rapidly.
I have KeyConfig extension, which allows to bind random JS to a key, so:
var prefs = Components.classes['@mozilla.org/preferences-service;1']
    .getService(Components.interfaces.nsIPrefBranch),
  state = prefs.getBoolPref('javascript.enabled');
prefs.setBoolPref('javascript.enabled', !state);

That's the whole thing, bound to something like Ctrl+\ (the one above Enter here), makes a nice "Turbo and Get Off My JS" key. Fairly sure there are addons that allow to toggle prefs ("javascript.enabled" above) via keys without needing any code, but I have this one.

Damn glad there are open-source (and uglifyjs-like) browsers like that, hope proprietary google-ware won't take over the world in the nearest future.

Mentioned patches are available in (and integrated with-) the firefox-nightly exheres in my repo, forked off awesome sardemff7-pending firefox-scm.exheres-0 / mozilla-app.exlib work.

Jan 28, 2013

Headless Skype to IRC gateway part 3 - bitlbee + skyped

As per previous entry, with mock-desktop setup of Xvfb, fluxbox, x11vnc and skype in place, the only thing left is to use skype interfaces (e.g. dbus) to hook it up with existing IRC setup and maybe insulate skype process from the rest of the system.

Last bit is even easier than usual, since all the 32-bit libs skype needs are collected in one path, so no need to allow it to scan whatever system paths. Decided to go with the usual simplistic apparmor-way here - apparmor.profile, don't see much reason to be more paranoid here.

Also, libasound, used in skype gets quite noisy log-wise about not having the actual hardware on the system, but I felt bad about supressing the whole stderr stream from skype (to not miss the crash/hang info there), so had to look up a way to /dev/null alsa-lib output.
General way seem to be having "null" module as "default" sink
pcm.!default {
  type null
}
ctl.!default {
  type null
}

(libasound can be pointed to a local config by ALSA_CONFIG_PATH env var)

That "null" module is actually a dynamically-loaded .so, but alsa prints just a single line about it being missing instead of an endless stream of complaints for missing hw, so the thing works, by accident.

Luckily, bitlbee has support for skype, thanks to vmiklos, with sane way to run bitlbee and skype setup on different hosts (as it actually is in my case) through "skyped" daemon talking to skype and bitlbee connecting to its tcp (tls-wrapped) socket.

Using skyped shipped with bitlbee (which is a bit newer than on bitlbee-skype github) somewhat worked, with no ability to reconnect to it (hangs after handling first connection), ~1/4 chance of connection from bitlbee failing, it's persistence in starting skype (even though it's completely unnecessary in my case - systemd can do it way better) and such.

It's fairly simple python script though, based on somewhat unconventional Skype4Py module, so was able to fix most annoying of these issues (code can be found in the skype-space repo).
Will try to get these merged into bitlbee as I'm not the only one having these issues, apparently (e.g. #966), but so many things seem to be broken in that code (esp. wrt socket-handling), I think some major rewrite is in order, but that might be much harder to push upstream.
One interesting quirk of skyped is that it uses TLS to protect connections (allowing full control of the skype account) between bitlbee module and the daemon, but it doesn't bothers with any authorization, making that traffic as secure as plaintext to anyone in-between.
Quite a bit worse is that it's documented that the traffic is "encrypted", which might get one to think "ok, so running that thing on vps I don't need ssh-tunnel wrapping", which is kinda sad.
Add to that the added complexity it brings, segfaults in the plugin (crashing bitlbee), unhandled errors like
Traceback (most recent call last):
  File "./skyped", line 209, in listener
    ssl_version=ssl.PROTOCOL_TLSv1)
  File "/usr/lib64/python2.7/ssl.py", line 381, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib64/python2.7/ssl.py", line 143, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 104] Connection reset by peer
...and it seem to be classic "doing it wrong" pattern.
Not that much of an issue in my case, but I guess there should at least be a big red warning for that.

Functionality-wise, pretty much all I needed is there - one-to-one chats, bookmarked channels (as irc channels!), file transfers (just set "accept all" for these) with notifications about them, user info, contact list (add/remove with allow/deny queries),

But the most important thing by far is that it works at all, saving me plenty of work to code whatever skype-control interface over irc, though I'm very tempted to rewrite "skyped" component, which is still a lot easier with bitlbee plugin on the other end.

Units and configs for the whole final setup can be found on github.

Jan 27, 2013

Headless Skype to IRC gateway part 2 - SkypeKit

Thought it should be (hardly) worth a notice that Skype (well, Microsoft now) offers a thing called SkypeKit.

To get it, one have to jump through a dozen of hoops, including long registration form, $5 "tax for your interest in out platform" and wait for indefinite amount of time for invite to the privileged circle of skype hackers.

Here's part of the blurb one have to agree to:

By registering with Skype Developer, you will have access to confidential information and documentation relating to the SkypeKit program that has not been publicly released ("Confidential Information") and you agree not to disclose, publish or disseminate the Confidential Information to any third party (including by posting on any developer forum); and to take reasonable measures to prevent the unauthorised use, disclosure, publication or dissemination of the Confidential Information.
Just WOW!
What a collossal douchebags people who came up with that must be.
I can't even begin to imagine sheer scale of idiocy that's going on in the organization to come up with such things.
And just as these things often go, here's the Pirate Bay link.
But I think I'd rather respect the right of whoever came up with that "hey, let's screw developers" policy, if only to avoid (admittedly remote) chance of creating something useful for a platform like that.

Jan 27, 2013

Skype to IRC gateway on a headless server as a systemd user session daemon

Skype is a necessary evil for me, but just for text messages, and it's quite annoying that its closed nature makes it hard to integrate it into existing IM/chat infrastructure (for which I use ERC + ZNC + bitlbee + ejabberd).

So, finally got around to pushing the thing off my laptop machine.

Despite being quite a black-box product, skype has a surprisingly useful API, allowing to do pretty much everything desktop client allows to, which is accessible via several means, one of them being dbus. Wish that API was accessible on one of their servers, but no such luck, I guess. Third-party proxies are actually available, but I don't think +1 point of trust/failure is necessary here.

Since they stopped providing amd64 binaries (and still no word of sources, of course) and all the local non-laptop machines around are amd64, additional quirk is either enabling multibuild and pulling it everything up to and including Qt and WebKit to the poor headless server or just put what skype needs there built on 32-bit machine.

Not too enthusiastic about building lots of desktop crap on atom-based mini-ITX server, decided to go with the latter option, and dependency libs turn out to be fairly lean:

% ldd /opt/skype/skype | awk '$3 {print $3}' |
        xargs ls -lH | awk '{sum+=$5} END {print sum}'
49533468

Naturally, 50M is not an issue for a reasonably modern amounts of RAM.

But, of course, skype runs on X server, so Xvfb (cousing of X, drawing to memory instead of some GPU hardware):

# cave resolve -zx1 xorg-server x11vnc fluxbox

Concrete example above is for source-based exherbo, I think binary distros like debian might package Xvfb binary separately from X (in some "xvfb" package). fluxbox is there to have easy time interacting with skype-created windows.

Note - no heavy DE stuff is needed here, and as I was installing it on a machine hosting cairo-based graphite web frontend, barely any packages are actually needed here, aside from a bunch of X protocol headers and the things specified.

So, to run Xvfb with VNC I've found a bunch of simple shell scripts, which were guaranteed to not provide a lot of things a normal desktop session does, miss stray pids, create multiple instances for all the things involved, loose output, no xdg session, etc.

In general (and incomplete) case, something like this should be done:

export DISPLAY=:0
Xvfb $DISPLAY -screen 0 800x600x16 &
x11vnc -display $DISPLAY -nopw -listen localhost &
fluxbox &
skype &
wait

So, to not reinvent the same square wheel, decided to go with trusty systemd --user, as it's used as a system init anyway.

skype-desktop.service:

[Service]
User=skype
PAMName=login
Type=notify
Environment=DISPLAY=:1
Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=%h/tmp/session_bus_socket
ExecStart=/usr/lib/systemd/systemd --user

[Install]
WantedBy=multi-user.target

Aside from a few quirks like hardcoding dbus socket, that already fixes a lot of XDG_* related env-stuff, proper start/stop cleanup (no process escapes from that cgroup), monitoring (state transitions for services are echoed on irc to me), logging (all output will end up in queryable journal and syslog) and such, so highly recommend not going the "simple" bash-way here.

Complimentary session units generally look like this (Xvfb.service):

[Service]
SyslogIdentifier=%p
ExecStart=/usr/bin/Xvfb $DISPLAY -screen 0 800x600x16

And with systemct start skype-desktop, nice (but depressingly empty) fluxbox desktop is now accessible over ssh+vnc (don't trust vnc enough to run it on non-localhost, plus should be rarely needed anyway):

% ssh -L 5900:localhost:5900 user@host &
% vncclient localhost

Getting skype to run on the target host was a bit more difficult than I've expected though - local x86 machine has -march=native in CFLAGS and core-i3 cpu, so just copying binaries/libs resulted in a predictable:

[271817.608818] traps: ld-linux.so.2[7169]
        trap invalid opcode ip:f77dad60 sp:ffb91860 error:0 in ld-linux.so.2[f77c6000+20000]

Fortunately, there're always generic-arch binary distros, so had to spin up a qemu with ubuntu livecd iso, install skype there and run the same collect-all-the-deps script.

Basically, what's needed for skype to run is it's own data/media files ("/opt/skype", "/usr/share/skype"), binary ("/usr/lib/skype", "/opt/skype/skype") and all the so's it's linked against.

There's no need to put them all in "/usr/lib" or such, aside from "ld-linux.so.2", path to which ("/lib/ld-linux.so.2") is hard-compiled into skype binary (and is honored by linker).
Should be possible to change it there, but iirc skype checked it's binary checksum as well, so might be a bit more complicated than just "sed".
LD_LIBRARY_PATH=. ./skype --resources=. is the recipie for dealing with the rest.
Skype started $DEITY-knows-where over VNC

Yay!

So, to the API-to-IRC scripts then... probably in the next entry, as I get to these myself. Also following might be revised apparmor profile for such setup and maybe a script to isolate the whole thing even further into namespaces (which is interesting thing to try, but not sure how it might be useful yet with LSM already in place).

All the interesting stuff for the whole endeavor can be found in the ad-hoc repo I've created for it: https://github.com/mk-fg/skype-space

← Previous Next → Page 2 of 4
Member of The Internet Defense League