Aug 14, 2018

rst-based org-mode-like calendar generator for conky

Basically converting some free-form .rst (as in ReST or reStructuredText) like this:

Minor chores
------------

Stuff no one ever remembers doing.

- Pick up groceries

  :ts: 2018-08-27 12:00

  Running low on salt, don't forget to grab some.

- Do the laundry

  :ts: every 2w interval

  That pile in the corner across the room.


Total Annihilation
------------------

:ts-start: 2018-09-04 21:00
:ts-end: 2018-09-20 21:00

For behold, the LORD will come in fire And His chariots like the whirlwind,
To render His anger with fury, And His rebuke with flames of fire. ... blah blah

Into this:

conky calendar from rst

Though usually with a bit less theatrical evil and more useful events you don't want to use precious head-space to always keep track of.

Emacs org-mode is the most praised and common solution for this (well, in non-web/gui/mobile world, that is), but never bothered to learn its special syntax, and don't really want such basic thing to be emacs/elisp-specific (like, run emacs to parse/generate stuff!).

rst, on the other hand, is an old friend (ever since I found markdown to be too limiting and not blocky enough), supports embedding structured data in there (really like how github highlights it btw - check out cal.rst example here), and super-easy to work with in anything, not just emacs, and can be parsed by a whole bunch of stuff.

Project on github: mk-fg/rst-icalendar-event-tracker

Should be zero-setup, light on dependencies and very easy to use.

Has simple iCalendar export for anything else, but with conky in particular, its generated snippet can be included via $(catp /path/to/output} directive, and be configured wrt columns/offsets/colors/etc both on script-level via -o/--conky-params option and per-event via :conky: <opts> rst tags.

Calendar like that is not only useful if you wear suits, but also to check on all these cool once-per-X podcasts, mark future releases of some interesting games or movies, track all the monthly bills and chores that are easy to forget about.

And it feels great to not be afraid to forget or miss all this stuff anymore.
In fact, having an excuse to structure and write it down helps a ton already.

But beyond that, feel like using transient and passing reminders can be just as good for tracking updates on some RSS feeds or URLs, so that you can notice update and check it out if you want to, maybe even mark as a todo-entry somewhere, but it won't hang over the head by default (and forever) in an inbox, feed reader or a large un-*something* number in bold someplace else.

So plan to add url-grepping and rss-checking as well, and have updates on arbitrary pages or feeds create calendar events there, which should at least be useful for aforementioned media periodics.

In fact, rst-configurable feed-checker (not "reader"!) might be a much better idea even for text-based content than some crappy web-based django mostrosity like the one I used before.

Apr 16, 2018

Emacs EMMS backend for long-running mpv processes

EMMS is the best music player out there (at least if you use emacs), as it allows full power and convenience of proper $EDITOR for music playlists and such.

All mpv backends for it that I'm aware of were restarting player binary for every track though, which is simple, good compatibility-wise, but also suboptimal in many ways.

For one thing, stuff like audio visualization is pita if it's in a constantly created/destroyed transient window, it adds significant gaps between played tracks (gapless+crossfade? forget it!), and - more due to why player starts/exit (know when playback ends) - feedback/control over it are very limited, since clearly no good APIs are used there, if wrapper relies on process exit as "playback ended" event.

Rewritten emms-player-mpv.el (also in "mpv-json-ipc" branch of emms git atm) fixes all that.

What's curious is that I didn't see almost all of these interesting use-cases, which using the tool in the sane way allows for, and only wrote new wrapper to get nice "playback position" feedback and out of petty pedantry over how lazy simple implementation seem to be.

Having separate persistent player window allows OSD config or lua to display any kind of metadata or infographics (with full power of lua + mpv + ffmpeg) about current tracks or playlist stuff there (esp. for online streams), enables subs/lyrics display, and getting stream of metadata update events from mpv allows to update any "now playing" meta stuff in emacs/emms too.

What seemed like a petty and almost pointless project to have fun with lisp, turned out to be actually useful, which seem to often be the case once you take a deep-dive into things, and not just blindly assume stuff about them (fire hot, water wet, etc).

Hopefully might get merged upstream after EMMS 5.0 and get a few more features and interesting uses like that along the way.

(though I'd suggest not waiting and just adding anything that comes to mind in ~/.emacs via emms-mpv-event-connect-hook, emms-mpv-event-functions and emms-mpv-ipc-req-send - should be really easy now)

Apr 12, 2018

mpv audio visualization

Didn't know mpv could do that until dropping into raw mpv for music playback yesterday, while adding its json api support into emms (emacs music player).

One option in mpv that I found essential over time - especially as playback from network sources via youtube-dl (youtube, twitch and such) became more common - is --force-window=immediate (via config), so that you can just run "mpv URL" in whatever console and don't have to wait until video buffers enough for mpv window to pop-up.

This saves a few-to-dozen seconds of annoyance as otherwise you can't do anything during ytdl init and buffering phase is done, as that's when window will pop-up randomly and interrupt whatever you're doing, plus maybe get affected by stuff being typed at the moment (and close, skip, seek or get all messed-up otherwise).

It's easy to disable this unnecessary window for audio-only files via lua, but other option that came to mind when looking at that black square is to send it to aux display with some nice visualization running.

Which is not really an mpv feature, but one of the many things that ffmpeg can render with its filters, enabled via --lavfi-complex audio/video filtering option.

E.g. mpv --lavfi-complex="[aid1]asplit[ao][a]; [a]showcqt[vo]" file.mp3 will process a copy of --aid=1 audio stream (one copy goes straight to "ao" - audio output) via ffmpeg showcqt filter and send resulting visualization to "vo" (video output).

As ffmpeg is designed to allow many complex multi-layered processing pipelines, extending on that simple example can produce really fancy stuff, like any blend of images, text and procedurally-generated video streams.

Some nice examples of those can be found at ffmpeg wiki FancyFilteringExamples page.

It's much easier to build, control and tweak that stuff from lua though, e.g. to only enable such vis if there is a blank forced window without a video stream, and to split those long pipelines into more sensible chunks of parameters, for example:

local filter_bg = lavfi_filter_string{
  'firequalizer', {
    gain = "'20/log(10)*log(1.4884e8"
      .."* f/(f*f + 424.36)"
      .."* f/(f*f + 1.4884e8)"
      .."* f/sqrt(f*f + 25122.25) )'",
    accuracy = 1000,
    zero_phase = 'on' },
  'showcqt', {
    fps = 30,
    size = '960x768',
    count = 2,
    bar_g = 2,
    sono_g = 4,
    bar_v = 9,
    sono_v = 17,
    font = "'Luxi Sans,Liberation Sans,Sans|bold'",
    fontcolor = "'st(0, (midi(f)-53.5)/12);"
      .."st(1, 0.5 - 0.5 * cos(PI*ld(0))); r(1-ld(1)) + b(ld(1))'",
    tc = '0.33',
    tlength = "'st(0,0.17);"
      .."384*tc/(384/ld(0)+tc*f/(1-ld(0)))"
      .." + 384*tc/(tc*f/ld(0)+384/(1-ld(0)))'" } }
local filter_fg = lavfi_filter_string{ 'avectorscope',
  { mode='lissajous_xy', size='960x200',
    rate=30, scale='cbrt', draw='dot', zoom=1.5 } }

local overlay = lavfi_filter_string{'overlay', {format='yuv420'}}
local lavfi =
  '[aid1] asplit=3 [ao][a1][a2];'
  ..'[a1]'..filter_bg..'[v1];'
  ..'[a2]'..filter_fg..'[v2];'
  ..'[v1][v2]'..overlay..'[vo]'

mp.set_property('options/lavfi-complex', lavfi)

Much easier than writing something like this down into one line.

("lavfi_filter_string" there concatenates all passed options with comma/colon separators, as per ffmpeg syntax)

Complete lua script that I ended-up writing for this: fg.lavfi-audio-vis.lua

With some grand space-ambient electronic score, showcqt waterfall can move in super-trippy ways, very much representative of the glacial underlying audio rythms:

mpv ffmpeg visualization snapshot

(track in question is "Primordial Star Clouds" [45] from EVE Online soundtrack)

Script won't kick-in with --vo=null, --force-window not enabled, or if "vo-configured" won't be set by mpv for whatever other reason (e.g. some video output error), otherwise will be there with more pretty colors to brighten your day :)

Apr 10, 2018

Linux X desktop "clipboard" keys via exclip tool

It's been a mystery to me for a while how X terminal emulators (from xterm to Terminology) manage to copy long bits of strings spanning multiple lines without actually splitting them with \n chars, given that there's always something like "screen" or "tmux" or even "mosh" running in there.

All these use ncurses which shouldn't output "long lines of text" but rather knows width/height of a terminal and flips specific characters there when last output differs from its internal "how it should be" state/buffer.

Regardless of how this works, terminals definitely get confused sometimes, making copy-paste of long paths and commands from them into a minefield, where you never know if it'll insert full path/command or just run random parts of it instead by emitting newlines here and there.

Easy fix: bind a key combo in a WM to always "copy stuff as a single line".
Bonus points - also strip spaces from start/end, demanding no select-precision.
Even better - have it expose result as both primary and clipboard, to paste anywhere.

For a while used a trivial bash script for that, which did "xclip -in" from primary selection, some string-mangling in bash and two "xclip -out" to put result back into primary and clipboard.

It's a surprisingly difficult and suboptimal task for bash though, as - to my knowledge - you can't even replace \n chars in it without running something like "tr" or "sed".

And running xclip itself a few times is bad enough, but with a few extra binaries and under CPU load, such "clipboard keys" become unreliable due to lag from that script.

Hence finally got fed up by it and rewritten whole thing in C as a small and fast 300-liner exclip tool, largely based on xclip code.

Build like this: gcc -O2 -lX11 -lXmu exclip.c -o exclip && strip exclip

Found something like it bound to a key (e.g. Win+V for verbatim copy, and variants like Win+Shift+V for stripping spaces/newlines) to be super-useful when using terminals and text-based apps, apps that mix primary/clipboard selections, etc - all without needing to touch the mouse.

Tool is still non-trivial due to how selections and interaction with X work - code has to be event-based, negotiate content type that it wants to get, can have large buffers sent in incremental events, and then have to hold these (in a forked subprocess) and negotiate sending to other apps - i.e. not just stuff N bytes from buffer somewhere server-side and exit ("cut buffers" can work like that in X, but limited and never used).

Looking at all these was a really fun dive into how such deceptively-simple (but ancient and not in fact simple at all) things like "clipboard" work.

E.g. consider how one'd hold/send/expose stuff from huge GIMP image selection and paste it into an entirely different app (xclip -out -t TARGETS can give a hint), especially with X11 and its network transparency.

Though then again, maybe humble string manipulation in C is just as fascinating, given all the pointer juggling and tricks one'd usually have to do for that.

May 15, 2017

Emacs slow font rendering fail

Mostly use unorthodox variable-width font for coding, but do need monospace sometimes, e.g. for jagged YAML files or .rst.

Had weird issue with my emacs for a while, where switching to monospace font will slow window/frame rendering significantly, to a noticeable degree, having stuff blink and lag, making e.g. holding key to move cursor impossible, etc.

Usual profiling showed that it's an actual rendering via C code, so kinda hoped that it'd go away in one of minor releases, but nope - turned out to be the dumbest thing in ~/.emacs:

(set-face-font 'fixed-pitch "DejaVu Sans Mono-7.5")

That one line is what slows stuff down to a crawl in monospace ("fixed-pitch") configuration, just due to non-integer font size, apparently.

Probably not emacs' fault either, just xft or some other lower-level rendering lib, and a surprising little quirk that can affect high-level app experience a lot.

Changing font size there to 8 or 9 gets rid of the issue. Oh well...

Feb 13, 2017

Xorg input driver - the easy way, via evdev and uinput

Got to reading short stories in Column Reader from laptop screen before sleep recently, and for an extra-lazy points, don't want to drag my hand to keyboard to flip pages (or columns, as the case might be).

Easy fix - get any input device and bind stuff there to keys you'd normally use.
As it happens, had Xbox 360 controller around for that.

Hard part is figuring out how to properly do it all in Xorg - need to build xf86-input-joystick first (somehow not in Arch core), then figure out how to make it act like a dumb event source, not some mouse emulator, and then stuff like xev and xbindkeys will probably help.

This is way more complicated than it needs to be, and gets even more so when you factor-in all the Xorg driver quirks, xev's somewhat cryptic nature (modifier maps, keysyms, etc), fact that xbindkeys can't actually do "press key" actions (have to use stuff like xdotool for that), etc.

All the while reading these events from linux itself is as trivial as evtest /dev/input/event11 (or for event in dev.read_loop(): ...) and sending them back is just ui.write(e.EV_KEY, e.BTN_RIGHT, 1) via uinput device.

Hence whole binding thing can be done by a tiny python loop that'd read events from whatever specified evdev and write corresponding (desired) keys to uinput.

So instead of +1 pre-naptime story, hacked together a script to do just that - evdev-to-xev (python3/asyncio) - which reads mappings from simple YAML and runs the loop.

For example, to bind right joystick's (on the same XBox 360 controller) extreme positions to cursor keys, plus triggers, d-pad and bumper buttons there:

map:

  ## Right stick
  # Extreme positions are ~32_768
  ABS_RX <-30_000: left
  ABS_RX >30_000: right
  ABS_RY <-30_000: up
  ABS_RY >30_000: down

  ## Triggers
  # 0 (idle) to 255 (fully pressed)
  ABS_Z >200: left
  ABS_RZ >200: right

  ## D-pad
  ABS_HAT0Y -1: leftctrl leftshift equal
  ABS_HAT0Y 1: leftctrl minus
  ABS_HAT0X -1: pageup
  ABS_HAT0X 1: pagedown

  ## Bumpers
  BTN_TL 1: [h,e,l,l,o,space,w,o,r,l,d,enter]
  BTN_TR 1: right

timings:
  hold: 0.02
  delay: 0.02
  repeat: 0.5
Run with e.g.: evdev-to-xev -c xbox-scroller.yaml /dev/input/event11
(see also less /proc/bus/input/devices and evtest /dev/input/event11).
Running the thing with no config will print example one with comments/descriptions.

Given how all iterations of X had to work with whatever input they had at the time, plus not just on linux, even when evdev was around, hard to blame it for having a bit of complexity on top of way simplier input layer underneath.

In linux, aforementioned Xbox 360 gamepad is supported by "xpad" module (so that you'd get evdev node for it), and /dev/uinput for simulating arbitrary evdev stuff is "uinput" module.

Script itself needs python3 and python-evdev, plus evtest can be useful.
No need for any extra Xorg drivers beyond standard evdev.

Most similar tool to such script seem to be actkbd, though afaict, one'd still need to run xdotool from it to simulate input :O=

Github link: evdev-to-xev script (in the usual mk-fg/fgtk scrap-heap)

May 15, 2016

Debounce bogus repeated mouse clicks in Xorg with xbindkeys

My current Razer E-Blue mouse had this issue since I've got it - Mouse-2 / BTN_MIDDLE / middle-click (useful mostly as "open new tab" in browsers and "paste" in X) sometimes produces two click events (in rapid succession) instead of one.

It was more rare before, but lately it feels like it's harder to make it click once than twice.

Seem to be either hardware problem with debouncing circuitry or logic in the controller, or maybe a button itself not mashing switch contacts against each other hard enough... or soft enough (i.e. non-elastic), actually, given that they shouldn't "bounce" against each other.

Since there's no need to double-click that wheel-button ever, it looks rather easy to debounce the click on Xorg input level, by ignoring repeated button up/down events after producing the first full "click".

Easiest solution of that kind that I've found was to use guile (scheme) script with xbindkeys tool to keep that click-state data and perform clicks selectively, using xdotool:

(define razer-delay-min 0.2)
(define razer-wait-max 0.5)
(define razer-ts-start #f)
(define razer-ts-done #f)
(define razer-debug #f)

(define (mono-time)
  "Return monotonic timestamp in seconds as real."
  (+ 0.0 (/ (get-internal-real-time) internal-time-units-per-second)))

(xbindkey-function '("b:8") (lambda ()
  (let ((ts (mono-time)))
    (when
      ;; Enforce min ts diff between "done" and "start" of the next one
      (or (not razer-ts-done) (>= (- ts razer-ts-done) razer-delay-min))
      (set! razer-ts-start ts)))))

(xbindkey-function '(Release "b:8") (lambda ()
  (let ((ts (mono-time)))
    (when razer-debug
      (format #t "razer: ~a/~a delay=~a[~a] wait=~a[~a]\n"
        razer-ts-start razer-ts-done
        (and razer-ts-done (- ts razer-ts-done)) razer-delay-min
        (and razer-ts-start (- ts razer-ts-start)) razer-wait-max))
    (when
      (and
        ;; Enforce min ts diff between previous "done" and this one
        (or (not razer-ts-done) (>= (- ts razer-ts-done) razer-delay-min))
        ;; Enforce max "click" wait time
        (and razer-ts-start (<= (- ts razer-ts-start) razer-wait-max)))
      (set! razer-ts-done ts)
      (when razer-debug (format #t "razer: --- click!\n"))
      (run-command "xdotool click 2")))))

Note that xbindkeys actually grabs "b:8" here, which is a "mouse button 8", as if it was "b:2", then "xdotool click 2" command will recurse into same code, so wheel-clicker should be bound to button 8 in X for that to work.

Rebinding buttons in X is trivial to do on-the-fly, using standard "xinput" tool - e.g. xinput set-button-map "My Mouse" 1 8 3 (xinitrc.input script can be used as an extended example).

Running "xdotool" to do actual clicks at the end seem a bit wasteful, as xbindkeys already hooks into similar functionality, but unfortunately there's no "send input event" calls exported to guile scripts (as of 1.8.6, at least).

Still, works well enough as it is, fixing that rather annoying issue.

[xbindkeysrc.scm on github]

May 19, 2015

twitch.tv VoDs (video-on-demand) downloading issues and fixes

Quite often recently VoDs on twitch for me are just unplayable through the flash player - no idea what happens at the backend there, but it buffers endlessly at any quality level and that's it.

I also need to skip to some arbitrary part in the 7-hour stream (last wcs sc2 ro32), as I've watched half of it live, which turns out to complicate things a bit.

So the solution is to download the thing, which goes something like this:

  • It's just a video, right? Let's grab whatever stream flash is playing (with e.g. FlashGot FF addon).

    Doesn't work easily, since video is heavily chunked.
    It used to be 30-min flv chunks, which are kinda ok, but these days it's forced 4s chunks - apparently backend doesn't even allow downloading more than 4s per request.
  • Fine, youtube-dl it is.

    Nope. Doesn't allow seeking to time in stream.
    There's an open "Download range" issue for that.

    livestreamer wrapper around the thing doesn't allow it either.

  • Try to use ?t=3h30m URL parameter - doesn't work, sadly.

  • mpv supports youtube-dl and seek, so use that.

    Kinda works, but only for super-short seeks.
    Seeking beyond e.g. 1 hour takes AGES, and every seek after that (even skipping few seconds ahead) takes longer and longer.
  • youtube-dl --get-url gets m3u8 playlist link, use ffmpeg -ss <pos> with it.

    Apparently works exactly same as mpv above - takes like 20-30min to seek to 3:30:00 (3.5 hour offset).

    Dunno if it downloads and checks every chunk in the playlist for length sequentially... sounds dumb, but no clue why it's that slow otherwise, apparently just not good with these playlists.

  • Grab the m3u8 playlist, change all relative links there into full urls, remove bunch of them from the start to emulate seek, play that with ffmpeg | mpv.

    Works at first, but gets totally stuck a few seconds/minutes into the video, with ffmpeg doing bitrates of ~10 KiB/s.

    youtube-dl apparently gets stuck in a similar fashion, as it does the same ffmpeg-on-a-playlist (but without changing it) trick.

  • Fine! Just download all the damn links with curl.

    grep '^http:' pls.m3u8 | xargs -n50 curl -s | pv -rb -i5 > video.mp4

    Makes it painfully obvious why flash player and ffmpeg/youtube-dl get stuck - eventually curl stumbles upon a chunk that downloads at a few KiB/s.

    This "stumbling chunk" appears to be a random one, unrelated to local bandwidth limitations, and just re-trying it fixes the issue.

  • Assemble a list of links and use some more advanced downloader that can do parallel downloads, plus detect and retry super-low speeds.

    Naturally, it's aria2, but with all the parallelism it appears to be impossible to guess which output file will be which with just a cli.

    Mostly due to links having same url-path, e.g. index-0000000014-O7tq.ts?start_offset=955228&end_offset=2822819 with different offsets (pity that backend doesn't seem to allow grabbing range of that *.ts file of more than 4s) - aria2 just does file.ts, file.ts.1, file.ts.2, etc - which are not in playlist-order due to all the parallel stuff.

  • Finally, as acceptance dawns, go and write your own youtube-dl/aria2 wrapper to properly seek necessary offset (according to playlist tags) and download/resume files from there, in a parallel yet ordered and controlled fashion.

    This is done by using --on-download-complete hook with passing ordered "gid" numbers for each chunk url, which are then passed to the hook along with the resulting path (and hook renames file to prefix + sequence number).

Ended up with the chunk of the stream I wanted, locally (online playback lag never goes away!), downloaded super-fast and seekable.

Resulting script is twitch_vod_fetch (script source link).

Update 2017-06-01: rewritten it using python3/asyncio since then, so stuff related to specific implementation details here is only relevant for old py2 version (can be pulled from git history, if necessary).


aria2c magic bits in the script:

aria2c = subprocess.Popen([
  'aria2c',

  '--stop-with-process={}'.format(os.getpid()),
  '--enable-rpc=true',
  '--rpc-listen-port={}'.format(port),
  '--rpc-secret={}'.format(key),

  '--no-netrc', '--no-proxy',
  '--max-concurrent-downloads=5',
  '--max-connection-per-server=5',
  '--max-file-not-found=5',
  '--max-tries=8',
  '--timeout=15',
  '--connect-timeout=10',
  '--lowest-speed-limit=100K',
  '--user-agent={}'.format(ua),

  '--on-download-complete={}'.format(hook),
], close_fds=True)

Didn't bother adding extra options for tweaking these via cli, but might be a good idea to adjust timeouts and limits for a particular use-case (see also the massive "man aria2c").

Seeking in playlist is easy, as it's essentially a VoD playlist, and every ~4s chunk is preceded by e.g. #EXTINF:3.240, tag, with its exact length, so script just skips these as necessary to satisfy --start-pos / --length parameters.

Queueing all downloads, each with its own particular gid, is done via JSON-RPC, as it seem to be impossible to:

  • Specify both link and gid in the --input-file for aria2c.
  • Pass an actual download URL or any sequential number to --on-download-complete hook (except for gid).

So each gid is just generated as "000001", "000002", etc, and hook script is a one-liner "mv" command.


Since all stuff in the script is kinda lenghty time-wise - e.g. youtube-dl --get-url takes a while, then the actual downloads, then concatenation, ... - it's designed to be Ctrl+C'able at any point.

Every step just generates a state-file like "my_output_prefix.m3u8", and next one goes on from there.
Restaring the script doesn't repeat these, and these files can be freely mangled or removed to force re-doing the step (or to adjust behavior in whatever way).
Example of useful restart might be removing *.m3u8.url and *.m3u8 files if twitch starts giving 404's due to expired links in there.
Won't force re-downloading any chunks, will only grab still-missing ones and assemble the resulting file.

End-result is one my_output_prefix.mp4 file with specified video chunk (or full video, if not specified), plus all the intermediate litter (to be able to restart the process from any point).


One issue I've spotted with the initial version:

05/19 22:38:28 [ERROR] CUID#77 - Download aborted. URI=...
Exception: [AbstractCommand.cc:398] errorCode=1 URI=...
  -> [RequestGroup.cc:714] errorCode=1 Download aborted.
  -> [DefaultBtProgressInfoFile.cc:277]
    errorCode=1 total length mismatch. expected: 1924180, actual: 1789572
05/19 22:38:28 [NOTICE] Download GID#0035090000000000 not complete: ...

Seem to be a few of these mismatches (like 5 out of 10k chunks), which don't get retried, as aria2 doesn't seem to consider these to be a transient errors (which is probably fair).

Probably a twitch bug, as it clearly breaks http there, and browsers shouldn't accept such responses either.

Can be fixed by one more hook, I guess - either --on-download-error (to make script retry url with that gid), or the one using websocket and getting json notification there.

In any case, just running same command again to download a few of these still-missing chunks and finish the process works around the issue.

Update 2015-05-22: Issue clearly persists for vods from different chans, so fixed it via simple "retry all failed chunks a few times" loop at the end.

Update 2015-05-23: Apparently it's due to aria2 reusing same files for different urls and trying to resume downloads, fixed by passing --out for each download queued over api.


[script source link]

Apr 11, 2015

Skype setup on amd64 without multilib/multiarch/chroot

Did a kinda-overdue migration of a desktop machine to amd64 a few days ago.
Exherbo has multiarch there, but I didn't see much point in keeping (and maintaining in various ways) a full-blown set of 32-bit libs just for Skype, which I found that I still need occasionally.

Solution I've used before (documented in the past entry) with just grabbing 32-bit Skype binary and full set of libs it needs from whatever distro still works and applies here, not-so-surprisingly.

What I ended up doing is:

  • Grab the latest Fedora "32-bit workstation" iso (Fedora-Live-Workstation-i686-21-5.iso).

  • Install/run it on a virtual machine (plain qemu-kvm).

  • Download "Dynamic" Skype version (distro-independent tar.gz with files) from Skype site to/on a VM, "tar -xf" it.

  • ldd skype-4.3.0.37/skype | grep 'not found' to see which dependency-libs are missing.

  • Install missing libs - yum install qtwebkit libXScrnSaver

  • scp build_skype_env.bash (from skype-space repo that I have from old days of using skype + bitlbee) to vm, run it on a skype-dir - e.g. ./build_skype_env.bash skype-4.3.0.37.

    Should finish successfully and produce "skype_env" dir in the current path.

  • Copy that "skype_env" dir with all the libs back to pure-amd64 system.

  • Since skype binary has "/lib/ld-linux.so.2" as a hardcoded interpreter (as it should be), and pure-amd64 system shouldn't have one (not to mention missing multiarch prefix) - patch it in the binary with patchelf:

    patchelf --set-interpreter ./ld-linux.so.2 skype
    
  • Run it (from that env dir with all the libs):

    LD_LIBRARY_PATH=. ./skype --resources=.
    

    Should "just work" \o/

One big caveat is that I don't care about any features there except for simple text messaging, which is probably not how most people use Skype, so didn't test if e.g. audio would work there.
Don't think sound should be a problem though, especially since iirc modern skype could use pulseaudio (or even using it by default?).

Given that skype itself a huge opaque binary, I do have AppArmor profile for the thing (uses "~/.Skype/env/" dir for bin/libs) - home.skype.

Mar 25, 2015

gnuplot for live "last 30 seconds" sliding window of "free" (memory) data

Was looking at a weird what-looks-like-a-memleak issue somewhere in the system on changing desktop background (somewhat surprisingly complex operation btw) and wanted to get a nice graph of "last 30s of free -m output", with some labels and easy access to data.

A simple enough task for gnuplot, but resulting in a somewhat complicated solution, as neither "free" nor gnuplot are perfect tools for the job.

First thing is that free -m -s1 doesn't actually give a machine-readable data, and I was too lazy to find something better (should've used sysstat and sar!) and thought "let's just parse that with awk":

free -m -s $interval |
  awk '
    BEGIN {
      exports="total used free shared available"
      agg="free_plot.datz"
      dst="free_plot.dat"}
    $1=="total" {
      for (n=1;n<=NF;n++)
        if (index(exports,$n)) headers[n+1]=$n }
    $1=="Mem:" {
      first=1
      printf "" >dst
      for (n in headers) {
        if (!first) {
          printf " " >>agg
          printf " " >>dst }
        printf "%d", $n >>agg
        printf "%s", headers[n] >>dst
        first=0 }
      printf "\n" >>agg
      printf "\n" >>dst
      fflush(agg)
      close(dst)
      system("tail -n '$points' " agg " >>" dst) }'

That might be more awk than one ever wants to see, but I imagine there'd be not too much space to wiggle around it, as gnuplot is also somewhat picky in its input (either that or you can write same scripts there).

I thought that visualizing "live" stream of data/measurements would be kinda typical task for any graphing/visualization solution, but meh, apparently not so much for gnuplot, as I haven't found better way to do it than "reread" command.

To be fair, that command seem to do what I want, just not in a much obvious way, seamlessly updating output in the same single window.

Next surprising quirk was "how to plot only last 30 points from big file", as it seem be all-or-nothing with gnuplot, and googling around, only found that people do it via the usual "tail" before the plotting.

Whatever, added that "tail" hack right to the awk script (as seen above), need column headers there anyway.

Then I also want nice labels - i.e.:

  • How much available memory was there at the start of the graph.
  • How much of it is at the end.
  • Min for that parameter on the graph.
  • Same, but max.
stats won't give first/last values apparently, unless I missed those in the PDF (only available format for up-to-date docs, le sigh), so one solution I came up with is to do a dry-run plot command with set terminal unknown and "grab first value" / "grab last value" functions to "plot".
Which is not really a huge deal, as it's just a preprocessed batch of 30 points, not a huge array of data.

Ok, so without further ado...

src='free_plot.dat'
y0=100; y1=2000;
set xrange [1:30]
set yrange [y0:y1]

# --------------------
set terminal unknown
stats src using 5 name 'y' nooutput

is_NaN(v) = v+0 != v
y_first=0
grab_first_y(y) = y_first = y_first!=0 && !is_NaN(y_first) ? y_first : y
grab_last_y(y) = y_last = y

plot src u (grab_first_y(grab_last_y($5)))
x_first=GPVAL_DATA_X_MIN
x_last=GPVAL_DATA_X_MAX

# --------------------
set label 1 sprintf('first: %d', y_first) at x_first,y_first left offset 5,-1
set label 2 sprintf('last: %d', y_last) at x_last,y_last right offset 0,1
set label 3 sprintf('min: %d', y_min) at 0,y0-(y1-y0)/15 left offset 5,0
set label 4 sprintf('max: %d', y_max) at 0,y0-(y1-y0)/15 left offset 5,1

# --------------------
set terminal x11 nopersist noraise enhanced
set xlabel 'n'
set ylabel 'megs'

set style line 1 lt 1 lw 1 pt 2 pi -1 ps 1.5
set pointintervalbox 2

plot\
  src u 5 w linespoints linestyle 1 t columnheader,\
  src u 1 w lines title columnheader,\
  src u 2 w lines title columnheader,\
  src u 3 w lines title columnheader,\
  src u 4 w lines title columnheader,\

# --------------------
pause 1
reread

Probably the most complex gnuplot script I composed to date.

Yeah, maybe I should've just googled around for an app that does same thing, though I like how this lore potentially gives ability to plot whatever other stuff in a similar fashion.

That, and I love all the weird stuff gnuplot can do.

For instance, xterm apparently has some weird "plotter" interface hardware terminals had in the past:

gnuplot and Xterm Tektronix 4014 Mode

And there's also the famous "dumb" terminal for pseudographics too.

Regular x11 output looks nice and clean enough though:

gnuplot x11 output

It updates smoothly, with line crawling left-to-right from the start and then neatly flowing through. There's a lot of styling one can do to make it prettier, but I think I've spent enough time on such a trivial thing.

Didn't really help much with debugging though. Oh well...

Full "free | awk | gnuplot" script is here on github.

Next → Page 1 of 4
Member of The Internet Defense League