Nov 18, 2022

AWK script to convert long integers to human-readable number format and back

Haven't found anything good for this on the internet before, and it's often useful in various shell scripts where metrics or tunable numbers can get rather long:

3400000000 -> 3_400M
500000 -> 500K
8123455009 -> 8_123_455_009

That is, to replace long stretches of zeroes with short Kilo/Mega/Giga/etc suffixes and separate the usual 3-digit groups by something (underscore being programming-language-friendly and hard to mistake for field separator), and also convert those back to pure integers from cli arguments and such.

There's numfmt in GNU coreutils, but that'd be missing on Alpine, typical network devices, other busybox Linux distros, *BSD, MacOS, etc, and it doesn't have "match/convert all numbers you want" mode anyway.

So alternative is using something that is available everywhere, like generic AWK, with a reasonably short scripts to implement that number-mangling logic.

  • Human-format all space-separated long numbers in stdin, like in example above:

    awk '{n=0; while (n++ <= NR) { m=0
      while (match($n,/^[0-9]+[0-9]{3}$/) && m < 5) {
        if (substr($n,k+1)=="000") { $n=substr($n,1,k); m++ }
        else while (match($n,/[0-9]{4}(_|$)/))
          $n = substr($n,1,RSTART) "_" substr($n,RSTART+1) }
      $n = $n substr("KMGTP", m, m ? 1 : 0) }; print}'
  • Find all human-formatted numbers in stdin and convert them back to long integers:

    awk '{while (n++ <= NR) {if (match($n,/^([0-9]+_)*[0-9]+[KMGTP]?$/)) {
      sub("_","",$n); if (m=index("KMGTP", substr($n,length($n),1))) {
        $n=substr($n,1,length($n)-1); while (m-- > 0) $n=$n "000" } }}; print}'

    I.e. reverse of the operation above.

Code is somewhat compressed for brevity within scripts where it's not the point. It should work with any existing AWK implementations afaik (gawk, nawk, busybox awk, etc), and not touch any fields that don't need such conversion (as filtered by the first regexp there).

Line-match pattern can be added at the start to limit conversion to lines with specific fields (e.g. match($1,/(Count|Threshold|Limit|Total):/) {...}), "n" bounds and $n regexps adjusted to filter-out some inline values.

Numbers here will use SI prefixes, not 2^10 binary-ish increments, like in IEC units (kibi-, mebi-, gibi-, etc), as is more useful in general, but substr() and "000"-extension can be replaced by /1024 (and extra -iB unit suffix) when working with byte values - AWK can do basic arithmetic just fine too.

Took me couple times misreading and mistyping overly long integers from/into scripts to realize that this is important enough and should be done better than counting zeroes with arrow keys like some tech-barbarian, even in simple bash scripts, and hopefully this hack might eventually pop-up in search for someone else coming to that conclusion as well.

Oct 21, 2022

Useful git hook - prepare-commit-msg with repo path, branch and last commits

I've been using this as a prepare-commit-msg hook everywhere for couple years now:

msg_file=$1 commit_src=$2 hash=$3
[[ -z "$msg_file" || "$GIT_EDITOR" = : ]] && exit 0
[[ -z "$commit_src" ]] || exit 0 # cherry-picks, merges, etc

echo >>"$msg_file" '#'
echo >>"$msg_file" "# Commit dir: ${GIT_PREFIX%/}"
echo >>"$msg_file" "#   Repo dir: $(realpath "$PWD")"
echo >>"$msg_file" '#'
git log -10 --format='# %s' >>"$msg_file"

Which saves a lot of effort of coming up with commit-messages, helps in monorepos/collections and to avoid whole bunch of mistakes.

Idea is that instead of just "what is to be comitted", comment below commit-msg in $EDITOR will now include something like this:

# On branch master
# Changes to be committed:
# modified:   PKGBUILD
# new file:
# Commit dir: waterfox
#   Repo dir: /home/myuser/archlinux-pkgbuilds
# waterfox: +9004.rebind_screenshot_key_to_ctrl_alt_s.patch
# waterfox: fix more build/install issues
# waterfox: +fix_build_with_newer_cbindgen.patch
# waterfox: update to G5.0.1
# +re2g-git
# waterfox: bump to G4.1.4
# +mount-idmapped-git
# telegram-tdlib-purple-*: makedepends=telegram-tdlib - static linking
# waterfox: update for G4.1.1
# +b2tag-git

It helps as a great sanity-check and reminder of the following things:

  • Which subdir within the repo you are working in, e.g. "waterfox" pkg above, so that it's easy to identify and/or use that as a part of commit message.

    With a lot of repositories I work with, there are multiple subdirs and components in there, not to mention collection-repos, and it's useful to have that in the commit msg - I always try to add them as a prefix, unless repo uses entirely different commit message style (and has one).

  • What is the repository directory that you're running "git commit" in.

    There can be a local dev repo, a submodule of it in a larger project, sshfs-mounted clone of it somewhere, more local clones for diff branches or different git-worktree dirs, etc.

    This easily tells that you're in the right place (or where you think you are), usually hard to miss by having repo under local simple dev dir, and not some weird submodule path or mountpoint in there.

  • List of last commits on this branch - incredibly useful for a ton of reasons.

    For one, it easily keeps commit-msgs consistent - you don't use different language, mood and capitalization in there by forgetting what is the style used in this particular repo, see any relevant issue/PR numbers, prefixes/suffixes.

    But also it immediately shows if you're on the wrong branch, making a duplicate commit by mistake, forgot to make commit/merge for something else important before this, undo some reset or other recent shenanigans - all at a glance.

    It was a usual practice for me to check git-log before every commit, and this completely eliminated the need for it.

Now when I don't see this info in the commit-msg comment, first thing to do is copy the hook script to whatever repo/host/place I'm working with, as it's almost as bad as not seeing which files you commit in there without it. Can highly recommend this tiny time-saver when working with any git repos from the command line.

Implementation has couple caveats, which I've added there over time:

[[ -z "$msg_file" || "$GIT_EDITOR" = : ]] && exit 0
[[ -z "$commit_src" ]] || exit 0 # cherry-picks, merges, etc

These lines are to skip running this hook for various non-interactive git operations, where anything you put into commit-msg will get appended to it verbatim, without skipping comment-lines, as it is done with interactive "git commit" ops.

Canonical version of the hook is in the usual mk-fg/fgtk dumping ground:

Which might get more caveats like above fixed in the future, should I bump into any, so might be better than current version in this post.

Oct 19, 2022

Make cursor stand-out more in Emacs by having it blink through different colors

When playing some game (Starsector, probably) on the primary display recently, having aux emacs frame (an extra X11 window) on the second one with some ERC chat in there (all chats translate to irc), I've had a minor bug somewhere and noticed that cursor in that frame/window wasn't of the usual translucent-green-foreground-text color (on a dark bg), but rather stayed simply #fff-white (likely because of some screwup in my theme-application func within that frame).

And an interesting observation is that it's actually really good when cursor is of a different color from both foreground-text and the background colors, because then it easily stands out against both, and doesn't get lost in a wall of text either - neat!

Quickly googling around for a hack to make this fluke permanent, stumbled upon this reply on Stack Overflow, which, in addition to easy (set-cursor-color ...) answer, went on to give an example of changing color on every cusor blink.

Which seems like a really simple way to make the thing stand out not just against bg/fg colors, but also against any syntax-highlighting color anywhere, and draw even more attention to itself, which is even better.

With only a couple lines replacing a timer-func that normally turns cursor on-and-off (aka blinking), now it blinks with a glorious rainbow of colors:

And at least I can confirm that it totally works for even greater/more-immediate visibility, especially in multiple emacs windows (e.g. usual vertical split showing two code buffers), where all these cursors now change colors and impossible to miss at a glance, so you always know which point in code you're jumping to upon switching there - can recommend to at least try it out.

Bit weird that most code editor windows seem to have narrow-line cursor, even if of a separate color, given how important that detail is within that window, compared to e.g. any other arbitrary glyph in there, which would be notably larger and often more distinct/visible, aside from blinking.

One interesting part with a set of colors, as usual, is to generate or pick these somehow, which can be done on a color wheel arbitrarily, with results often blending-in with other colors and each other, more-or-less.

But it's also not hard to remember about L*a*b* colorspace and pick colors that will be almost certainly distinct according to ranges there, as you'd do with a palette for dynamic lines on a graph or something more serious like that.

These days I'm using i want hue generator-page to get a long-ish list of colors within specified ranges for Lightness (L component, to stand-out against light/dark bg) and Chroma parameters, and then pass css list of these into a color-b64sort script, with -b/--bg-color and -c/--avoid-color settings/thresholds.

Output is a filtered list with only colors that are far enough from the specified ones, to stand-out against them in the window, but is also sorted to have every next color picked to be most visually-distinct against preceding ones (with some decay-weight coefficient to make more-recent diff[s] most relevant), so that color doesn't blink through similar hues in a row, and you don't have to pick/filter/order and test these out manually.

Which is the same idea as with the usual palette-picks for a line on a chart or other multi-value visualizations (have high contrast against previous/other values), that seem to pop-up like this in surprisingly many places, which is why at some point I just had to write that dedicated color-b64sort thingie to do it.

Tweaking parameters to only get as many farthest-apart colors as needed for blinks until cursor goes static (to avoid wasting CPU cycles blinking in an idle window), ended up with a kind of rainbow-cursor, counting bright non-repeating color hues in a fixed order, which is really nice for this purpose and also just kinda cool and fun to look at.

Oct 18, 2022

Revisiting POSIX ACLs and Capabilities in python some 15 years later

A while ago as I've discovered for myself using xattrs, ACLs and capabilities for various system tasks, and have been using those in python2-based tools via C wrappers for libacl and libcap (in old mk-fg/fgc repo) pretty much everywhere since then.

Tools worked without any issues for many years now, but as these are one of the last scripts left still in python2, time has come to update those, and revisit how to best access same things in python3.

Somewhat surprisingly, despite being supported on linux since forever, and imo very useful, support for neither ACLs nor capabilities haven't made it into python 3.10's stdlib, but there is now at least built-in support for reading/writing extended attributes (without ctypes, that is), and both of these are simple structs stored in them.

So, disregarding any really old legacy formats, parsing ACL from a file in a modern python can be packed into something like this:

import os, io, enum, struct, pwd, grp

class ACLTag(enum.IntEnum):
  uo = 0x01; u = 0x02; go = 0x04; g = 0x08
  mask = 0x10; other = 0x20
  str = property(lambda s: s._name_)

class ACLPerm(enum.IntFlag):
  r = 4; w = 2; x = 1
  str = property(lambda s: ''.join(
    (v._name_ if v in s else '-') for v in s.__class__ ))

def parse_acl(acl, prefix=''):
  acl, lines = io.BytesIO(acl), list()
  if (v := != b'\2\0\0\0':
    raise ValueError(f'ACL version mismatch [ {v} ]')
  while True:
    if not (entry := break
    elif len(entry) != 8: raise ValueError('ACL length mismatch')
    tag, perm, n = struct.unpack('HHI', entry)
    tag, perm = ACLTag(tag), ACLPerm(perm)
    match tag:
      case ACLTag.uo | ACLTag.go:
      case ACLTag.u | ACLTag.g:
          name = ( pwd.getpwuid(n).pw_name
            if tag is tag.u else grp.getgrgid(n).gr_name )
        except KeyError: name = str(n)
      case ACLTag.other: lines.append(f'o::{perm.str}')
      case ACLTag.mask: lines.append(f'm::{perm.str}')
      case _: raise ValueError(tag)
  lines.sort(key=lambda s: ('ugmo'.index(s[0]), s))
  return '\n'.join(f'{prefix}{s}' for s in lines)

p = 'myfile.bin'
xattrs = dict((k, os.getxattr(p, k)) for k in os.listxattr(p))
if acl := xattrs.get('system.posix_acl_access'):
  print('Access ACLs:\n' + parse_acl(acl, '  '))
if acl := xattrs.pop('system.posix_acl_default', ''):
  print('Default ACLs:\n' + parse_acl(acl, '  d:'))

Where it's just a bunch of 8B entries with uids/gids and permission bits in them, and capabilities are even simpler, except for ever-growing enum of them:

import os, io, enum, struct, dataclasses as dcs

CapSet = enum.IntFlag('CapSet', dict((cap, 1 << n) for n, cap in enumerate((
  ' chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap'
  ' linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock'
  ' ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin'
  ' sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease'
  ' audit_write audit_control setfcap mac_override mac_admin syslog wake_alarm'
  ' block_suspend audit_read perfmon bpf checkpoint_restore' ).split())))

class Caps: effective:bool; permitted:CapSet; inheritable:CapSet

def parse_caps(cap):
  cap = io.BytesIO(cap)
  ver, eff = ((v := struct.unpack('I',[0]) >> 3*8) & 0xff, v & 1
  if ver not in [2, 3]: raise ValueError(f'Unsupported capability v{ver}')
  perm1, inh1, perm2, inh2 = struct.unpack('IIII',
  if (n := len( != (n_tail := {2:0, 3:4}[ver]):
    raise ValueError(f'Cap length mismatch [ {n} != {n_tail} ]')
  perm_bits, inh_bits = perm2 << 32 | perm1, inh2 << 32 | inh1
  perm, inh = CapSet(0), CapSet(0)
  for c in CapSet:
    if perm_bits & c.value: perm |= c; perm_bits -= c.value
    if inh_bits & c.value: inh |= c; inh_bits -= c.value
  if perm_bits or inh_bits:
    raise ValueError(f'Unrecognized cap-bits: P={perm_bits:x} I={inh_bits:x}')
  return Caps(eff, perm, inh)

p = 'myfile.bin'
try: print(parse_caps(os.getxattr(p, 'security.capability')))
except OSError: pass

Bit weird that wrappers along these lines can't be found in today's python 3.10, but maybe most people sadly still stick to suid and more crude hacks where more complex access permissions are needed.

One interesting thing I found here is how silly my old py2 stracl.c and strcaps.c look in comparison - it's screenfuls of lines of more complicated C code, tied into python's c-api, and have to be compiled wherever these tools are used, with an extra python wrappers on top - all for parsing a couple of trivial structs, which under linux ABI compatibility promises, can be relied upon to be stable enough anyway.

Somehow it's been the obvious solution back then, to have compiler check all headers and link these libs as compatibility wrappers, but I'd never bother these days - it'll be either ctypes wrapper, or parsing simple stuff in python, to avoid having extra jank and hassle of dependencies where possible.

Makes me wonder if that's also the dynamic behind relatively new js/rust devs dragging in a bunch of crap (like the infamous left-pad) into their apps, still thinking that it'd make life simpler or due to some "good practice" dogmas.

May 30, 2022

LESSOUTPUT filter workaround for broken unicode en-dash characters in manpages

Using man <something> in the terminal as usual, I've been noticing more and more manpages being broken by tools that produce them over the years in this one specific way - long command-line options with double-dash are being mangled into having unicode en-dash "–" prefix instead of "--".

Most recent example that irked me was yt-dlp(1) manpage, which looks like this atm (excerpt as of 2022-05-30):

-h, –help
  Print this help text and exit
  Print program version and exit
-i, –ignore-errors
  Ignore download and postprocessing errors. The download
  will be considered successful even if the postprocessing fails
  Continue with next video on download errors;
  e.g. to skip unavailable videos in a playlist (default)
  Abort downloading of further videos if an error occurs
  (Alias: –no-ignore-errors)

Update 2023-09-10: current feh(1) manpage has another issue - unicode hyphens instead of the usual ascii hyphen/minus signs (which look similar, but aren't the same thing - even more evil!):

       ‐A, ‐‐action [flag][[title]]action

If you habitually copy-paste any of the long opts there into a script (or yt-dlp config, as it happens), to avoid retyping these things, it won't work, because e.g –help cli option should of course actually be --help, i.e. have two ascii hyphens and not any kind of unicode dash characters.

From a brief look, this seem to happen because of conversion from markdown and probably not enough folks complaining about it, which is a pattern that I too chose to follow, and make a workaround instead of reporting a proper bug :)

(tbf, youtube-dl forks have like 1000s of these in tracker, and I'd rather not add to that, unless I have a patch for some .md tooling it uses, which I'm too lazy to look into)

"man" command (from man-db on linux) uses a pager tool to display its stuff in a terminal cli (controlled by PAGER= or MANPAGER= env vars), which is typically set to use less tool on desktop linuxes (with the exception of minimal distros like Alpine, where it comes from busybox).

"less" somewhat-infamously supports filtering of its output (which is occasionally abused in infosec contexts to make a file that installs rootkit if you run "less" on it), which can be used here for selective filtering and fixes when manpage is being displayed through it.

Relevant variable to set in ~/.zshrc or ~/.bashrc env for running a filter-script is:

LESSOPEN='|-man-less-filter %s'

With |- magic before man-less-filter %s command template indicating that command should be also used for pipes and when less is displaying stdin.

"man-less-filter" helper script should be in PATH, and can look something like this to fix issues in yt-dlp manpage excerpt above:

## Script to fix/replace bogus en-dash unicode chars in broken manpages
## Intended to be used with LESSOPEN='|-man-less-filter %s'

[ -n "$MAN_PN" ] || exit 0 # no output = use original input

# \x08 is backspace-overprint syntax that "man" uses for bold chars
# Bold chars are used in option-headers, while opts in text aren't bold
  `' s/\([ [:punct:]]\)–\([a-z0-9]\)/\1--\2/'
[ "$1" != - ] || exec sed "$seds"
exec sed "$seds" "$1"

It looks really funky for a reason - simply using s/–/--/ doesn't work, as manpages use old typewriter/teletype backspace-overtype convention for highlighting options in manpages.

So, for example, -h, –help line in manpage above is actually this sequence of utf-8 - -\x08-h\x08h,\x08, –\x08–h\x08he\x08el\x08lp\x08p\n - with en-dash still in there, but \x08 backspaces being used to erase and retype each character twice, which makes "less" display them in bold font in the terminal (using its own different set of code-bytes for that from ncurses/terminfo).

Simply replacing all dashes with double-hyphens will break that overtyping convention, as each backspace erases a single char before it, and double-dash is two of those.

Which is why the idea in the script above is to "exec sed" with two substitution regexps, first one replacing all overtyped en-dash chars with correctly-overtyped hyphens, and second one replacing all remaining dashes in the rest of the text which look like options (i.e. immediately followed by letter/number instead of space), like "Alias: –no-ignore-errors" in manpage example above, where text isn't in bold.

MAN_PN env-var check is to skip all non-manpage files, where "less" understands empty script output as "use original text without filtering". /bin/dash can be used instead of /bin/sh on some distros (e.g. Arch, where sh is usually symlinked to bash) to speed up this tiny script startup/runs.

Not sure whether "sed" might have to be a GNU sed to work with unicode char like that, but any other implementation can probably use \xe2\x80\x93 escape-soup instead of in regexps, which will sadly make them even less readable than usual.

Such manpage bugs should almost certainly be reported to projects/distros and fixed, instead of using this hack, but thought to post it here anyway, since google didn't help me find an exising workaround, and fixing stuff properly is more work.

Update 2023-09-10: Current man(1) from man-db has yet another issue to work around - it sanitizes environment before running "less", dropping LESSOPEN from there entirely.

Unfortunately, at least current "less" doesn't seem to support config file, to avoid relying entirely on apps passing env-vars around (maybe for a good reason, given how they like to tinker with its configuration), so easy fix is a tiny less.c wrapper for it:

#include <unistd.h>
#include <stdlib.h>
int main(int argc, char *argv[]) {
  if (getenv("MAN_PN")) setenv("LESSOPEN", "|-man-less-filter %s", 1);
  execv("/usr/bin/less", argv); return 1; }

gcc -O2 less.c -o less && strip less, copy it to ~/bin or such, and make sure env uses PAGER=less or a wrapper path, instead of original /usr/bin/less.

May 05, 2022

Alpine Linux on ODROID-C2 or any other ARM boards

Alpine Linux is a tiny distro (like ~50 MiB for the whole system), which is usually very good for a relatively single-purpose setup, dedicated to running only a couple things at once.

It doesn't really sacrifice much in the process, since busybox and OpenRC there provide pretty complete toolkit for setting things up to start and debugging everything, its repos have pretty much anything, and what they don't have is trivial to build via APKBUILD files very similar to Arch's PKGBUILDs.

Really nice thing seem to have happened to it with the rise of docker/podman and such single-app containers, when it got a huge boost in popularity, being almost always picked as an OS base in these, and so these days it seem to be well-supported to cover all linux architectures, and likely will be for quite a while.

So it's a really good choice for an appliance-type ARM boards to deploy and update for years, but it doesn't really bother specializing itself for each of those or supporting board-specific quirks, which is why some steps to set it up for a specific board have to be done manually.

Since I've done it for a couple different boards relatively recently, and been tinkering with these for a while, thought to write down a list of steps on how to set alpine up for a new board here.

In this case, it's a relatively-old ODROID-C2 that I have from 2016, which is well-supported in mainline linux kernel by now, as popular boards tend to be, after a couple years since release, and roughly same setup steps should apply to any of them.

Current Alpine is 3.15.4 and Linux is 5.18 (or 5.15 LTS) atm of writing this.

  • Get USB-UART dongle (any dirt-cheap $5 one will do) and a couple jumper wires, lookup where main UART pins are on the board (ones used by firmware/u-boot).

    Official documentation or wiki is usually the best source of info on that. There should be RX/TX and GND pins, only those need to be connected.

    screen /dev/ttyUSB0 115200 command is my go-to way to interact with those, but there're other tools for that too.

    If there's another ARM board laying around, they usually have same UART pins, so just twist TX/RX and connect to the serial console from there.

  • Check which System-on-Chip (SoC) family/vendor device uses, look it up wrt mainline linux support, and where such effort might be coordinated.

    For example, for Amlogic SoC in ODC2, has all the info.
    Allwinner SoCs tend to be documented at wiki.
    Beaglebones' TI SoC at, RPi's site/community, ... etc etc.

    There're usually an IRC or whatever chat channels linked on those too, and tons of helpful info all around, though sometimes outdated and distro-specific.

  • Get U-Boot for the board, ideally configured for some sane filesystem layout.

    "Generic ARM" images on Alpine Downloads page tend to have these, but architecture is limited to armv7/aarch64 boards, but a board vendor's u-boot will work just as well, and it might come with pre-uboot stages or signing anyway, which definitely have to be downloaded from their site and docs.

    With ODC2, there're docs and download links for bootloader specifically, which include an archive with a signing tool and a script to sign/combine all blobs and dd them into right location on microSD card, where firmware will try to run it from.

    Bootloader can also be extracted from some vendor-supplied distro .img (be it ubuntu or android), if there's no better option - dd that on the card and rm -rf all typical LFS dirs on the main partition there, as well as kernel/initramfs in /boot - what's left should be a bootloader (incl. parts of it dd'ed before partitions) and its configs.

    Often all magic there is "which block to dd u-boot.img to", and if Alpine Generic ARM has u-boot blob for specific board (has one for ODROID-C2), then that one should be good to use, along with blobs in /efi and its main config file in /extlinux - dd/cp u-boot to where firmware expects it to be and copy those extra stage/config dirs.

    Alpine wiki might have plenty of board-specific advice too, e.g. check out Odroid-C2 page there.

  • Boot the board into u-boot with UART console connected, get that working.

    There should be at least output from board's firmware, even without any microSD card, then u-boot will print a lot of info about itself and what it's doing, and eventually it (or later stages) will try to load linux kernel, initramfs and dtb files, specified in whatever copied configs.

    If "Generic ARM" files or whatever vendor distro was not cleaned-up from card partitions, it might even start linux and some kind of proper OS, though kernel in that generic alpine image seem to be lacking ODC2 board support and has no dtb for it.

    Next part is to drop some working linux kernel/dtb (+initramfs maybe), rootfs, and that should boot into something interactive. Conversely, no point moving ahead with the rest of the setup until bootloader and console connected there all work as expected.

  • Get or build a working Linux kernel.

    It's not a big deal (imo at least) to configure and build these manually, as there aren't really that many options when configuring those for ARM - you basically pick the SoC and all options with MESON or SUNXI in the name for this specific model, plus whatever generic features/filesystems/etc you will need in there. Idea is to enable with =y (compile-in) all drivers that are needed for SoC to boot, and the rest can be =m (loadable modules).

    Ideally after being built somewhere, kernel should be installed via proper Alpine package, and for my custom ODC2 kernel, I just reused linux-lts APKBUILD file from "aports" git repository, running "make nconfig" in src/build-... dir to make new .config and then copy it back for abuild to use.

    Running abuild -K build rootpkg is generally how kernel package can be rebuilt after any .config modifications in the build directory, i.e. after fixing whatever forgotten things needed for boot. (good doc/example of the process can be found at strfry's blog here)

    These builds can be run in same-arch alpine chroot, in a VM or on some other board. I've used an old RPi3 board to do aarch64 build with an RPi Alpine image there - there're special pre-configured ones for those boards, likely due to their massive popularity (so it was just the easiest thing to do).

    I'd also suggest renaming the package from linux-lts to something unique for specific build/config, like "linux-odc2-5-15-36", with idea there being able to install multiple such linux-* packages alongside each other easily.

    This can help a lot if one of them might have issues in the future - not booting, crashes/panics, any incompatibilities, etc - as you can then simply flip bootloader config to load earlier version, and test new versions without comitting to them easily via kexec (especially cool for remote headless setups - see below).

    For C2 board kconfig, I've initially grabbed one from superna9999/meta-meson repo, built/booted that (on 1G-RAM RPi3, that large build needed zram enabled for final vmlinuz linking), to test if it works at all, and then configured much more minimal kernel from scratch.

    One quirk needed there for ODC2 was enabling CONFIG_REGULATOR_* options for "X voltage supplied by Y" messages in dmesg that eventually enable MMC reader, but generally kernel's console output is pretty descriptive wrt what might be missing, especially if it can be easily compared with an output from a working build like that.

    For the purposes of initial boot, linux kernel is just one "vmlinuz" file inside resulting .apk (tar can unpack those), but dtbs-* directory must be copied from package alongside it too for ARM boards like these odroids, as they use blobs there to describe what is connected where and how to kernel.

    Specific dtb can also be concatenated into kernel image file, to have it all-in-one, but there should be no need to do that, as all such board bootloaders expect to have dtbs and have configuration lines to load them already.

    So, to recap:

    • Linux configuration: enable stuff needed to boot and for accessing MMC slot, or whatever rootfs/squashfs storage.
    • Build scripts: use APKBUILD from Alpine aports, just tweak config there and maybe rename it, esp. if you want multiple/fallback kernels.
    • Build machine: use other same-arch board or VM with alpine or its chroot.
    • Install: "tar -xf" apk, drop vmlinuz file and dtbs dir into /boot.
  • Update extlinux.conf or whatever bootloader config for new kernel files, boot that.

    E.g. after dropping vmlinuz and dtbs from "linux-c51536r01" package for u-boot.bin from Alpine's "Generic ARM" build (running efi blob with extlinux.conf support):

    TIMEOUT 20
    PROMPT 1
    DEFAULT odc2
    LABEL odc2
    MENU LABEL Linux odc2
    KERNEL /boot/vmlinuz-c51536r01
    # INITRD /boot/initramfs-c51536r01
    FDTDIR /boot/dtbs-c51536r01
    APPEND root=/dev/mmcblk0p1

    Idea with this boot config is to simply get kernel to work and mount some rootfs without issues, so e.g. for custom non-generic/modular one, built specifically for ODROID-C2 board in prev step, there's no need for that commented-out INITRD line.

    Once this boots and mounts rootfs (and then presumably panics as it can't find /sbin/init there), remaining part is to bootstrap/grab basic alpine rootfs for it to run userspace OS parts from.

    This test can also be skipped for more generic and modular kernel config, as it's not hard to test it with proper roots and initramfs later either.

  • Setup bootable Alpine rootfs.

    It can be grabbed from the same Alpine downloads URL for any arch, but if there's already a same-arch alpine build setup for kernel package, might be easier to bootstrap it with all necessary packages there instead:

    # apk='apk -p odc2-root -X -U --allow-untrusted'
    # $apk --arch aarch64 --initdb add alpine-base
    # $apk add linux-firmware-none mkinitfs
    # $apk add e2fsprogs # or ones for f2fs, btrfs or whatever
    # $apk add linux-c51536r01-5.15.36-r1.apk

    To avoid needing --allow-untrusted for anything but that local linux-c51536r01 package, apk keys can be pre-copied from /etc/apk to odc2-root/etc/apk just like Alpine's setup-disk script does it.

    That rootfs will have everything needed for ODC2 board to boot, including custom kernel with initramfs generated for it, containing any necessary modules. Linux files on /boot should overwrite manually-unpacked/copied ones in earlier test, INITRD or similar line can be enabled/corrected for bootloader, but after cp/rsync, rootfs still needs a couple additional tweaks (again similar to what setup-disk Alpine script does):

    • etc/fstab: add e.g. /dev/mmcblk0p1 / ext4 rw,noatime 0 1 to fsck/remount rootfs.

      I'd add tmpfs /tmp tmpfs size=30%,nodev,nosuid,mode=1777 0 0 there as well, since Alpine's OpenRC is not systemd and doesn't have default mountpoints all over the place.

    • etc/inittab: comment-out all default tty's and enable UART one.

      E.g. ttyAML0::respawn:/sbin/getty -L ttyAML0 115200 vt100 for ODROID-C2 board.

      This is important to get login prompt on the right console, so make sure to check kernel output to find the right one, e.g. with ODC2 board:

      c81004c0.serial: ttyAML0 at MMIO 0xc81004c0 (irq = 23, base_baud = 1500000) is a meson_uart

      But it can also be ttyS0 or ttyAMA0 or something else on other ARM platforms.

    • etc/securetty: add ttyAML0 line for console there too, same as with inittab.

      Otherwise even though login prompt will be there, getty will refuse to ask for password on that console, and immediately respond with "user is disabled" or something like that.

    This should be enough to boot and login into working OS on the same UART console.

  • Setup basic stuff on booted rootfs.

    With initial rootfs booting and login-able, all that's left to get this to a more typical OS with working network and apk package management.

    "vi" is default editor in busybox, and it can be easy to use once you know to press "i" at the start (switching it to a notepad-like "insert" mode), and press "esc" followed by ":wq" + "enter" at the end to save edits (or ":q!" to discard).

    • /etc/apk/repositories should have couple lines like, appropriate for this Alpine release and repos (just main + community, probably)

    • echo nameserver > /etc/resolv.conf + /etc/network/interfaces:

      printf '%s\n' >/etc/network/interfaces \
        'auto lo' 'iface lo inet loopback' 'auto eth0' 'iface eth0 inet dhcp'

      Or something like that.

      Also, to dhcp-configure network right now:

      ip link set eth0 up && udhcpc
      ntpd -dd -n -q -p
    • rc-update add anything used/useful from /etc/init.d.

      There aren't many scripts in there by default, and all should be pretty self-explanatory wrt what they are for.

      Something like this can be a good default:

      for s in bootmisc hostname modules sysctl syslog ; do rc-update add $s boot; done
      for s in devfs dmesg hwdrivers mdev ; do rc-update add $s sysinit; done
      for s in killprocs mount-ro savecache ; do rc-update add $s shutdown; done
      for s in networking ntpd sshd haveged ; do rc-update add $s; done

      Run rc-update show -v to get a full picture of what is setup to run when with openrc, should be much simpler and more comprehensible than systemd in general.

      sshd and haveged there can probably be installed after network works, or in earlier rootfs-setup step (and haveged likely unnecessary with more recent kernels that fixed blocking /dev/random).

    • ln -s /usr/share/zoneinfo/... /etc/localtime maybe?

    That should be it - supported and upgradable Alpine with a custom kernel apk and bootloader for this specific board.

  • Extra: kexec for updating kernels safely, without breaking the boot.

    Once some initial linux kernel boots and works, and board is potentially tucked away somewhere where it's hard to reach for pulling microSD card out, it can be scary to update kernel to potentially something that won't be able to boot, start crashing, etc.

    Easy solution for that is kexec - syscall/mechanism to start a new kernel from another working kernel/OS.

    Might need to build/install kexec-tools apk to use it - it's missing on some architectures, but APKBUILD from aports and the tool itself should work just fine without changes. Also don't forget to enable it in the kernel.

    Using multiple kernel packages alongside each other like suggested above, something like this should work for starting new linux from an old one immediately:

    # apk install --allow-untrusted linux-c51705r08-5.17.5-r0.apk
    # kexec -sf /boot/vmlinuz-c51705r08 --initrd /boot/initramfs-c51705r08 --reuse-cmdline

    It works just like userspace exec() syscalls, but gets current kernel to exec a new one instead, which generally looks like reboot, except without involving firmware and bootloader at all.

    This way it's easy to run and test anything in new kernel or in its cmdline options safely, with simple reboot or power-cycling reverting it back to an old known-good linux/bootloader setup.

    Once everything is confirmed-working with new kernel, bootloader config can potentially be updated to use it instead, and old linux-something package replaced with a new one as a known-good fallback on the next update.

    This process is not entirely foolproof however, as sometimes linux drivers or maybe hardware have some kind of init-state mismatch, which actually happens with C2 board, where its built-in network card fails to work after kexec, unless its linux module is unloaded before that.

    Something like this inside "screen" session can be used to fix that particular issue:

    # rmmod dwmac_meson8b stmmac_platform stmmac && kexec ... ; reboot

    "reboot" at the end should never run if "kexec" works, but if any of this fails, C2 board won't end up misconfigured/disconnected, and just reboot back into old kernel instead.

    So far I've only found such hack needed with ODROID-C2, other boards seem to work fine after kexec, so likely just a problem in this specific driver expecting NIC to be in one specific state on init.

This post is kinda long because of how writing works, but really it all boils down to "dd" for board-specific bootloader and using board-specific kernel package with a generic Alpine rootfs, so not too difficult, especially given how simple and obvious everything is in Alpine Linux itself.

Supporting many different boards, each with its unique kernel, bootloader and quirks seem to be a pain for most distros, which can barely get by with x86 support, but if the task is simplified to just providing rootfs and pre-built package repository, a reasonably well-supported distro like Alpine has a good chance to work well in the long run, I think.

ArchLinuxARM, Armbian and various vendor distros like Raspbian provide nicer experience out-of-the-box, but eventually have to drop support for old hardware, while these old ARM boards don't really go anywhere in my experience, and keep working fine for their purpose, hence more long-term bet on something like alpine seems more reasonable than distro-hopping or needless hardware replacement every couple years, and alpine in particular is just a great fit for such smaller systems.

May 04, 2022

Bit-banging interfaces on a PC motherboard

It's usually easy to get computers to talk over ethernet, but making traditional PC hardware to talk to electronics or a GPIO line tends to be more difficult.

Issue came up when I wanted to make a smart wakeup switch for a local backup ARM-board the other day, which would toggle separate 5V power to it and its drive in-between weekly backups, so went on to explore the options there.

Simplest one is actually to wake up an ARM board using Wake-on-LAN, which can then control own and peripherals' power via its GPIO lines, but don't think mine has that.

Next easiest one (from my pov) seem to be grabbing any Arduino-ish MCU laying around, hook that up to a relay and make it into a "smart switch" via couple dozen lines of glue code.

Problem with that approach is that PC mobos tend not to have simple GPIO lines easily exposed and intended for use from userspace, but there are options:

  • Parallel port, if motherboard is ancient enough, or if hooked-up via one of cheap USB-to-LPT cables with tiny controller in the connector.

    These have a bunch of essentially GPIO lines, albeit in a funny connector package, and with a weird voltage, but pretty sure only really old mobos have that built-in by now.

  • RS-232 COM port, likely via a header on the board.

    Dunno about latest hardware, but at least ones from 6-7 years ago can still have these.

    Even though it's a serial port, they have DTR and RTS pins, which are easy to control directly, using ioctl() with TIOCM_DTR and TIOCM_RTS flags on /dev/ttyS*.

    Outside of pure bit-banging, RS-232 can usually be hooked-up to simpler hardware's UART pins via RS232-to-TTL chip, instead of just GPIO via optocoupler.

  • PC Speaker!

    That's a 5V signal too (PWM, sure, but still no problem to read), and not like it's in much demand for anything else these days.

    Might have to either emit/detect distinct frequency or sequence of beeps to make sure that's not the signal of machine booting on the other side.

  • 3-pin fan headers with a voltage level controls.

    Can be a neat use for those analog pins on arduinos, though not all monitoring chips are RE-ed enough to have linux drivers with control over fans, unfortunately.

  • Power on USB ports as a signal - at least some mobos can put those to sleep via sysfs, I think, usually with per-controller level of granularity.

For my purposes, COM was available via an easy pin header, and it was easy enough to use its control pins, with some simple voltage-level shifting (COM itself has -12V/+12V for 0/1).

Bit of a shame that PCs don't traditionally have robust GPIO headers for users, feel like that'd have enabled a lot of fancy cas modding, home automation, or got more people into understanding basic electronics, if nothing else.

Apr 05, 2022

Dynamic policy routing to work around internet restrictions

Internet have been heavily restricted here for a while (Russia), but with recent events, local gov seem to have gotten a renewed shared interest in blocking whatever's left of it with everyone else in the world, so it obviously got worse.

Until full whitelist and/or ML-based filtering for "known-good" traffic is implemented though, simple way around it seem to be tunneling traffic through any IP that's not blocked yet.

I've used simple "ssh -D" socks-proxy with an on-off button in a browser to flip around restrictions (or more like different restriction regimes), but it does get tiresome these days, and some scripts and tools outside of browsing start to get affected. So ideally traffic to some services/domains should be routed around internal censorship or geo-blocking on the other side automatically.

Linux routing tables allow to do that for specific IPs (which is usually called "policy(-based) routing" or PBR), but one missing component for my purposes was something to sync these tables with the reality of internet services using DNS hostnames of ever-shifting IPs, blocks applied on HTTP protocol level or dropped after DPI filter on connection, and these disruptions being rather dynamic from day to day, most times looking like some fickle infants flipping on/off switches on either side.

Having a script to http(s)-check remote endpoints and change routing on the fly seem to work well for most stuff in my case, having a regular linux box as a router here, which can run such script and easily accomodate for such changes, implemented here:

Checks are pretty straightforward parallel curl fetches (as curl can be generally relied upon to work with all modern http[s] stuff properly), but processing their results can be somewhat tricky, and dynamic routing itself can be implemented in a bunch of different ways.

Generally sending traffic through tunnels needs some help from the firewall (esp. since straight-up "nat" in "ip rule" is deprecated), to at least NAT packets going through whatever local IPs and exclude checker app itself from workarounds (so that it can do its checks), so it seem to make sense marking route-around traffic there as well.

fwmarks based on nftables set work well for that, together with "ip rule" sending marked pkts through separate routing table, while simple skuid match or something like cgroup service-based filtering works for exceptions. There's a (hopefully) more comprehensive example of such routing setup in the nbrpc project's repo/README.

Strongly suspect that this kind of workaround is only a beginning though, and checks for different/mangled content (instead of just HTTP codes indicating blocking) might be needed there, workarounds being implemented as something masquerading as multiplexed H2 connections instead of straightforward encrypted wg tunnels, with individual exit points checked for whether they got discovered and have to be cycled today, as well as ditching centrally-subverted TLS PKI for any kind of authenticity guarantees.

This heavily reminds me of relatively old "Internet Perspectives" and Covergence ideas/projects (see also second part of old post here), most of which seem to have been long-abandoned, as people got used to corporate-backed internet being mostly-working and mostly-secure, with apparently significant money riding on that being the case.

Only counter-examples being heavily-authoritarian, lawless and/or pariah states where global currency no longer matters as much, and commercial internet doesn't care about those. Feel like I might need to at least re-examine those efforts here soon enough, if not just stop bothering and go offline entirely.

Aug 31, 2021

Easy control over applications' network access using nftables and systemd cgroup-v2 tree

Linux 5.13 finally merged nftables feature that seem to have been forgotten and lost as a random PATCH on LKML since ~2016 - "cgroupsv2" path match (upstreamed patch here).

Given how no one seemed particulary interested in it for years, guess either distros only migrated to using unified cgroup hierarchy (cgroup-v2) by default rather recently, so probably not many people use network filtering by these either, even though it's really neat.

On a modern linux with systemd, cgroup tree is mostly managed by it, and looks something like this when you run e.g. systemctl status:

/init.scope # pid-1 systemd itself

I.e. every defined app neatly separated into its own cgroup, and there are well-defined ways to group these into slices (or manually-started stuff into scopes), which is done automatically for instantiated units, user sessions, and some special stuff.

See earlier "cgroup-v2 resource limits for apps with systemd scopes and slices" post for more details on all these and some neat ways to use them for any arbitrary pids (think "systemd-run") as well as on-the-fly cgroup-controller resource limits.

Such "cgroup-controller" resource limits notably do not include networking, as it's historically been filtered separately from cpu/memory/io stuff via systemd-wide firewalls - ipchains, iptables, nftables - and more recently via various eBPFs microkernels - either bpfilter or cgroup and socket-attached BPFs (e.g. "cgroup/skb" eBPF).

And that kind of per-cgroup filtering is a very useful concept, since you already have these nicely grouping and labelling everything in the system, and any new (sub-)groups are easy to add with extra slices/scopes or systemd-run wrappers.

It allows you to say, for example - "only my terminal app is allowed to access VPN network, and only on ssh ports", or "this game doesn't get any network access", or "this ssh user only needs this and this network access", etc.

systemd allows some very basic filtering for this kind of stuff via IPAddressAllow=/IPAddressDeny= (systemd.resource-control) and custom BPFs, and these can work fine in some use-cases, but are somewhat unreliable (systemd doesn't treat missing resource controls as failure and quietly ignores them), have very limited matching capabilities (unless you code these yourself into custom BPFs), and are spread all over the system with "systemd --user" units adding/setting their own restrictions from home dirs, often in form of tiny hard-to-track override files.

But nftables and iptables can be used to filter by cgroups too, with a single coherent and easy-to-read system-wide policy in one file, using rules like:

add rule inet filter vpn.whitelist socket cgroupv2 level 5 \
  "user.slice/user-1000.slice/user@1000.service/app.slice/claws-mail.scope" \
  ip daddr mail.intranet.local tcp dport {25, 143} accept

I already use system/container-wide firewalls everywhere, so to me this looks like a more convenient and much more powerful approach from a top-down system admin perspective, while attaching custom BPF filters for apps is more useful in a bottom-up scenario and should probably mostly be left for devs and packagers, shipped/bundled with the apps, just like landlock or seccomp rulesets and such wrappers - e.g. set in apps' systemd unit files or flatpak containers (flatpaks only support trivial network on/off switch atm though).

There's a quirk in these "socket cgroupsv2" rules in nftables however (same as their iptables "-m cgroup --path ..." counterpart) - they don't actually match cgroup paths, but rather resolve them to numeric cgroup IDs when such rules are loaded into kernel, and not automatically update them in any way afterwards.

This means that:

  • Firewall rules can't be added for not-yet-existing cgroups.

    I.e. loading nftables.conf with a rule like the one above on early boot would produce "Error: cgroupv2 path fails: No such file or directory" from nft (and "xt_cgroup: invalid path, errno=-2" error in dmesg for iptables).

  • When cgroup gets removed and re-created, none of the existing rules will apply to it, as it will have new and unique ID.

Basically such rules in a system-wide policy config only work for cgroups that are created early on boot and never removed after that, which is not how systemd works with its cgroups, obviously - they are entirely transient and get added/removed as necessary.

But this doesn't mean that such filtering is unusable at all, just that it has to work slightly differently, in a "decoupled" fashion:

Following the decoupled approach: If the cgroup is gone, the filtering
policy would not match anymore. You only have to subscribe to events
and perform an incremental updates to tear down the side of the
filtering policy that you don't need anymore. If a new cgroup is
created, you load the filtering policy for the new cgroup and then add
processes to that cgroup. You only have to follow the right sequence
to avoid problems.

There seem to be no easy-to-find helpers to manage such filtering policy around yet though. It was proposed for systemd itself to do that in RFE-7327, but as it doesn't manage system firewall (yet?), this seem to be a non-starter.

So had to add one myself - mk-fg/systemd-cgroup-nftables-policy-manager (scnpm) - a small tool to monitor system/user unit events from systemd journal tags (think "journalctl -o json-pretty") and (re-)apply rules for these via libnftables.

Since such rules can't be applied directly, and to be explicit wrt what to monitor, they have to be specified as a comment lines in nftables.conf, e.g.:

# postfix.service :: add rule inet filter vpn.whitelist \
#   socket cgroupv2 level 2 "system.slice/postfix.service" tcp dport 25 accept

# app-mail.scope :: add rule inet filter vpn.whitelist socket cgroupv2 level 5 \
#   "user.slice/user-1000.slice/user@1000.service/app.slice/app-mail.scope" \
#   ip daddr mail.intranet.local tcp dport {25, 143} accept

add rule inet filter output oifname my-vpn jump vpn.whitelist
add rule inet filter output oifname my-vpn reject with icmpx type admin-prohibited

And this works pretty well for my purposes so far.

One particularly relevant use-case as per example above is migrating everything to use "zero-trust" overlay networks (or just VPNs), though on modern server setups access to these tend to be much easier to manage by running something like innernet (or tailscale, or one of a dozen other WireGuard tunnel managers) in netns containers (docker, systemd-nspawn, lxc) or VMs, as access in these systems tend to be regulated by just link availability/bridging, which translates to having right crypto keys for a set of endpoints with wg tunnels.

So this is more of a thing for more complicated desktop machines rather than proper containerized servers, but still very nice way to handle access controls, instead of just old-style IP/port/etc matching without specifying which app should have that kind of access, as that's almost never universal (outside of aforementioned dedicated single-app containers), composing it all together in one coherent systemd-wide policy file.

Aug 30, 2021

Sharing Linux kernel build cache between machines

Being an old linux user with slackware/gentoo background, I still prefer to compile kernel for local desktop/server machines from sources, if only to check which new things get added there between releases and how they're organized.

This is nowhere near as demanding on resources as building a distro kernel with all possible modules, but still can take a while, so not a great fit for my old ultrabook or desktop machine, which both must be 10yo+ by now.

Two obvious ways to address this seem to be distributed compilation via distcc or just building the thing on a different machine.

And distcc turns out to be surprisingly bad for this task - it doesn't support gcc plugins that modern kernel uses for some security features, requires suppressing a bunch of gcc warnings, and even then with or without pipelining it eats roughly same amount of machine's CPU as a local build, without even fully loading remote i5 machine, as I guess local preprocessing and distcc's own overhead is a lot with the kernel code already.

Second option is a fully-remote build, but packaging just kernel + module binaries like distros do there kinda sucks, as that adds an extra dependency (for something very basic) and then it's hard to later quickly tweak and rebuild it or add some module for some new networking or hardware thingy that you want to use - and for that to be fast, kbuild/make's build cache of .o object files needs to be local as well.

Such cache turns out to be a bit hard to share/rsync between machines, due to following caveats:

  • Absolute paths used in intermediate kbuild files.

    Just running mv linux-5.13 linux-5.13-a will force full rebuild for "make" inside, so have to build the thing in the same dir everywhere, e.g. /var/src/linux-5.13 on both local/remote machines.

    Symlinks don't help with this, but bind mounts should, or just using consistent build location work as well.

    There's a default-disabled KBUILD_ABS_SRCTREE make-flag for this, with docs saying "Kbuild uses a relative path to point to the tree when possible", but that doesn't seem to be case for me at all - maybe "when possible" is too limited, or was only true with older toolchains.

  • Some caches use "ls -l | md5" as a key, which breaks between machines due to different usernames or unstable ls output in general.

    One relevant place where this happens for me is kernel/, and can be worked around using "find -printf ..." there:

    % patch -tNp1 -l <<'EOF'
    diff --git a/kernel/ b/kernel/
    index 34a1dc2..bfa0dd9 100755
    --- a/kernel/
    +++ b/kernel/
    @@ -44,4 +44,5 @@ all_dirs="$all_dirs $dir_list"
    -headers_md5="$(find $all_dirs -name "*.h"                  |
    -           grep -v "include/generated/compile.h"   |
    -           grep -v "include/generated/autoconf.h"  |
    -           xargs ls -l | md5sum | cut -d ' ' -f1)"
    +           find $all_dirs -name "*.h" -printf '%p :: %Y:%l :: %s :: %T@\n' |
    +           grep -v "include/generated/compile.h" |
    +           grep -v "include/generated/autoconf.h" |
    +           sed 's/\.[0-9]\+$//' | LANG=C sort | md5sum | cut -d ' ' -f1)"
    @@ -50 +51 @@ headers_md5="$(find $all_dirs -name "*.h"                     |
    -this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
    +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)"

    One funny thing there is an extra sed 's/\.[0-9]\+$//' to cut precision from find's %T@ timestamps, as some older filesystems (like reiserfs, which is still great for tiny-file performance and storage efficiency) don't support too high precision on these, and that will change them in this output without any complaints from e.g. rsync.

  • Host and time-dependent KBUILD_* variables.

    These embed build user/time/host etc, are relevant for reproducible builds, and maybe not so much here, but still best to lock down for consistency via e.g.:


    All these vars are documented under Documentation/ in the kernel tree.

  • Compiler toolchain must be roughly same between these machines.

    Not hard to do if they're both same Arch, but otherwise probably best way to get this is to have same distro(s) for the build within containers (e.g. nspawn).

This allows to rsync -rtlz the build tree from remote after "make" and do the usual "make install" or tweak it locally later without doing slow full rebuilds.

← Previous Next → Page 2 of 16