Dec 29, 2015

Getting log of all function calls from specific source file using gdb

Maybe I'm doing debugging wrong, but messing with code written by other people, first question for me is usually not "what happens in function X" (done by setting a breakpoint on it), but rather "which file/func do I look into".

I.e. having an observable effect - like "GET_REPORT messages get sent on HID level to bluetooth device, and replies are ignored", it's easy to guess that it's either linux kernel or bluetoothd - part of BlueZ.

Question then becomes "which calls in app happen at the time of this observable effect", and luckily there's an easy, but not very well-documented (unless my google is that bad) way to see it via gdb for C apps.

For scripts, it's way easier of course, e.g. in python you can do python -m trace ... and it can dump even every single line of code it runs.

First of all, app in question has to be compiled with "-g" option and not "stripped", of course, which should be easy to set via CFLAGS, usually, defining these in distro-specific ways if rebuilding a package to include that (e.g. for Arch - have debug !strip in OPTIONS line from /etc/makepkg.conf).

Then running app under gdb can be done via something like gdb --args someapp arg1 arg2 (and typing "r" there to start it), but if the goal is to get a log of all function calls (and not just in a "call graph" way profiles like gprof do) from a specific file, first - interactivity has to go, second - breakpoints have to be set for all these funcs and then logged when app runs.

Alas, there seem to be no way to add break point to every func in a file.

One common suggestion (does NOT work, don't copy-paste!) I've seen is doing rbreak device\.c: ("rbreak" is a regexp version of "break") to match e.g. profiles/input/device.c:extract_hid_record (as well as all other funcs there), which would be "filename:funcname" pattern in my case, but it doesn't work and shouldn't work, as "rbreak" only matches "filename".

So trivial script is needed to a) get list of funcs in a source file (just name is enough, as C has only one namespace), and b) put a breakpoint on all of them.

This is luckily quite easy to do via ctags, with this one-liner:

% ctags -x --c-kinds=fp profiles/input/device.c |
  awk 'BEGIN {print "set pagination off\nset height 0\nset logging on\n\n"}\
    {print "break", $1 "\ncommands\nbt 5\necho ------------------\\n\\n\nc\nend\n"}\
    END {print "\n\nrun"}' > gdb_device_c_ftrace.txt

Should generate a script for gdb, starting with "set pagination off" and whatever else is useful for logging, with "commands" block after every "break", running "bt 5" (displays backtrace) and echoing a nice-ish separator (bunch of hyphens), ending in "run" command to start the app.

Resulting script can/should be fed into gdb with something like this:

% gdb -ex 'source gdb_device_c_ftrace.txt' -ex q --args\
  /usr/lib/bluetooth/bluetoothd --nodetach --debug

This will produce the needed list of all the calls to functions from that "device.c" file into "gdb.txt" and have output of the app interleaved with these in stdout/stderr (which can be redirected, or maybe closed with more gdb commands in txt file or before it with "-ex"), and is non-interactive.

From here, seeing where exactly the issue seem to occur, one'd probably want to look thru the code of the funcs in question, run gdb interactively and inspect what exactly is happening there.

Definitely nowhere near the magic some people script gdb with, but haven't found similar snippets neatly organized anywhere else, so here they go, in case someone might want to do the exact same thing.

Can also be used to log a bunch of calls from multiple files, of course, by giving "ctags" more files to parse.

May 19, 2014

Displaying any lm_sensors data (temperature, fan speeds, voltage, etc) in conky

Conky sure has a ton of sensor-related hw-monitoring options, but it still doesn't seem to be enough to represent even just the temperatures from this "sensors" output:

Adapter: ACPI interface
Vcore Voltage:      +1.39 V  (min =  +0.80 V, max =  +1.60 V)
+3.3V Voltage:      +3.36 V  (min =  +2.97 V, max =  +3.63 V)
+5V Voltage:        +5.08 V  (min =  +4.50 V, max =  +5.50 V)
+12V Voltage:      +12.21 V  (min = +10.20 V, max = +13.80 V)
CPU Fan Speed:     2008 RPM  (min =  600 RPM, max = 7200 RPM)
Chassis Fan Speed:    0 RPM  (min =  600 RPM, max = 7200 RPM)
Power Fan Speed:      0 RPM  (min =  600 RPM, max = 7200 RPM)
CPU Temperature:    +42.0°C  (high = +60.0°C, crit = +95.0°C)
MB Temperature:     +43.0°C  (high = +45.0°C, crit = +75.0°C)

Adapter: PCI adapter
temp1:        +30.6°C  (high = +70.0°C)
                       (crit = +90.0°C, hyst = +88.0°C)

Adapter: PCI adapter
temp1:        +51.0°C

Given the summertime, and faulty noisy cooling fans, decided that it'd be nice to be able to have an idea about what kind of temperatures hw operates on under all sorts of routine tasks.

Conky is extensible via lua, which - among other awesome things there are - allows to code caches for expensive operations (and not just repeat them every other second) and parse output of whatever tools efficiently (i.e. without forking five extra binaries plus perl).

Output of "sensors" though not only is kinda expensive to get, but also hardly parseable, likely unstable, and tool doesn't seem to have any "machine data" option.

lm_sensors includes a libsensors, which still doesn't seem possible to call from conky-lua directly (would need some kind of ffi), but easy to write the wrapper around - i.e. this sens.c 50-liner, to dump info in a useful way:

atk0110-0-0__in0_input 1.392000
atk0110-0-0__in0_min 0.800000
atk0110-0-0__in0_max 1.600000
atk0110-0-0__in1_input 3.360000
atk0110-0-0__in3_max 13.800000
atk0110-0-0__fan1_input 2002.000000
atk0110-0-0__fan1_min 600.000000
atk0110-0-0__fan1_max 7200.000000
atk0110-0-0__fan2_input 0.000000
atk0110-0-0__fan3_max 7200.000000
atk0110-0-0__temp1_input 42.000000
atk0110-0-0__temp1_max 60.000000
atk0110-0-0__temp1_crit 95.000000
atk0110-0-0__temp2_input 43.000000
atk0110-0-0__temp2_max 45.000000
atk0110-0-0__temp2_crit 75.000000
k10temp-0-c3__temp1_input 31.500000
k10temp-0-c3__temp1_max 70.000000
k10temp-0-c3__temp1_crit 90.000000
k10temp-0-c3__temp1_crit_hyst 88.000000
radeon-0-400__temp1_input 51.000000

It's all lm_sensors seem to know about hw in a simple key-value form.

Still not keen on running that on every conky tick, hence the lua cache:

sensors = {
  ts_read_i=120, ts_read=0,

function conky_sens_read(name, precision)
  local ts = os.time()
  if os.difftime(ts, sensors.ts_read) > sensors.ts_read_i then
    local sh = io.popen(sensors.cmd, 'r')
    sensors.values = {}
    for p in string.gmatch(sh:read('*a'), '(%S+ %S+)\n') do
      local n = string.find(p, ' ')
      sensors.values[string.sub(p, 0, n-1)] = string.sub(p, n)
    sensors.ts_read = ts

  if sensors.values[name] then
    local fmt = string.format('%%.%sf', precision or 0)
    return string.format(fmt, sensors.values[name])
  return ''

Which can run the actual "sens" command every 120s, which is perfectly fine with me, since I don't consider conky to be an "early warning" system, and more of an "have an idea of what's the norm here" one.

Config-wise, it'd be just cpu temp: ${lua sens_read atk0110-0-0__temp1_input}C, or a more fancy template version with a flashing warning and hidden for missing sensors:

template3 ${color lightgrey}${if_empty ${lua sens_read \2}}${else}\
${if_match ${lua sens_read \2} > \3}${color red}\1: ${lua sens_read \2}C${blink !!!}\
${else}\1: ${color}${lua sens_read \2}C${endif}${endif}

It can then be used simply as ${template3 cpu atk0110-0-0__temp1_input 60} or ${template3 gpu radeon-0-400__temp1_input 80}, with 60 and 80 being manually-specified thresholds beyond which indicator turns red and has blinking "!!!" to get more attention.

Overall result in my case is something like this:

conky sensors display

sens.c (plus Makefile with gcc -Wall -lsensors for it) and my conky config where it's utilized can be all found in de-setup repo on github (or my git mirror, ofc).

Member of The Internet Defense League