(gdb) break *0x972

Debugging, GNU± Linux and WebHosting and ... and ...

i3 named workspaces

Monday, September 19, 2016 - No comments

I recently improved my i3-wm configuration regarding the handling of workspaces. The idea is to have a quicker access to Firefox and Emacs, the main GUI tools I use for working. I initially added bindings to Urxvt terminal, but I happen to use it too much, so I cannot bind it to a particular workspace, they have to appear where ever I am :-)

Workspace names

set $wp_web 1: web
set $wp_edit 2: emacs
set $wp_work 3: work

set $wp_1 1: web # cannot use $variable here ...
set $wp_2 2: emacs
set $wp_3 3: work

assign [class="Firefox"] ? $wp_web
assign [class="^Emacs$"] ? $wp_edit

Switching workspace

# Win+1, Win+2 ...
bindcode Mod4+10 workspace $wp_1
bindcode Mod4+11 workspace $wp_2
bindcode Mod4+12 workspace $wp_3
bindcode Mod4+13 workspace 4
...

# quick Fn accesses
# F1 is Firefox, always focus to it during quick switches. 
# Same for F2 and emacs
bindsym F1 workspace $wp_1 ; [class="Firefox"] focus 
bindsym F2 workspace $wp_2 ; [class="Emacs"] focus
bindsym F3 workspace $wp_3
bindsym F4 workspace 4
# skip F5 (refresh in Firefox)
...

# quick back-and-forth with special key
bindsym Menu workspace back_and_forth # I would prefer a back-and-forth *on output*

# win-left/right
bindcode Mod4+86 workspace next
bindcode Mod4+82 workspace prev


# Mouse switch 1: left  2: middle  3: right
bindsym --whole-window Mod4+button1 workspace prev_on_output
bindsym --whole-window Mod4+button2 workspace back_and_forth
bindsym --whole-window Mod4+button3 workspace next_on_output

Moving workspaces and windows

# quick move
bindsym Shift+F1 move workspace $wp_1
bindsym Shift+F2 move workspace $wp_2
bindsym Shift+F3 move workspace $wp_3
bindsym Shift+F4 move workspace 4
...

# MOVE+Next/Prior keys to move the workspace
bindsym $Mod_move+Next move workspace next
bindsym $Mod_move+Prior move workspace prev

# Special key (UP/X) to move only window
bindcode $Mod_move+66 move window to output right;focus output right

# F12/F11 to move to hidden scratchpad / show it 
bindsym F12 move scratchpad
bindsym F11 scratchpad show

Simple strace debugging

Tuesday, August 30, 2016 - No comments

Today my URxvt terminals are stuck. All of them, at the same time.

I know they're just clients (urxvtc) for the main daemon (urxvtd) so I'm not surprised that all of them are stuck, but I don't know why.

URxvt standalone version (urxvt) works well, so I can start inverstigating (by the way, in my i3 environment I mainly use URxvt client (bound to Mod+Shift+Enter), because it's faster to start, but I have a backup binding Mod+Ctrl+Enter for the standalone version, just in case things like that happen :).

Then, with the right tool, it's easy to find the problem:

 $ strace -p $(pidof urxvtd)
 strace: Process 1518 attached
 wait4(20063, 
 ^C
strace: Process 1518 detached
 <detached ...>

So PID 20063 is the problem, urxvtd is waiting for it forever. But who is that?

$ ps aux | grep 20063 
kevin    20063  0.0  0.0  13628  2920 ?        S    10:45   0:00 sh -c echo -en toto\.ods | xsel -i -b -p

Alright, it's my clipboard management script that doesn't return ...

$ kill -9 20063

Bye bye clipboard, and hello again my terminals :-)

Looking for a Developer/Researcher Position in April 2017

Monday, May 02, 2016 - No comments

My contract with the University of Grenoble ends at the end of March 2017, so I'm looking for a developer/researcher position in Grenoble area, or remotely, in the field of debugging tools, but also runtime environments, compilers, verification & validation for HPC and embedded systems.

A printer-friendly version of my CV is also available. Non (or more) anonymized versions can also be generated on demand.

Publié dans :  

Break if outside of section

Tuesday, April 26, 2016 - No comments

Just a little example of how to use gdb.py to create a breakpoint that only fires if called outside of a give section :

int main() {
  srand(time(NULL));

  for (int i = 0; i < 100; i++) {
    if (rand() > THRESHOLD) do_in_section(0);

    start();
    do_in_section(1);
    stop();
  }
}

Functions start() and stop() delimit the section: it is okay to call do_in_section() inside the section. However, every once in a while, do_in_section() is called from outside of this section. (Forget about that boolean flag parameter, it's just for logging and consistency checks. Real-life codes won't have such an easy way to discriminate outside from inside call ... otherwise you don't need an advanced breakpoint!)

(gdb) break do_in_section

will lead to many useless stops, as most of the call to do_in_section() are legal, so what should do is:

  • set an internal breakpoint on start(), where we set a flag and continue
  • set an internal breakpoint on stop()` where we unset the flag and continue
  • set a breakpoint on do_in_section() that checks that flag, and only stop if outside of the section

which gives in Python:

import gdb

in_section = False

class StartStopBreakpoint(gdb.Breakpoint):
    def __init__(self, loc, is_start):
        gdb.Breakpoint.__init__(self, loc, internal=True)
        self.silent = True

        self.is_start = is_start

    def stop(self):
        global in_section

        if self.is_start: # I know that this 'if' is not necessary ...
            assert not in_section
            in_section = True
        else:
            assert in_section
            in_section = False

        return False # never stop here

class SectionBreakpoint(gdb.Breakpoint):
    def __init__(self, location):
        gdb.Breakpoint.__init__(self, location, internal=True)
        self.silent = True

    def stop(self):
        if in_section:
            print("ignore hit in section")
            assert gdb.parse_and_eval("inside") == 1
            return False

        else:
            print("")
            print("Section breakpoint hit outside of section")
            print("")
            assert gdb.parse_and_eval("inside") == 0
            return True

StartStopBreakpoint("start", is_start=True)
StartStopBreakpoint("stop", is_start=False)
SectionBreakpoint("do_in_section")

Just source that file, or input it in GDB python commandline. The three last line instantiate the breakpoints:

$ gdb -ex "source test.py" a.out
GNU gdb (GDB) 7.10.50.20160122-git
...
(gdb) run
Starting program: /home/kevin/a.out 
ignore
ignore ....

Section breakpoint hit outside of section

(gdb) print inside
$1 = 0

Et voila!

Extending Linux Perf Stat with LD_PRELOAD

Friday, March 25, 2016 - No comments

For my current work on monitoring, I need to use Linux perf stat. Perf tools read and dislay the harware counters, either for the wall execution of a process, or while attaced to it:

perf stat --pid $(pidof firefox)
^C
 Performance counter stats for process id '4257':

     13.860180      task-clock (msec)
            79      context-switches           
            16      cpu-migrations
            11      page-faults             
    18,397,934      cycles
    13,964,242      stalled-cycles-frontend 
     9,787,703      stalled-cycles-backend
     8,320,570      instructions
     1,743,632      branches
        93,080      branch-misses

   1.942768382 seconds time elapsed

That's great, I can attach perf to my process, run it for a while and stop it. But if I want to start, and stop, and start again, I can't. And that's what I want to do, from inside gdb.py.

Signals would be great for that, when ever I send a signal, perf dumps the counter values to stderr, and continues. But that's not implemented ...

Studying perf-stat source code

So let's see in perf-stat source code what we can find for that.

static void abs_printout(int id, int nr, struct perf_evsel *evsel, double avg)
static void print_aggr(char *prefix)

these look like good candidates, but their symbols are not exported ...

(gdb) print abs_printout
No symbol "abs_printout" in current context.
(gdb) print print_aggr
No symbol "print_aggr" in current context.

Next candidate?

static void process_interval(void)
(gdb) p process_interval
$3 = {void (void)} 0x441660 <process_interval>

Oh, great, and with no arguments, that's even easier ! Let's try it:

(gdb) call process_interval()
  3858.     571793.500217      task-clock
  3858          1,880,243      context-switches
  3858            115,610      cpu-migrations           
  3858          8,639,477      page-faults              
  3858  1,430,093,310,944      cycles
  3858  1,037,644,029,921      stalled-cycles-frontend
  3858    756,612,594,751      stalled-cycles-backend 
  3858    820,483,606,606      instructions
  3858    176,772,490,245      branches
  3858      5,454,270,151      branch-misses

Exactly what we were looking for !

Triggering process_interval() with a signal

Next, we need to be able to trigger this function remotely, and without modifying perf-stat source code. (The easy option would be to patch perf-stat, but then our tool would be harder to distribute).

That can be done with the help of Linux LD_PRELOAD trick: we preload a bit of code inside perf-stat address space, and during the application initialization, we register the signal handler:

void (*process_internal)(void) = (void *) PROCESS_INTERNAL;

void my_handler(int signum) {
    if (signum != SIGUSR2) {
       return;
    }

    printf("Received SIGUSR2!\n");
    process_internal();
}

void init(void) __attribute__((constructor));

void init(void){
  printf("Received init!\n");
  signal(SIGUSR2, my_handler);
}

Here the (GCC specific) function attribute constructor tells the linker to execute the function when it's loaded. In this constructor, we just register the signal handler, and in the signal handler we call process_internal().

But how do we get the address of process_internal? that's where it is not really really clean: it get it from a lookup of the binary's symbol addresses:

nm -a /usr/bin/perf | grep process_interval
0000000000441660 t process_interval

and we inject it manually in the code:

#define PROCESS_INTERNAL 0x0000000000441660
void (*process_internal)(void) = (void *) PROCESS_INTERNAL;

A better way would be to pass it as an environment variable:

#define PROCESS_INTERNAL ascii_to_long($PROCESS_INTERNAL_ADDR)

and in the shell:

PROCESS_INTERNAL_ADDR=0x$(nm -a /usr/bin/perf | grep process_interval | cut -d" " -f1)

and that works pretty well !

LD_PRELOAD=/path/to/preload.so perf stat -x, -p $PID_TO_PROFILE

kill -USR2 $(pidof perf)

Note that SIGUSR1 does't work, certainly because it's used by perf-stat. And don't forget the -x, argument if you want to parse perf-stat output.

Also, keep in mind that this is not a robust implementation (of course :), as there might be concurrency problems, for instance if the signal is received while perf is updating the counters.

If it ain't broken don't fix it; I'll wait for problems before going any further!

Riddle with Python 2/3 and GDB.py checkpoint-restart

Wednesday, March 09, 2016 - No comments

At the moment, I'm playing with thread checkpoint-restart, implemented somehow like setjmp/longjmp :

REGS = "rax","rbx","rcx","rdx","rsi","rdi","rbp","rsp","r8","r9", ...

def checkpoint(): 
  reg_values = map(gdb.newest_frame().read_register, REGS))
  return reg_values

def restart(reg_values):
  for reg, val in zip(REGS, reg_values):
    gdb.execute("set ${} = {}".format(reg, val), to_string=True)

This a naive implementation, but with a few more tricks, it does work (what I have to do is save/restore rip and rbp for the frame above, I'm not sure why).

However this code works for Python 2, but not for Python 3. Can you find why?

I'll let you guess, the answer is written below in white:

The map function in Python 3 returns a generator, which is lazily evaluated. Thus in Python 3, the registers are actually read ... in the restart function! So, of course, they don't hold the values of the check-point, and hence the restart function does nothing!

Photowall generator

Thursday, February 04, 2016 - No comments

This is a GPL 3.0 Python photowall generator, relying on ImageMagick. It is (and will remain) in "beta testing" version : I use it and it fits my requirements, but there is no guarantee that it won't crash in any different circunstances.

Python developers should be able to fix it and update it fairly easy, although the code is a bit old and could benefit from a face lifting (I wrote it in 2012).

An example of customization for Python developers: Polaroid captions are computed by function photowall.get_file_details. Just rewrite this function to decide what you want to use for caption. Current code (in the exception handler :$) takes the filename without extension and replaces underscores by new lines.

Usage:

 photowall.py <path> <target> [options]

 Arguments:
   <path>        The path where photos are picked up from.
   <target>      The path where the target photo is written. 
                 Except in POLAROID+RANDOM mode, the image will 
                 be blanked out first.

 Options:
   --polaroid      Use polaroid-like images for the wall
   --pick-random   Pick images randomly in the <path> folder.
   --help          Display this message

 Size options:
   --nb-lines <nb>         Number on lines of the target image. 
   --line-height <height>  Set the height of a single image.
   --width <width>         Set final image width. 
   --no-resize    Resize images before putting in the wall.

 Polaroid mode options:
   --crop-size <crop>      Minimum size to allow cropping 
                           an image, if it doesn't fit
   --no-caption            Disable caption.
   --put-random            Put images randomly
                           instead of linearily.
   --sleep <time>          If --put-random, time (in seconds)
                           to go asleep 
                           before adding a new image.

 Collage mode options:
   --do-wrap               Finish images on the next line? 

Examples

Polaroid mode with caption

--polaroid  --nb-lines 1 --pick-random

Polaroid mode with caption

Polaroid mode with no caption

--polaroid --no-caption --nb-lines 1 --pick-random

Polaroid mode with no caption

Collage mode

--nb-lines 1 --pick-random

Collage mode

Polaroid random mode

--put-random --nb-lines 1 --pick-random

Polaroid random mode

Graphical interface

I built a graphical interface that should support all the command-line arguments. Use it with caution, I don't like writting GUIs !

GUI

Tricks for Python debugger pdb

Tuesday, February 02, 2016 - No comments

Using pdb in PyGTK applications

If you try to run pdb.set_trace() on a PyGTK application, the command-line prompt will be unusable, certainly because of (native) multithreading. Solution is simple once you got it, but not necessarily easy to find on the Internet:

def import_pdb_set_trace():
  '''Set a tracepoint in PDB that works with Qt'''
  from PyQt5.QtCore import pyqtRemoveInputHook
  pyqtRemoveInputHook()

  import pdb; pdb.set_trace() 

PDB aliases [for gdb.py scripting]

# gdb.py
alias ge import gdb;gdb.execute("%*") #  gdb execute
alias gq import gdb;gdb.execute("quit") # gdb quit

# force quit
alias fq import os;os._exit(0) 

# quick print
alias pr print(%1)

# dir()
alias dir for k in dir(%1): print("{}".format(k))
alias dirv for k in dir(%1): print("{} --> {}".format(k, getattr(%1, k)))

alias ds dir self

Hacktoberfest 2015

Sunday, January 10, 2016 - No comments

During the month of October, I participated to DigitalOcean's Hacktoberfest. The goal is to foster Open Source programming, and the rule is easy: you just have to submit 3 pull-requests on Github. No matter the size of the pull request, no matter which project, big, small, even your own project, and the pull request doesn't even have to be accepted. It's easy to "cheat", but I played to game fairly and prepared three pull-request with actual features/improvements.

Selfloss RSS Reader

Add support for private (@) and hidden (#) tags

Selfloss is a PHP feed reader, with a per-source tagging mechanism. I needed two things:

1/ private tags (with a @ in the tag name): items and sources with such tags are not visible if the user is not logged in.

2/ "not important" tags (with a # in the tag name): items are not visible in the "all tags" feed, but only when you click directly on the tag or source.

Orochi

Add Python 3 support

Orochi is a command-line client for 8tracks online music player. It didn't support Py3, which is the default on my Archlinux system. Instead of fixing shebang, I updated the code to support Python 3.

PyQuadStick

Add a keyboard controller

PyQuadStick/PyQuadSim is a quadcopter simulator written in Python for the Virtual Robot Experimentation Platform (V-REP). PyQuadStick worked with R/C transmitters, joysticks and PS3 controllers, but I had none of them, so I wrote and contributed a keyboard controller.

Shaarli

Allow setting custom thumbnails

Shaarli is a micro-blogging engine, focused on link sharing. For my online library, I wanted to be able to provide custom thumbnails, for the book covers. Unfortunately (for my patch), they are in the middle of a refactoring, with a focus on plugins, to not clober the core of the engine and keep it minimal. So I rewrote my original patch to fit that design, but it won't be included until the refactoring is done.

Simple GDB Extensions with Python

Friday, January 08, 2016 - No comments

On Stackoverflow, I noticed that some people find missing features in GDB. There features are nothing complicated, just combination of existing commands that would be useful for them, but that are not part of the native set of commands.

GDB/Python interface is perfect to solve that kind of problem :-) It just requires simple Python skills and a little bit of documentation reading:

Does GDB have a step-to-next-call instruction?

SO question

Simple answer: no, step-to-next-call is not part of GDB commands.

GDB/Python-aware answer: no, it's not part of GDB commands, but it's easy to implement!

  • To stop before, you need to stepi/nexti (next assembly instruction) until you see call in the current instruction:

--

import gdb

class StepBeforeNextCall (gdb.Command):
    def __init__ (self):
        super (StepBeforeNextCall, self).__init__ ("step-before-next-call",
                                                   gdb.COMMAND_OBSCURE)

    def invoke (self, arg, from_tty):
        arch = gdb.selected_frame().architecture()

        while True:
            current_pc = addr2num(gdb.selected_frame().read_register("pc"))
            disa = arch.disassemble(current_pc)[0]
            if "call" in disa["asm"]: # or startswith ?
                break

            SILENT=True
            gdb.execute("stepi", to_string=SILENT)

        print("step-before-next-call: next instruction is a call.")
        print("{}: {}".format(hex(int(disa["addr"])), disa["asm"]))

def addr2num(addr):
    try:
        return int(addr)  # Python 3
    except:
        return long(addr) # Python 2

StepBeforeNextCall()
  • To stop after the call, you compute the current stack depth, then step until it's deeper:

--

import gdb

def callstack_depth():
    depth = 1
    frame = gdb.newest_frame()
    while frame is not None:
        frame = frame.older()
        depth += 1
    return depth

class StepToNextCall (gdb.Command):
    def __init__ (self):
        super (StepToNextCall, self).__init__ ("step-to-next-call", 
                                               gdb.COMMAND_OBSCURE)

    def invoke (self, arg, from_tty):
        start_depth = current_depth =callstack_depth()

        # step until we're one step deeper
        while current_depth == start_depth:
            SILENT=True
            gdb.execute("step", to_string=SILENT)
            current_depth = callstack_depth()

        # display information about the new frame
        gdb.execute("frame 0")

StepToNextCall() 

Relevant documentation is there:

Debug a source file using GDB without stepping into library calls?

SO question

You can see my answer to Does GDB have a “step-to-next-call” instruction? : there is no native GDB command for that (as far as I know, they may have worked on that), but it's easy to do in Python:

import gdb

class StepNoLibrary (gdb.Command):
    def __init__ (self):
        super (StepNoLibrary, self).__init__ ("step-no-library",
                                              gdb.COMMAND_OBSCURE)

    def invoke (self, arg, from_tty):
        step_msg = gdb.execute("step", to_string=True)

        fname = gdb.newest_frame().function().symtab.objfile.filename

        if fname.startswith("/usr"):
            # inside a library
            SILENT=False
            gdb.execute("finish", to_string=SILENT)
        else:
            # inside the application
            print(step_msg[:-1])

    StepNoLibrary()

It's easy to read what it does:it goes one step forward, and if the step ends up in a file stored in /usr/*, it finishes the function to come back to the application.

How to set skipping of uninteresting functions from gdbinit script?

SO question

Problem: if in .gdbinit you write skip uninteresting_function, gdb complains No function found named ... because the symbols files are not loaded yet.

Python solution: new command skip_pending

import gdb

to_skip = []

def try_pending_skips(evt=None):
    for skip in list(to_skip): # make a copy for safe remove
        try:
            # test if the function (aka symbol is defined)
            symb, _ = gdb.lookup_symbol(skip)
            if not symb:
                continue
        except gdb.error:
            # no frame ?
            continue
        # yes, we can skip it
        gdb.execute("skip {}".format(skip))
        to_skip.remove(skip)

    if not to_skip:
        # no more functions to skip
        try:
            gdb.events.new_objfile.disconnect(try_pending_skips) # event fired when the binary is loaded
        except ValueError:
            pass # was not connected

class cmd_pending_skip(gdb.Command):
    self = None

    def __init__ (self):
        gdb.Command.__init__(self, "pending_skip", gdb.COMMAND_OBSCURE)

    def invoke (self, args, from_tty):
        global to_skip

        if not args:
            if not to_skip:
                print("No pending skip.")
            else:
                print("Pending skips:")
                for skip in to_skip:
                    print("\t{}".format(skip))
            return

        new_skips = args.split()
        to_skip += new_skips

        for skip in new_skips:
            print("Pending skip for function '{}' registered.".format(skip))

        try:
            gdb.events.new_objfile.disconnect(try_pending_skips) 
        except ValueError: pass # was not connected

        # new_objfile event fired when the binary and libraries are loaded in memory
        gdb.events.new_objfile.connect(try_pending_skips)

        # try right away, just in case
        try_pending_skips()

cmd_pending_skip()

Save this code into a Python file pending_skip.py (or surrounded with python ... end in your .gdbinit), then:

source pending_skip.py
pending_skip fct1
pending_skip fct2 fct3
pending_skip # to list pending skips

The Python code will automatically check if the function can be skipped (i.e., if it is defined) whenever a symbol file is loaded. Running the command with no argument list the remaining pending skips.

Documentation references:

How can I use gdb to catch the moment when a function returns false?

SO question

import gdb
class FunctionFinishBreakpoint (gdb.FinishBreakpoint):
    def __init__ (self):
        gdb.FinishBreakpoint.__init__(self, gdb.newest_frame(), 
                                      internal=True)
        self.silent = True 

    def stop(self):
        #print("after: {}".format(int(self.return_value)))
        return not int(self.return_value)

class FunctionBreakpoint(gdb.Breakpoint):
    def __init__ (self, spec):
        gdb.Breakpoint.__init__(self, spec)
        self.silent = True

    def stop (self):
        #print("before")
        FunctionFinishBreakpoint() # set breakpoint on function return

        return False # do not stop at function entry

FunctionBreakpoint("test")

Save that in a finish.py file, edit it to your needs and source it from GDB, or run it between python ... end or in python-interactive (pi).

This code creates a FunctionBreakpoint, that triggers FunctionBreakpoint.stop eachtime function test is hit. The callback is silent, and only creates a FunctionFinishBreakpoint, that stops at the end of the current frame (ie, at the end of your function). That second stop calls FunctionFinishBreakpoint.stop, which tests if the return value evaluates to true or false. If it is "not true", it tells GDB to stop the execution.

Documentation references:

(gdb.FinishBreakpoint was added to GDB Python interface for that very purpose, by myself :-)

(last time I checked, there was an efficiency problem with these FinishBreakpoint, you may notice it if your function is called very often)