It seems like halInit() or chSysInit() resets all ports to the default settings in board.h. So my pad configuration got overwritten. After changing the order of the function calls it works correctly...
I can't believe it took me almost 2 days to find this - I should get a debugger hardware...