high-level overview

we are given a binary called a.out and an assignment which hints towards XOR (Exclusive OR) operations. from the Break the Syntax 2025 CTF, this was the rev task with the least amount of solves, which is surprising due to its simple nature, as opposed to other rev tasks.

control flow architecture

(i used Binja with Pseudo C decompilation)

the binary uses a tightly-scoped entrypoint, passing all logic through a single function:

int main(int argc, char** argv) {
    sub_40119d(argv[1]);
    return 0;
}

this is a single entry point into decryption and validation logic. the function sub_40119d sets up the xor-based stage exec.

sub_40119d

inside of the sub_40119d function, we observe :

1. the first input byte is extracted and used as a decryption keyinput[0]
2. the function sub_401149 is called to xor-decrypt a 0xc7-byte block at 0x4073d2
3. the binary then enters a loop, calling each entry in a function pointer table stored at 0x404c60, incrementing through the table

simplified , cleaned up pseudocode (i renamed the function sub_401149 that was mentioned in step 2 as xor_decrypt

void sub_40119d(char* input) {
    uint8_t xor_key = input[0];
    xor_decrypt(0x4073d2, 0xc7, xor_key);

    void (**table)() = (void**)0x404c60;
    size_t i = 1;
    while (1) {
        table[i - 1](input[i]);
        i++;
    }
}

xor decryption logic

sub_401149 , or as we renamed it, xor_decrypt , applies a basic in-place xor over a fixed-size buffer:

void xor_decrypt(uint8_t* data, size_t len, uint8_t key) {
    for (size_t i = 0; i < len; ++i) {
        data[i] ^= key;
    }
}

function pointer chain

starting at 0x404c60 , the binary defines a flat array of 48 function pointers. each points to an encrypted memory section (e.g., .f_0_section, .f_1_section, ...). these regions contain encrypted functions that validate one byte of input each.

pointer layout:

0x404c60 → 0x4073d2
0x404c68 → 0x40749b
...

none of these function blocks are valid code until decrypted. after decryption, the first few bytes conform to a typical function prologue. this is the key to decryption, remember

decrypting stage 0: key discovery

to decrypt the first block at 0x4073d2 , we brute-force the xor key until we find a valid x86 function signature. one such example ->

55             push   rbp
48 89 e5       mov    rbp, rsp

== mentioned prologue.

only one key yields this result: 0x42 (ascii 'B'). so the correct first byte of input is B.

flag recovery

each stage uses a predictable function header and the validation logic is simple. so our plan is to :

1. extract the encrypted block for each function pointer
2. brute force xor keys 0x00-0xff until the decrypted version starts with :

55 48 89 e5

3. record the key — it corresponds to one flag byte.

this approach yields a sequence of 48 xor keys, each independently discoverable :

}3M1T_4_T4_R0X_3NO_ZSU3D4T_N4P_GN1TPYRC3D{FTCStB

oh wait, its reversed? O_o (think about why)

final flag :

BtSCTF{D3CRYPT1NG_P4N_T4D3USZ_ON3_X0R_4T_4_T1M3}

solved by tlsbollei