The challenge give us two file, challenge.py and encrypted.bin.

#!/usr/bin/python3

import random
from math import gcd

def encrypt(dt):
        mod = 256
        while True:
                a = random.randint(1,mod)
                if gcd(a, mod) == 1: break
        b = random.randint(1,mod)

        res = b''
        for byte in dt:
                enc = (a*byte + b) % mod
                res += bytes([enc])
        return res

dt = open('letter.pdf', 'rb').read()

res = encrypt(dt)

f = open('encrypted.bin', 'wb')
f.write(res)
f.close()

It seems like Affine Cipher encryption since it doing multiplication of a with byte from the file, then add the result with b and find the modulo with 256.

If we googling about affine cipher decryption method, it will show us that it can be decrypted with

a^-1 * ( x - b) % mod

Implementation of Affine Cipher - GeeksforGeeks

Because i'm bad at math, i will just brute the value from a and b since we know that pdf file always start with %PDF- or 25 50 44 46 2D in hex.

First we need to get first 5 bytes from the encrypted file, we can do that like this

Then we can make a bruteforcer to find the exact a and b value with this code

And we can now decode the encrypted file with this code

Bring it all into one file, now we have a decryptor