A webhook in VFP loses its signature with a Polish name

The symptom

My licensing portal takes payments through a payment gateway. When someone pays, the gateway calls a webhook on my backend, which is Visual FoxPro 9 running under FoxServer, and that webhook issues the license and sends the email. Every call is signed with an HMAC-SHA256 of the request body, computed with a secret that only the gateway and I know. If the signature does not match, the event is discarded and a line is left in the log: firma invalida.

The server logs had four of those lines in three weeks. They could have been tests from the gateway’s simulator or retries outside the five-minute window. But I was designing a second webhook, the Patreon one, which also signs over the body, and before reusing the same path I wanted to confirm it was correct.

The hypothesis

The body of an HTTP request arrives as bytes. FoxServer decodes it as UTF-8 into a .NET string and hands it to Visual FoxPro over COM. When crossing COM into VFP, that string is converted to the machine’s ANSI code page, CP1252, and whatever does not fit in CP1252 is replaced with another character. The helper that verifies the signature, also in .NET, gets back what VFP passes it, encodes it as UTF-8 again and computes the HMAC over that.

With José Muñoz nothing happens, because é and ñ are in CP1252 and the bytes come back unchanged. With Şahin, the Ş is not in CP1252, the bytes are no longer the ones the gateway signed and the HMAC cannot match. Before changing anything, I wanted to measure it.

The measurement

I wrote a test bench with FoxProof, my test runner for VFP, without starting the server. The two COM crossings are reproduced with ADODB.Stream: reading UTF-8 bytes and returning a BSTR is equivalent to what the host does when it delivers the body, and receiving a VFP string as a BSTR and writing it as UTF-8 is equivalent to what the helper does when it signs. The reference HMAC is computed by bcrypt.dll through DECLARE, checked beforehand against the published test vectors. The last group of tests calls the real .NET helper, the same one that runs in production, with a signature computed over the original bytes.

FUNCTION HmacHex(lcAlg, lcKey, lcData)
    LOCAL lnAlg, lnHash, lnRet, lcOut, lnLen
    DECLARE INTEGER BCryptOpenAlgorithmProvider IN bcrypt.dll ;
        INTEGER @hAlg, STRING pszAlgId, INTEGER pszImpl, INTEGER dwFlags
    DECLARE INTEGER BCryptCreateHash IN bcrypt.dll ;
        INTEGER hAlg, INTEGER @hHash, INTEGER pbHashObject, INTEGER cbHashObject, ;
        STRING pbSecret, INTEGER cbSecret, INTEGER dwFlags
    DECLARE INTEGER BCryptHashData IN bcrypt.dll ;
        INTEGER hHash, STRING pbInput, INTEGER cbInput, INTEGER dwFlags
    DECLARE INTEGER BCryptFinishHash IN bcrypt.dll ;
        INTEGER hHash, STRING @pbOutput, INTEGER cbOutput, INTEGER dwFlags

    lnAlg = 0
    lnHash = 0
    * BCRYPT_ALG_HANDLE_HMAC_FLAG = 8; the algorithm name goes in UTF-16
    lnRet = BCryptOpenAlgorithmProvider(@lnAlg, STRCONV(lcAlg + CHR(0), 5), 0, 8)
    lnRet = BCryptCreateHash(lnAlg, @lnHash, 0, 0, lcKey, LEN(lcKey), 0)
    lnLen = IIF(UPPER(lcAlg) == "MD5", 16, 32)
    lcOut = REPLICATE(CHR(0), lnLen)
    lnRet = BCryptHashData(lnHash, lcData, LEN(lcData), 0)
    lnRet = BCryptFinishHash(lnHash, @lcOut, lnLen, 0)
    RETURN LOWER(STRCONV(lcOut, 15))
ENDFUNC

Inside the test the payloads are written in hexadecimal, because the .prg itself is CP1252 and the characters I wanted to test cannot be written in it.

The result, with 14 tests:

The hypothesis held, and the measurement places the loss where I expected. The host decodes the UTF-8 correctly; the characters are lost in the conversion from Unicode to CP1252 when entering VFP, before any controller touches the body. Two of the replacements are Windows “best fit” mappings, which turn Ş into S and Ł into L, so the resulting string looks correct and nothing in the log shows that a character has changed.

Whatever fits in CP1252 arrives intact, the euro sign included. A Spanish, German or Swedish customer would never have noticed. A Turkish, Polish, Czech or Vietnamese customer, or anyone with an emoji in their name, paid and was left without a license.

The fix

Passing the body from VFP to the helper in Base64 would not help, because by the time VFP has it the characters are already gone. The fix had to be in the host. FoxServer 0.9.25 keeps the body bytes as they arrive and also hands them to VFP in Base64, which crosses COM without any conversion. In the controller it looks like this:

lcPayload = req.GetRawBody()           && the text, to read it
lcSig     = req.GetHeader("X-Signature", "")
TRY
    lcPayloadB64 = req.GetRawBodyBase64()  && the original bytes, to sign them
CATCH
    lcPayloadB64 = ""                      && older host or core: sign the text
ENDTRY
llOk = loPaymentSvc.VerifySignature(lcPayload, lcSig, lcPayloadB64)

The .NET helper decodes the Base64 and computes the HMAC over those bytes. GetRawBody() stays the same, because to read the content the CP1252 text is what a controller needs; only the signature needs the original bytes.

I wanted to avoid breaking anything that already worked. The host makes the new call through a TryInvoke, which returns false without raising an error if the project has an older core without that method, so an old project keeps working as before. And the controller falls back to signing the text if the Base64 arrives empty, so the portal DLL also works with an older host.

The tests document the loss

The host test suite now has 16 cases that send those four payloads to a test endpoint and check two things: that GetRawBodyBase64() returns the same bytes the client sent, and that GetRawBody(), the text, still loses them in the two cases that do not fit in CP1252. The second check records the reason the first one exists, for whoever reads the tests a year from now.

The FoxProof bench went from 10 green tests and 4 red ones to 26 green. The four red tests became tests that document the loss through the text path, and new groups were added to check that the Base64 path verifies every case, emoji included.

What I learned

• An HMAC signature is computed over bytes. Any code page conversion between the network and the code that verifies it changes the signature, even when you cannot see it. If you receive signed webhooks in VFP, this will happen to you.
• A “best fit” replacement is harder to spot than a question mark: a ? in a name is obvious, an S in place of an Ş is not.
• It pays to measure a path before reusing it. The Patreon webhook would have had the same fault and, because it returns 401 when the signature does not match, Patreon would have retried those events again and again.
• A test that documents a limitation is as useful as one that proves a fix, because it explains why there are two methods where one seemed enough.

The fix went to the server that same day. The next post covers the Patreon webhook, which notifies my VFP backend every time a patron joins, leaves or changes tier.