Verifying Signatures of the Query Parameters

Recently I have spent few hours trying to understand why the verification of the digital signature of the parameter, which was sent as query parameter on the url, fails.

I put two logging messages on the parameter: one at request as it arrives, before any parsing is done at all and one just before the signature verification.

When verifying the first output, the verification was successful, but the second one was failing. "What a voodoo!" - thought I, and decided to compare this string in text edit. You know to take one string, type "Search" and try to find the second one. It succeeded. "Voodoo!" - thought I again. How the strings can be identical in a text editors and have a different signature?

If you want to think about it, just stop reading here and think.

Personally I started thinking about some hidden characters and so on... But the answer was much simple: Text editor compare text ignoring the case. It's so obvious that we usually don't think about it. Some (very few) of the characters of the text had a different case and that's why the text editor found them, but the signature failed.

However, another question was left: How happened that the query parameter, which was sent on the url, changed the case of some of its characters? The answer is a bit tricky. The parameter that is sent on the url should be url-encoded. The sender was encoding the url before creating a digital signature. The receiver was aware of this functionality and moreover he was also aware that the servlet container decodes the parameters as it receives them. So the receiver was encoding the parameter back to url-encoded format before validating the signature and everything worked.
Until one day came another sender, who did exactly the same logic, but who used a different library for the url encoding. He used lower case characters for the encoded values instead of the upper case characters that are usually used by most of the libraries.

So the scenario for the incorrect behavior is clear. And where is the defect?
The defect is in the receiver's part of code. Receiver should never get parameters from the servlet container, but get the whole query string before it's going to be parsed and do the parsing by himself. It's incorrect to do the parsing and then to correct the parsing errors by url-encoding the parameter back to its original state. The parameter should be verified as is.