My friend Andrew recently posted a teaser for a new project he’s working on, but with part of the headline pixelated to obscure what the project actually is. My curiosity got the best of me and I decided to do what any self-respecting geek would do: write a program to figure out what the censored text said.
Ultimately I failed to recover most of the censored text (except “to”), so I had to cheat a little. The following video is the program running on a very similar image I created. This proves it works in ideal conditions, but needs some improvement to work in less than ideal cases.
(and no, as far as I know my friend’s project has nothing to do with eating monkeys)
Applying a filter like Photoshop’s “mosaic” filter obscures the original data, but doesn’t remove it entirely. If we can reconstruct an image with *known* text that looks very similar to original image, then we can be pretty sure the original text is the same as our known text. This is very similar in principle to brute-force cracking a password hash. For a more detailed explanation see this article.
Photoshop was an obvious choice since I needed to recreate the exact same fancy styling as the original image, then apply the exact same mosaic filter. I figured I would have to write a script that tells Photoshop to generate images, then use an external tool to actually compare them to the original.
CS3 added two other features that are critical to this task: Smart Filters, and Measurements. Smart Filters lets you edit a layer (namely the text with effects applied) *after* you apply a filter that would have previously require rasterization. This lets us apply the censoring filter to our styled text, and later change the text without having to manually reapply the filter. The “measurements” feature lets you record various statistics about an image or portion of an image: in our case we’ll want the “average gray value” of the “difference” between the original and generated images.
First we need to prepare the environment. Open the original image in Photoshop, and attempt to replicate the original un-censored text as closely as possible (you need *some* uncensored text as a reference). Place your text layer on top of the original and toggle between normal and “difference” blending modes to see how you close you are. Ideally everything will be black in “difference” mode. It’s very important to precisely match the font, size, spacing, color, effects like drop shadows or outlines, and even the background. If these are off even by a little bit it will throw things off. I ended up having to cheat because I couldn’t match the slick styling of the original text with my lame Photoshop design skills.
Once the text matches and is lined up perfectly, select the layer then choose “Convert for Smart Filters” from the “Filter” menu. Now select the censored portion of the text and apply the same filter used on the original image, again matching it as closely as possible. For the mosaic filter, you can line up the “grid” by adjusting the origin and size of the selection (yeah, it’s a pain).
Rather than attempting to explain it in detail here, just read the code and comments. Here’s a quick summary:
- Start with the first character. Try setting it to each of the possibilities (a through z, and a space), and record the difference score between the original image and generated image. Only look at the first half of the current character (since the second half will be influenced by the *next* character).
- Sort the results. Lower scores are better (less different)
- Now try each of the top 3 characters along with every possibility for the *next* character. This time record score for the whole width of the current character since we’re checking the next character as well.
- Pick the best choice, either the best permutation out of all 81 combinations (3 best * 27 possible), or out of the 3 averages for each best.
- Repeat for the next character until done.
The raw code and sample Photoshop file are available on GitHub.
This problem is particularly tricky for proportional fonts, since if you get any character wrong and it’s width is different than the actual character, then all subsequent characters will be misaligned, causing more incorrect guesses, compounding the problem even more, and so on. I’m not sure how to deal with this, other than improving the overall matching quality. Ideally we would test every possible combination for the entire string, but that would require 27^n tests, where n is the number of unknown characters. This is obviously not feasible.
With the simplistic method of iterating over each position and trying each possible character, it turned out that almost every single “guess” was for the letters “m” or “w”. This was because for positions where the original was narrower characters, the “m” would “bleed” over into the *next* position, improving the score regardless of how well it actually matched the current character. To get around this, we only look at the difference for the first *half* of the character’s position.
Since looking at the first half of the character removes some valuable information, we then do a second pass using the top several guesses from the first pass, this time looking at the full width of the current character along with each of the possible next characters (27 tests + 3 runs times 27 tests results in 108 tests per character).
Further improvements could definitely be made, but I’ve already spent several hours too many on this.
The current algorithm runs at about 3 characters per minute. The overhead of Photoshop saving the Smart Object document on every individual test case is significant. If this were a special purpose program manipulating images directly it would likely be much faster. The tradeoff, of course, is you have all of Photoshop’s flexibility at your disposal for matching the original document’s font, size, style, spacing, and censoring effects, which is very important. For small amounts of text speed isn’t a problem.
While my original goal of recovering the censored text on my friend’s page was never achieved, the project was a success. It works well on my test image, and I learned about 3 obscure but cool and useful features of Photoshop!
Oh, and *that’s* why ██████████ uses black ink to ██████ their ██████!