Recovering Censored Text Using Photoshop and JavaScript
My friend Andrew recently posted a teaser for a new project he’s working on, but with part of the headline pixelated to obscure what the project actually is. My curiosity got the best of me and I decided to do what any self-respecting geek would do: write a program to figure out what the censored text said.
Ultimately I failed to recover most of the censored text (except “to”), so I had to cheat a little. The following video is the program running on a very similar image I created. This proves it works in ideal conditions, but needs some improvement to work in less than ideal cases.
(and no, as far as I know my friend’s project has nothing to do with eating monkeys)
Applying a filter like Photoshop’s “mosaic” filter obscures the original data, but doesn’t remove it entirely. If we can reconstruct an image with known text that looks very similar to original image, then we can be pretty sure the original text is the same as our known text. This is very similar in principle to brute-force cracking a password hash. For a more detailed explanation see this article.
Photoshop was an obvious choice since I needed to recreate the exact same fancy styling as the original image, then apply the exact same mosaic filter. I figured I would have to write a script that tells Photoshop to generate images, then use an external tool to actually compare them to the original.
It turns out that Photoshop CS3 has all the features necessary to pull the whole thing off without any other programs or tools. The most important feature is the JavaScript scripting environment built into Photoshop, which is far more powerful than the AppleScript environment (and a much nicer language, in my opinion).
CS3 added two other features that are critical to this task: Smart Filters, and Measurements. Smart Filters lets you edit a layer (namely the text with effects applied) after you apply a filter that would have previously require rasterization. This lets us apply the censoring filter to our styled text, and later change the text without having to manually reapply the filter. The “measurements” feature lets you record various statistics about an image or portion of an image: in our case we’ll want the “average gray value” of the “difference” between the original and generated images.
First we need to prepare the environment. Open the original image in Photoshop, and attempt to replicate the original un-censored text as closely as possible (you need some uncensored text as a reference). Place your text layer on top of the original and toggle between normal and “difference” blending modes to see how you close you are. Ideally everything will be black in “difference” mode. It’s very important to precisely match the font, size, spacing, color, effects like drop shadows or outlines, and even the background. If these are off even by a little bit it will throw things off. I ended up having to cheat because I couldn’t match the slick styling of the original text with my lame Photoshop design skills.
Once the text matches and is lined up perfectly, select the layer then choose “Convert for Smart Filters” from the “Filter” menu. Now select the censored portion of the text and apply the same filter used on the original image, again matching it as closely as possible. For the mosaic filter, you can line up the “grid” by adjusting the origin and size of the selection (yeah, it’s a pain).
Finally, make sure your layer is on top of the original, and the blending mode on your layer is set to “difference”. Double-click the Smart Object layer to open it’s source document, and adjust the variables listed at the top of the JavaScript to match the names and layers. Also, in the menu “Analysis”: “Select Data Points”: “Custom…” make sure only “Gray Value (Mean)” is checked.
Code
Rather than attempting to explain it in detail here, just read the code and comments. Here’s a quick summary:
- Start with the first character. Try setting it to each of the possibilities (a through z, and a space), and record the difference score between the original image and generated image. Only look at the first half of the current character (since the second half will be influenced by the *next* character).
- Sort the results. Lower scores are better (less different)
- Now try each of the top 3 characters along with every possibility for the *next* character. This time record score for the whole width of the current character since we’re checking the next character as well.
- Pick the best choice, either the best permutation out of all 81 combinations (3 best * 27 possible), or out of the 3 averages for each best.
- Repeat for the next character until done.
baseDocName = "base.psd";
baseDocTextLayer = 0;
textDocName = "The easy way to do somethingss12.psb";
textDocTextLayer = 0;
knownString = "The easy way "; // the part of the string that’s already known
missingLength = 20; // number of characters to figure out
method = 3;
debug = false;
function main()
{
baseDoc = documents[baseDocName];
textDoc = documents[textDocName];
// get the top left corner of the text layer in the main doc
var mainBounds = baseDoc.artLayers[baseDocTextLayer].bounds,
mainX = mainBounds[0].as("px"),
mainY = mainBounds[1].as("px");
// possible characters include space and lowercase.
var possibleCharacters = [" "];
for (var i = 0; i < 26; i++)
{
possibleCharacters.push(String.fromCharCode("a".charCodeAt(0) + i));
//possibleCharacters.push(String.fromCharCode("A".charCodeAt(0) + i)); // uncomment for uppercase letters
}
var fudgeFactor = 3, // number of top choices to try
guess = ""; // guessed letters so far
for (var charNum = 0; charNum < missingLength; charNum++)
{
results = [];
// get the beginning and potential end (width of a "M") of the next character
var w1 = getStringBounds(knownString + guess),
w2 = getStringBounds(knownString + guess + "M");
// PASS 1: half the potential width, since we’re not looking at the next character yet
// half the width of "M"
setSelection(mainX, mainY, (w1[2].as("px") + w2[2].as("px")) / 2, 15);//w2[3].as("px"));
// get the score for every letter
for (var i = 0; i < possibleCharacters.length; i++)
{
var val = getStringScore(knownString + guess + possibleCharacters[i])
var res = { ch: possibleCharacters[i], v: val };
results.push(res);
}
// sort from best (lowest) to worst score
results = results.sort(function (a,b) { return a.v – b.v; });
// method 1: too simple, poor results
if (method == 1)
{
guess += results[0].ch;
}
else
{
// PASS 2: full (potential) width of the current character, testing each of the few top matches and every possible next character
// full width of "M"
setSelection(mainX, mainY, w2[2].as("px"), 15);//w2[3].as("px"));
var minValue = Number.MAX_VALUE,
minChar = null,
minSum = Number.MAX_VALUE,
minSumChar = null;
// try the few best from the first pass
for (var i = 0; i < fudgeFactor; i++)
{
var sum = 0;
for (var j = 0; j < possibleCharacters.length; j++)
{
// get the score for the potential best PLUS each possible next character
var val = getStringScore(knownString + guess + results[i].ch + possibleCharacters[j])
sum += val;
if (val < minValue)
{
minValue = val;
minChar = results[i].ch;
}
}
if (sum < minSum)
{
minSum = sum;
minSumChar = results[i].ch;
}
}
// if the results aren’t consistent let us know
if (debug && results[0].ch != minSumChar || minChar != minSumChar)
alert(minChar + "," + minSumChar + " (" +results[0].ch + "," + results[1].ch+ "," + results[2].ch+ ")");
if (method == 2)
{
// method 2: best of all permutations
guess += minChar;
}
else
{
// method 3: best average
guess += minSumChar;
}
}
WaitForRedraw();
}
}
// measure the gray value mean in the current selection
function getMeasurement()
{
// delete existing measurements
app.measurementLog.deleteMeasurements();
// record new measurement
app.activeDocument = baseDoc;
app.activeDocument.recordMeasurements();//MeasurementSource.MEASURESELECTION, ["GrayValueMean"]);
// export measurements to a file
var f = new File ("/tmp/crack-tmp-file.txt");
app.measurementLog.exportMeasurements(f);//, MeasurementRange.ACTIVEMEASUREMENTS, ["GrayValueMean"]);
// open the file, read, and parse
f.open();
var line = f.read();
var matches = line.match(/[0-9]+(\.[0-9]+)?/);
if (matches)
{
var val = parseFloat(matches[0]);
return val;
}
return null;
}
// sets the value of the test string
function setString(string)
{
app.activeDocument = textDoc;
app.activeDocument.artLayers[textDocTextLayer].textItem.contents = string;
WaitForRedraw();
}
// gets the difference between the original and test strings in the currently selected area
function getStringScore(string)
{
setString(string);
// save document to propagate changes parent of smart object
app.activeDocument = textDoc;
app.activeDocument.save();
// return the average gray value
return getMeasurement();
}
// get the bounds of the text
function getStringBounds(string)
{
app.activeDocument = textDoc;
// set the string of the text document
setString(string);
// select top left pixel. change this if it’s not empty
app.activeDocument.selection.select([[0,0], [0,1], [1,1], [1,0]]);
// select similar pixels (i.e. everything that’s not text)
app.activeDocument.selection.similar(1, false);
// invert selection to get just the text
app.activeDocument.selection.invert();
// return the bounds of the resulting selection
return app.activeDocument.selection.bounds;
}
// sets the base document’s selection to the given rectange
function setSelection(x, y, w, h)
{
app.activeDocument = baseDoc;
app.activeDocument.selection.select([[x,y], [x,y+h], [x+w,y+h], [x+w,y]]);
}
// pauses for Photoshop to redraw. taken from reference docs.
function WaitForRedraw()
{
// return; // uncomment for slight speed boost
var eventWait = charIDToTypeID("Wait")
var enumRedrawComplete = charIDToTypeID("RdCm")
var typeState = charIDToTypeID("Stte")
var keyState = charIDToTypeID("Stte")
var desc = new ActionDescriptor()
desc.putEnumerated(keyState, typeState, enumRedrawComplete)
executeAction(eventWait, desc, DialogModes.NO)
}
main();
The raw code and sample Photoshop file are available on GitHub.
Issues
This problem is particularly tricky for proportional fonts, since if you get any character wrong and it’s width is different than the actual character, then all subsequent characters will be misaligned, causing more incorrect guesses, compounding the problem even more, and so on. I’m not sure how to deal with this, other than improving the overall matching quality. Ideally we would test every possible combination for the entire string, but that would require 27^n tests, where n is the number of unknown characters. This is obviously not feasible.
With the simplistic method of iterating over each position and trying each possible character, it turned out that almost every single “guess” was for the letters “m” or “w”. This was because for positions where the original was narrower characters, the “m” would “bleed” over into the next position, improving the score regardless of how well it actually matched the current character. To get around this, we only look at the difference for the first half of the character’s position.
Since looking at the first half of the character removes some valuable information, we then do a second pass using the top several guesses from the first pass, this time looking at the full width of the current character along with each of the possible next characters (27 tests + 3 runs times 27 tests results in 108 tests per character).
Further improvements could definitely be made, but I’ve already spent several hours too many on this.
The current algorithm runs at about 3 characters per minute. The overhead of Photoshop saving the Smart Object document on every individual test case is significant. If this were a special purpose program manipulating images directly it would likely be much faster. The tradeoff, of course, is you have all of Photoshop’s flexibility at your disposal for matching the original document’s font, size, style, spacing, and censoring effects, which is very important. For small amounts of text speed isn’t a problem.
Conclusion
While my original goal of recovering the censored text on my friend’s page was never achieved, the project was a success. It works well on my test image, and I learned about 3 obscure but cool and useful features of Photoshop!
Oh, and that’s why ██████████ uses black ink to ██████ their ██████!