Twitter this week announced it would be changing how image cropping works on its website after concerns that the machine learning-based algorithm the company was using was racially biased in how it cropped images, in particular by favoring white faces over Black ones.
In a post diving into the issue, Parag Agrawal, the company’s chief technology officer, and design chief Dantley Davis explained how the company tested the model for racial or gender bias before it implemented the system. But Twitter didn’t publish how it had done those tests at the time so that external analysis could be performed, in what the company calls “an oversight.”
To fix that, Twitter is “currently conducting additional analysis to add further rigor to our testing, are committed to sharing our findings, and are exploring ways to open-source our analysis so that others can help keep us accountable,” the duo writes.
The company is also working on more concrete changes to Twitter to ensure it shows images the way users intend. Specifically, Twitter is promising to focus less on using machine learning-based techniques for image cropping, and instead will develop tools that show users what the images in their posts will look like when they compose a tweet. The company also says it’s already started experimenting with new options for image cropping and previews to give users more control.
Going forward, Twitter promises to enact a “what you see is what you get” policy, so that whatever photo you attach when writing a tweet is what you (and other users) will see when viewing it on Twitter. There are some edge cases that Twitter will have to work out, like very tall or very wide photos, but the goal is to make sure users will know right from the start how their images will be displayed on the site.
The company switched to its current machine learning-based cropping system back in 2018, which uses a neural network to crop image previews based on “saliency.” Unlike the company’s previous system, which was built to focus on faces, the current model is designed to focus on what the algorithm determines is “the most interesting part of the image” by predicting where an average person would first look when viewing the full image.
There’s no word yet as to when these changes to the image cropping system will roll out on Twitter.
