Computer Vision

Imagine a friend has put together a photo album, but hasn’t labelled the pictures. As you flip through the images, you’re left guessing where they were taken. You might make some fairly accurate estimates – but a new U of T computer program would probably do much better.

Computer science professor Aaron Hertzmann, PhD student Evangelos Kalogerakis and others have developed an algorithm that can analyze a series of photos and determine where they were taken. The program – the first of its kind, Hertzmann says – isn’t designed to identify individual images, although it can make a rough guess at one-off photos. Rather, its power lies in its ability to identify a whole series of images, if it knows the sequence in which they were taken.

Hertzmann’s program exploits the enormous image database of the popular photo-sharing website, Flickr.com. On Flickr, people have the option of “tagging” photos to indicate when and where they were taken. Hertzmann’s program uses these tags – mini-summaries of the photo-taking habits of thousands of Flickr users – to determine where photos were taken.

“If you take a picture of some city street, it could be anywhere in the world,” Hertzmann says. “But if half an hour later you take a picture of Big Ben or the Eiffel Tower, then it becomes much clearer where that first picture was taken.”

On a recent holiday, Hertzmann photographed the Odeon of Herodes Atticus, an ancient Greek amphitheatre in Athens. A day later he photographed a nearly featureless seascape off Santorini, with just a sliver of coastline showing. Taken individually, the program does just fine with the Odeon photo because many people have posted similar shots to Flickr; it puts a dot on a world map at Athens. But the seascape could be anywhere. Without any additional information, the program puts dots all over the world’s seas. Feed the program both photos – along with the time frame in which they were shot – and suddenly it recognizes the seascape as being from the Aegean and puts a dot near Santorini. (The program knows not only that Santorini is close to Athens, but also that it’s a popular target for Flickr photographers who have recently visited Athens.)

Hertzmann imagines many practical uses for the program. Tourism workers and urban planners could benefit from a detailed knowledge of the routes that people take between – and within – cities. And health-care professionals could use it to help fight epidemics, especially if the photo-taking data is combined with information such as air travel routes and cellphone usage.