How Things Work: Microsoft's Sensecam

Credit: Courtesy of Credit: Courtesy of

Human memory is not without faults, and forgetfulness is one such characteristic fault of memory. The idea of storing and implanting memories has existed in science fiction for decades. Philip K. Dick, in his novelette We Can Remember It for You Wholesale, describes a procedure for implanting prefabricated memories in a patient’s mind. While creating and implanting memories today seems a far-fetched concept, recording memories is not new. Man has recorded scenes from his life on cave walls, canvas, celluloid, and, of late, even on silicon.

An average photographer would take a series of time-disconnected pictures or videos while not recording any particular event in its entirety. Therefore, as a recording of memories, this system yields an incomplete documentation of the particular scene. A team at Microsoft Research has attempted to overcome this problem by devising a camera with certain built-in sensors that automatically take pictures of the user’s environment.

This provides a constant stream of pictures, similar to those in a movie reel, only separated by a few seconds in time. The team christened the camera Sensecam.

This is the main input device for MyLifeBits, a project run by Microsoft Research. The goal of this project is to provide a means by which a user can document and browse through his memories using pictures, videos, and other media. In 1945, the American engineer Vannevar Bush proposed a system that would allow the user to sift through media using hyperlinks. He called it Memex, a portmanteau of memory and index. This system was vaguely similar to the World Wide Web as we know it today, and the idea has been adopted for the MyLifeBits project.

According to a paper published by the Microsoft Research team, the Sensecam is fitted with a number of sensors that measure such parameters as body heat and motion, among others. An onboard microcontroller monitors the status of the different sensors and makes decisions regarding the capturing of images. The camera can be set to take pictures every 30 seconds, though it also automatically takes pictures if any of the sensors detect interesting occurrences. For example, a camera around the user’s neck will take pictures every 30 seconds and will also capture on film any person who steps in front of the user.

Existing models are provided with 1GB of memory, allowing the device to store up to 30,000 images captured by its VGA camera. The fish-eye lens allows for a much wider area to be photographed, making the pictures more faithful to images captured by the human eye. The rechargeable battery attached allows the camera to operate for periods longer than 24 hours, assuming it captures an image every 30 seconds.

Capturing memories is only half the objective. The team has also created an application to navigate through the many photos that the camera takes. This software can smooth out pictures that are distorted by the fish-eye lens, edit poor quality pictures, and rapidly display pictures as a time-compressed slide show in a method known as rapid serial visual presentation.
The Sensecam was initially meant to be a camera to help patients suffering from memory ailments. It was meant to be worn by the patient or clipped onto a belt or pocket, from where it can take pictures. The pictures are then stored and reviewed by the patient to recall various memories.

The Sensecam is not currently available for personal use, and only a handful have been made — only for research purposes. The biggest user of the Sensecam is Gordon Bell, a researcher working with Microsoft Research. He has volunteered to be the first test user of the MyLifeBits. Bell carries a Sensecam with him most days, makes a copy of all his bills, and even documents the different sites he visits online.

To date, his “e-memories” exceed 350GB of stored data, and he constantly adds to his bank of memories. When CNN asked Bell if he felt this kind of technology would make humans mentally sluggish, he stated that it was one of the general concerns, but he did not believe that would be the case.

“To me, I feel a lot freer. In a way, I feel like I still remember all that stuff, but I generally remember that [the computer is] remembering something for me so I can find it,” he said.