Back in 2010, a team from Stanford University’s computer graphics lab got their hands on a Nokia N900. It had a pretty good camera by smartphone standards at the time, but the researchers thought they could make it better with a little bit of code.
The Stanford team, led by professor Mark Levoy, was working on the cutting edge of a nascent field known as computational photography. The theory was that software algorithms could do more than dutifully process photos, but actually make photos better in the process.
“The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera,” is how the group described its efforts at the time.
Fast forward to today, and many of the techniques that Levoy and his team worked on — yielding features like HDR and better photos in low light — are now commonplace. And in Cupertino, Calif,. on Tuesday, Apple’s iPhone event was another reminder of just how far smartphone technology has come.
What we think of as a camera is largely a collection of software algorithms that expands with each passing year.
Take Portrait Lighting, a feature new to the iPhone 8 Plus and iPhone X. Apple says it brings “brings dramatic studio lighting effects to iPhone.” And it’s all done in software, of course. Here’s how an Apple press release describes it:
“It uses the dual cameras and the Apple-designed image signal processor to recognize the scene, create a depth map and separate the subject from the background. Machine learning is then used to create facial landmarks and add lighting over contours of the face, all happening in real time.”
In other words, Apple is combining techniques used in augmented reality and facial recognition to create a photo that, to paraphrase the Stanford team, no traditional camera could take. On the iPhone X, the company is also using its facial recognition camera system, which can sense depth, to do similar tricks.
While the underlying techniques behind many of these features aren’t necessarily new, faster and more capable processors have made it feasible to do them on a phone. (Apple says its new phones even have a dedicated chip for machine learning tasks.)