INTERVIEW | Dr. Richard Marks and the Sony EyeToy

Henner Thomsen interviews the EyeToy-Inventor Dr. Richard Marks.

The Sony EyeToy camera, introduced in 2003 for the PlayStation 2, was a groundbreaking accessory that revolutionized gaming by allowing players to interact using gestures, movements, and sounds. Acting as a specialized webcam, the EyeToy enabled motion-based gameplay, but only for games designed to support its unique features. Bundled with the popular mini-game collection EyePlay, the EyeToy became an instant hit among gamers. This innovative PlayStation 2 accessory paved the way for future motion-sensing technologies, making it a significant milestone in the evolution of gaming.

Dr. Richard Marks obtained his Ph.D. in Aeronautics and Astronautics in 1995. In 1999, he joined Sony Interactive Entertainment, where he headed the development of the PS2 EyeToy camera, followed by the PlayStation Eye and PlayStation Move, among other projects. 

Stay Forever: With your scientific background, what led you to venture into the gaming industry?

Dr. Marks: Growing up, I played lots of video games. My parents bought a Magnavox Odyssey when I was very young, my best friend had an Atari 2600, and then we got a Mattel Intellivision when I was in middle school. In 1983 my father opened a video game store that I worked at nearly every day when I was in 10th grade. Unfortunate timing :) But while it lasted, it was great; I got to try all the latest systems, games, and controllers. I was always interested in working in the game industry, but I wasn’t sure if it would be a good career choice.

Stay Forever: How did the concept for the EyeToy camera come to you? Many sources suggest that you first saw the PlayStation 2 at the Game Developers Conference in March 1999 and spontaneously conceived the idea for a gaming camera – had this thought perhaps crossed your mind earlier?

Dr. Marks: Before Sony, I was at a startup called Teleos Research that applied computer vision to various applications. I worked on a real-time platform for computer vision research we called AVP-100. I also worked on something we called PeopleTracker for automating control of a pan/tilt/zoom cameras, intended for security and to be a „virtual cameraman“. One of my coworkers and I talked about using the same technology for games someday. Later, when I saw the PlayStation 2 announced, I thought the hardware design was perfect for doing this kind of processing. So I reached out to Sony.

Stay Forever: Were you aware of other gaming cameras at the time? Intel introduced the Me2Cam with a similar concept in February 1999 (though it was launched the following year).

Dr. Marks: I don’t remember being aware, but I probably was at the time, because I kept an eye on gaming. I do remember at some point thinking how beneficial it was to be working on a platform connected to a big TV instead of a small computer monitor. There were also some arcade games that put you „into the game“ as a 2D sprite. But initially at Sony, I focused more on what we called Video as Input, where you didn’t see yourself, you just used your body or a tracked hand-held prop to interact with a game. Later, when we pivoted to create what became EyeToy Play, we made the conscious choice to „build the game around you“, not „stick you into the game“.

Stay Forever: After submitting your idea to Sony, you were hired – but some sources indicate that you were initially not permitted to work on the camera concept. Is that accurate?

Dr. Marks: No, that is not accurate, and I’ve never heard that before! My hiring is an interesting story; Sony invited me to come talk to their newly formed research group. I did not realize they meant it as an interview, but I thought it be great to talk to them, and I shared the ideas I had about using a camera as an input. I also shared ideas about connecting PS2 to Aibo. I thought it was a great conversation, and then they said they wanted me to talk to the other half of the group. At that point, I realized it was an interview :) While everyone thought the ideas about Aibo were interesting, the market size for that would be really small, so there was less support for that. But I was always encouraged to work on the camera ideas.

Stay Forever: During development, did you consider casual gamers who might find traditional controllers too complex, or were you also aiming to create a new experience for core gamers? Was the EyeToy intended to replace or complement the controller?

Dr. Marks: One of the earliest demo videos I made was my 3-year-old son holding a colored ball and moving it around to interact with 3D graphical objects. This talk has lots of background info, including the video of my son at the 13:15 mark. I wouldn’t say we were initially focused on casual gamers, but early on we definitely wanted to make something accessible to everybody. I never intended the camera input to be ONLY for casual games; much of the tech we developed later went into the Move and PS VR, and it definitely enabled some great non-casual experiences.

Stay Forever: You mentioned the EyeToy camera featuring an LED for „retroreflective things“, could you elaborate on that?

Dr. Marks: EyeToy had a blue LED to indicate power and a game-controllable red LED that was much brighter. The field of illumination of the red LED was matched to the field of view of the camera. Retroreflective paint or materials reflect light back in the direction it comes from instead of bouncing normally. The paint of street signs does this. We experimented with tracking red retroreflective props, and it worked great in dim conditions (the object would glow), but in sunlight the LED wasn’t bright enough to outshine the sun, and other red objects or clothing would interfere.

Stay Forever: Did the final system align with your original vision, or were there modifications?

Dr. Marks: I was very happy with EyeToy as a product. Please note that the London studio did all the industrial design and manufacturing related work. I think EyeToy was very well matched to PS2 and was the right product for the time. But we learned a lot, so for PS3 we made a much better device (PS Eye). PS3 had a much higher bandwidth with USB 2.0, so naturally we could have higher resolution and/or framerate. Also, we made the lens much better (<1% pin-cushion distortion at the edges) and added a built-in wide angle. To avoid artifacts that interfere with computer vision algorithms, it could also output uncompressed and even raw video. Many computer vision labs around the world still use PS Eye today (with a PC driver)!

Stay Forever: What was the most significant technical challenge during development? From your perspective, what issues did the final system encounter?

Dr. Marks: Like PCs at the time, the PS2 had USB 1.1, so it could only transfer 8Mbits/sec. Most low-cost webcams at the time streamed a JPEG for every frame, used much of the PC CPU to decompress the images, and 30 frames/sec was considered fast. We didn’t want to use up the CPU or vector units, and I really wanted to match the 60 frames/sec target we’d set for PS2 games. Luckily, we found a great partner in Omnivision, and they adapted their frame compression to output MPEG I-frames instead of JPEG. PS2 had DVD hardware support, so it could decompress the MPEG I-frames in hardware. There weren’t any official USB peripherals, so there was no real USB driver team, so I wrote the USB driver myself (with Omnivision’s support). That was a learning experience :)

Another big technical challenge of course was writing fast image processing code. Since the vector units were generally consumed doing graphics, I decided the CPU was actually a better choice if I wanted developers to be able to integrate the camera effectively. I spent a lot of time optimizing for the PS2 CPU’s cache and custom wide integer instructions, which were very powerful.

But really, the fundamental technical challenge that dominated all others was the extremely variable lighting in people’s homes. Early on I gave a demo of a research prototype during our VP’s keynote at the Game Developer Conference. We locked down all the color-tracking thresholds, and then the lighting changed drastically during the actual presentation. Nothing worked, and I was devastated. After that, I implemented dynamic settings for everything. Later, in the London team’s office, the producer had a sign over his desk that said, „There can be no lighting issues!“

Stay Forever: Why do you think the system was more popular in Europe and parts of Asia compared to the USA?

Dr. Marks: I’ve been asked this many times by different Sony executives, especially marketing and sales :) The answer is complex, and I believe has multiple factors. First, EyeToy Play was made by the London studio, and had certain elements that appealed to that audience (for example, no one in the U.S. knew the Wishi Washi song). Second, in Europe EyeToy was marketed as an experience that needed a camera, while in the U.S. it was marketed as a camera that came with some minigames. Third, the internal accounting for first-party games (at least at the time) made it less advantageous for the U.S. to market it more than other titles. And fourth, I think in the U.S. I don’t think the „now everyone can play games“ message resonated as well. (Later, I think the Wii did a great job overcoming that, and executives at Nintendo have pointed at EyeToy as helping to lay the groundwork for that.) But all of this is speculation on my part.

Stay Forever: Were you also involved in the development of the successor camera, the PS Eye, and the Move system?

Dr. Marks: Yes, I led the development of PS Eye, working closely with the London Studio (in particular Mike Haigh and Ron Festejo). I also led the Move team. Anton Mikhailov was a co-creator of Move, and Eric Larsen joined the team and really improved the tracking.

Stay Forever: Why did these gaming cameras eventually disappear from the market?

Dr. Marks: I think the gaming camera popularity waned along with other related experiences such as Guitar Hero, dancing games, and singing games. I’m not really sure why this has happened. Perhaps the Wii „flooded“ the market with highly accessible games. Perhaps social media took over people’s lives. Mobile gaming was probably a big factor too, and the trend toward online gaming versus couch gaming. Maybe VR has sufficiently filled the physical gaming appetite. Again, probably many factors. 

Stay Forever: I think there’s another factor. Most of the EyeToy games (as well as SingStar, Buzz and Guitar Hero) are not targeting the so-called core gamer, who immerses themselves deeply in a game world and spends hundreds of hours there, but rather casual gamers, like those who play Candy Crush on the phone. This is a large target group, but between Eyetoy and this group lies the effort of game preparation.

Dr. Marks: Yes, friction is a big factor as well, especially for casual games. Mobile games have much less friction than many other options. Also, I believe there was a technology novelty of seeing yourself on TV with special effects, but that technology is commonplace now.

Stay Forever: Is there anything else you would like to share?

Dr. Marks: I have many stories about creating EyeToy, but one of my favorites involves Chris Deering, CEO of PlayStation Europe at the time. At one point, I was alone in a room and he walked in and he looked at me very seriously and asked „Do you really believe camera gaming is going to be a thing?“ I thought about it for a moment and said „Yes, I think so“.
Later, Chris was present during the sensor pricing negotiations with Omnivision. Our team was hoping we’d put in an order for 100k units, but we would have been okay with 50k. Chris asked what the pricing would be for 500k, which made the team internally gasp, then he asked the pricing for a million units. When I heard he decided on 1million, I couldn’t believe, and I admit my first thoughts went to this.

But without that decision from Chris, EyeToy would never have been able to gain the momentum that it did; and it still sold out!

Full disclaimer: I was not actually present at the pricing negotiations, but the story was immediately related to me by the London team, and I relived the shock with them, so I almost feel like I was there. Maybe see this for some background.

I should also mention that Phil Harrison’s belief in me and this concept was a huge part of what made EyeToy possible. It was during his keynote that my demo didn’t work, yet once he took over the PlayStation studios, he assigned a game team and invited me to London to turn the research into a reality. He championed EyeToy from the beginning, and he even coined the name (we were calling it an „interactive toy“ for some legal reasons, which we shortened internally to iToy as a spoof of Apple’s product naming convention, and Phil changed it to EyeToy).

Stay Forever: What aspect of EyeToy are you especially proud of?

Dr. Marks: I have loved witnessing the joy many people have had with EyeToy, and I especially enjoy observing the creativity they have applied when playing it. Examples: I’ve seen an entire family play Wishi Washi at the same time, covering the entire screen with moving limbs, people play Mirror Time with color wristbands to help them, and kids set up pillows on the floor so that can do flying kicks hit the robot in Boxing Chump.

And while I did always hope the interface would enable more people to play games, that abstract thought became poignantly real sometimes. I once received a letter from a mother of a child with severely impaired fine motor skills; she thanked us for making a game that both of her children could play together.

But I want to emphasize that EyeToy was the result of many people’s efforts. I contributed the initial technology, but many other people were crucial in turning it into a successful product. Especially all people in the London studio that worked on the games; the content is really what matters, and they added the „fun“ to the technology and did the heavy lifting of turning prototype into product. It was a joy and an honor to get to work with them.