Every quarter, the tech team at Shutterstock holds CodeRage, a 24-hour hackathon where they're encouraged to work on any project that can bring value to the company.
This quarter, one of the winning projects was called Projector. It's a web app that lets you turn your webcam into a projector for sharing drawings and diagrams with people in other locations. Here’s a quick demo of how it works.
Dave K, the lead engineer on our footage team, wrote the app, and we were so impressed, we decided to grill him for more info on how it came about.
What problem were you trying to solve?
Dave: I was onboarding one of the new developers in our Denver office, and I was in New York. I wanted to show him how our different systems were set up and how they work together. I thought, "Wow, every time I onboard somebody, I usually go to a whiteboard and sketch out how this works, because it’s so hard to talk about it without diagrams."
I couldn’t find any good online solutions for drawing with a touchpad, and if you point a webcam at a whiteboard, it's really hard for the person on the other side to see anything. So, a few days later, I was thinking it would be cool if we could use a webcam to show a facsimile of a piece of paper you have in front of you to someone across the country, and be able to make changes in real time.
How did you approach the problem?
For this project, the main problem was how to detect where the piece of paper was. So I brainstormed a bit about that. But sometimes it doesn't work out the way you expect, and this was a perfect example — the initial plan I had didn’t work at all, and I had to re-formulate it and sleep on it to find a better solution.
What was the first approach you tried?
Well, I needed to detect the edges of the piece of paper. Originally, I was going to have a setup where you made four black dots on a piece of paper. Then, I was going to try converting the webcam image to black and white, and then detect every shape on the page. Any shape that was touching the corner of the image, I’d delete from the shapes that I’m looking at. And then I’d try to discover the shapes that were closest to the corners of the frame, because those would probably be my black reference dots.
Part of trying to solve that problem was to write a fill algorithm, so I created a structure of every black pixel on the page, and then I’d loop through the pixels and try to determine if they were part of a bigger shape based on neighboring pixels. It worked on a small scale — like a 10×10 pixel image — but on a bigger image, I was getting a stack overflow. It was just using too much memory.
Then I thought, "Wait. Why make these dots black? What happens if we add a color in there?" You could do a simple color detection based on different quadrants of the image. I felt a little better going to sleep with that idea in my head. The next morning, I woke up and just focused on that, and it seemed to work pretty well.
What third-party libraries did you use for the project?
A lot of this is reliant on these new, awesome features available in HTML5. One of the things that was really crucial was the getUserMedia() HTML5 function. That lets the browser get access to your webcam. Then I used some of the canvas manipulation tools. HTML5 lets you draw an image to the canvas, and then you can get RGBA values for every pixel on that canvas, so you can determine what color something is. You basically have an array you can loop through, and that’s how I'm able to find the green pixels.
I used ImageMagick for server-side image processing, and ran a threshold function on the image so everything that fell into the lighter 50% of a black and white image turned to white, and everything in the darker 50% turned to black. That makes it easier to create a facsimile of the image. The place I got the idea for that was from an app called JotNot, which lets you scan documents by using the camera on your smartphone. It uses a similar approach to thresholds to make the scanned text clearer.
The other thing I used ImageMagick for is for perspective distortion. What’s nice is that if I’m holding a piece of paper, no matter how I’m holding it, it keeps it in place so it doesn’t jump around. It also makes it so that it doesn’t look squashed.
Have you thought of open sourcing this project?
Yeah, I have to clean it up a bit and make it a little more practical to use, but then I think we could release it!
For a more in-depth look at this project, read a longer version of this interview on our tech blog, shutterbits. While you're there, sign up for email updates to find out when Projector will be available and other tech news.