From ex-Google, deep-learning guru Adam Berenzweig
Adam Berenzweig (pictured above, second from left) built the initial recommendation system for Google Music. In his 10-year tenure there, he also worked on Google Goggles, developing the mobile app that allows users to point a camera at anything in the world, while the AI would try to identify it and offer information relevant to the image. Keep in mind, this was well before deep learning and general object recognition had its legs.
“We could do local feature extraction and fingerprinting, which is great for logos, artworks, posters. [So] it would recognize a movie poster across the street, but it didn’t know the difference between a cat or dog, or a chair versus a table,” says Adam. It was around this time, say 2012, that neural networks were able to take an image and recognise the stuff within it.
I’ve spent most of my career building products around machine learning. I enjoy taking the latest research breakthroughs and figuring out how to apply them in useful, high-impact projects.
After a decade as a Google staffer, Adam was lured by an opportunity to start machine-learning company Clarifai with Matt Zeiler. The aim was to “bring large-scale deep learning into everyday use”. As CTO, Adam built most of the infrastructure around the machinery and also dabbled in the product. “Matt won ImageNet in 2013, and that was right around the time when these models were really proving themselves,” says Adam.
“Having worked in the field for a while and having seen steady, but slow, incremental progress in these kinds of systems, he showed me his early demos and it was clearly a huge cut above — performance wise — and it was obvious that a whole new class of applications would be opened up by that level of performance gain.” After clocking two years with Clarifai, Adam left just a few weeks ago to return to his earlier passion for personalization and recommendation.
“These days, all the data that everyone has and is generating, is spread out across a lot of services. There is so much that we could do if we could concentrate that in one place and allow users to have better direct access to the kind of person-modeling that takes place in the big tech companies,” says Adam. This is what he hopes to tackle next in his career. “Thinking about how to build products that are driven by machine learning in ways that are data-centric, and also present the results of machinery learning in ways that are not confusing.”
In an online masterclass for Hacker Unit, Adam shared with us his insights into better machine learning, as someone who’s kept a keen hand and eye on the industry for the past 13 years.
Adam’s advice for better machine learning
1. “A system for building experiments counts for a lot”
The system in which you build experiments is a critical piece: How quickly are you able to generate experiments by variations on parameters and configuring? How fast can you kick off new experiments and building models without a lot of engineering effort? Then there’s visualizing the results and making it easy to see what’s working quickly, and then decide what’s worth pursuing and what’s not. In my experience, I ended up building things from scratch a lot, but it seems like there are enough people working on these things [now], so the open-source tools are getting better.
2. “Text presentation will impact the volume and value of training data”
When gathering training data from labelers, think about how you present it on a page. Even down to the level of, what size is the font and how many items do you have on a page? This might seem silly and not hardcore machinery stuff, but they actually can have a huge impact on how much training data you get, and how good that training data is.
3. “Give your users the opportunity to correct errors”
These days, so many of the products are built with machine learning inside at some level. So think about how to build the whole product with the following in mind: Is the data that’s coming out of this product going to be good training data? That, I think, is something that should always be on the table. Some examples of how to do it right would be giving users opportunities to correct errors when they occur, and making sure that it’s done in the flow of the product. It should be presented in a way that the user feels like it’s going to provide value, because they are helping to fix the product and helping to improve it for their own benefit.
4. “Find good labelers and develop an ongoing relationship”
My personal experience with Amazon Mechanical Turk was that it was hard to get great labelers, because people come and go too quickly. It actually is important for lots of labelling tasks to allow people to get comfortable with a particular task and basically be trained. We ended up using other systems, other casual workforce systems, to get a set of people that we had an ongoing relationship with. That is important.
5. “A demo on your website captures the imagination”
At the time we raised at Clarifai, we didn’t have a significant amount of revenue. We had a product, we had users on the API, and there was just a belief that this was going to be a big opportunity. The other thing about Clarifai, I think, was that images are so easy to demo, so we just had this demo on our website from the early days, and it immediately shows people what the possibilities are and gets their imagination. In other domains, where either the data is quite specific, let’s say enterprise data within a particular company, or it’s something that’s not as easy to demo visually, I think that is a bigger challenge.
I mean, at the time we raised at Clarifai, we didn’t have a significant amount of revenue. We had a product, we had users on the API, and there was just a belief that this was going to be a big opportunity.
6. “Knowing how to wrangle the GPUs is important”
We were able to get a lot of mileage from a single GPU. This might change over the next year or two, but the multi-GPU system, the general problem of how best to optimize training across multiple GPUs, I don’t think has been solved. As a result, most people don’t need to be thinking about it. I think you can get a lot of work out of the same GPU, in which case the scalability is really just about being able to move data around easily during training, and then it’s getting a long-term service.
7. “Open-source stuff is not to be overlooked”
The open-source stuff has a really strong community around it, and it is improving very fast. Take a look and see if it makes sense to switch over. I think that’s probably very dependent on the domain. Maybe also the highest order bit on that one is, what are the skills of the team? And in what languages are people used to writing in? Are they really familiar with some of these frameworks? I think they are all pretty good.
Clarifai, in the beginning, had a lot going for it, but one criticism we got was that we were trying to do so many things at once.
8. “Take on custom work if it’s a direction you want to tackle”
We always took the approach we wanted. We would do custom work for a client if it was in a direction that we wanted to go in anyway; if we thought it was going to be a useful thing to have for other reasons. Like, if somebody wanted a food-specific model or something that we thought, “That’s a great domain, we ought to have that”. Then we would do that. Sometimes, we would charge for it and that was a way to justify spending the extra engineering time. Sometimes we wouldn’t, it was just basically part of the deal process. It really depended on the customer size.
9. “Find what works for you, then find the right investor to match”
There’s a list of things that will work, and you need a couple of them. It can be a team with a track record; it can just be a technology that nobody else has; it can be really great product feel, execution skills around a demo, vision and imagination; and seeing new opportunities that have not been pursued. Clarifai, in the beginning, had a lot going for it, but one criticism we got was that we were trying to do so many things at once. We were building the API, we also had the photo app. We pulled it off, but some investors turned us away because of that. The key thing here is finding what works for you, and finding the right investor match.
Have you got a top tip for improving deep-learning systems? Share it with us!