The Center for Data Innovation spoke with Andrea Burbank, search and data mining engineer at Pinterest. Burbank discussed challenges in managing billions of data points and how analyzing user data can improve the user experience.
Josh New: You were named one of Fortune’s “2014 Big Data All Stars.” To what do you attribute your success in the field?
Andrea Burbank: I’m fortunate to be doing something I love and that happens to align well with the rapidly expanding availability of data and computing resources. It hasn’t always been the case that a business magazine would be interested in data scientists. We had a job shadow day when I was in eighth grade and no one knew what to do with me when I said I wanted to be a statistician; I ended up instead shadowing a curator at the military museum (which, as it turned out, was not my calling).
Today I have a dream job where I get to use statistics and analytical tools to drive product direction and understand user engagement.
New: Pinterest has millions of users and billions of interactions. While having an abundance of data is no doubt useful, how do you ensure that such vast quantities of data are put to good use and not wasted?
Burbank: We actually dedicate a fair amount of time at the outset of a project to ensure that we know which interactions are important and what we’d like to learn. We actively monitor what we’re logging and proactively remove logs we think aren’t important so we’re only left with useful data.
That still leaves billions of interactions. Each night we sort those interactions into trends to help us understand how the product is doing. We use individual interactions to improve our recommendation and search algorithms and to understand user behavior better so that we can improve the service as a whole, especially the discovery engine at its core. As we’ve grown, we’re increasingly able to use historical data that we might not have looked at before to understand large-scale trends on the service and come up with new ideas for the product.
New: You’ve talked about the value of A/B testing in deriving data-driven insights to improve user experiences. Were there any challenges in getting Pinterest to make this kind of testing the norm?
Burbank: Yes, there were definitely challenges. Although Pinterest has always been a data-driven company, we’ve also prided ourselves on the ability to move quickly. For a long time the rhetoric around new features focused on “shipping to 100 percent of users” rather than launching an experiment. It wasn’t until we’d proven the value of learning from some high-profile experiments that leadership across the company started to demand better understanding of the impact of new features. About two years ago, we’d undertaken a complete rewrite of the website, and I managed to convince the product manager in charge of the rollout to run a controlled test to measure any differences in user behavior between the new and old versions of the website. When we identified major differences in the behavior of the new and old sites, we were able to fix bugs and add missing functionality before the full rollout. We discovered features that we hadn’t thought were important were actually driving engagement, so we added those features back rather than inadvertently making users unhappy with their disappearance. This experience, which the whole company was heavily focused on, demonstrated the value of A/B testing and had a big impact on the company’s mentality around running experiments to understand changes in the service.
New: Of the improvements to Pinterest made through analysis of user data, what are you most proud of?
Burbank: When people visit Pinterest, the first thing they see is a feed of Pins called the home feed. One of our analyses of user behavior showed that most people really didn’t understand how to customize their home feed to see the content they were interested in. We found that most users did some customization on the day they signed up, but then didn’t refine their settings as their interests changed (for example, after they finished a home remodel). When we discovered this, different teams across Pinterest brainstormed ways that they could improve users’ experience and understanding. We’ve since changed our new user orientation so that users explicitly follow interests rather than boards, which helps them understand where content is coming from. We built a sophisticated recommendations model that helps to infuse a person’s feed with content we think they’ll be interested in, even if they don’t ever customize the site themselves.
New: How has data science affected how companies like Pinterest approach the user experience?
Burbank: I think every company today is constantly balancing data analysis and the tremendous value that still comes from traditional tools like user research, individual user stories, and big strategic bets. Pinterest is no exception, and we focus relentlessly on creating a positive user experience using all the tools we have available, which now include data science as well as user feedback and our own intuition. One of the really exciting things about Pinterest is how much people love using the service, and the joy it brings them in tackling new adventures, discovering new ideas, or mixing things up in their daily routines. Capturing that individuality in the data is one of the keys to our success.