Etsy CTO explains how the marketplace is using big data and the cloud

Other online stores may have their own handmade or vintage shops, but Etsy remains top of mind for consumers, at least according to the company's CTO, Mike Fisher.

The company has placed a special emphasis on these goods, which has required a lot of technological innovation.

Etsy has also moved completely to the cloud, enabling it to be more nimble with huge sets of data.

Visit Business Insider's homepage for more stories.

Though other websites with handmade and vintage marketplaces, like Amazon Handmade or eBay, have tried to imitate Etsy, Etsy still remains the go-to for these goods, according to the company's CTO, Mike Fisher.

Etsy has doubled down on this advantage, pouring tech talent and resources into creating the best, most empowered place for artisans and their buyers to connect.

Now that Etsy is moving its database to the cloud, its engineering team is able to focus on helping the site's 40 million buyers search through 60 million unique items from two million sellers.

Read more: Etsy exec explains why the company has a huge advantage over everyone else who tries to copy it

Fisher recently spoke with Business Insider about its move to the cloud and how it builds its site to work for both buyers and sellers.

The following interview has been edited for length and clarity.

The power of data

Mike Fisher: The thing that always impressed me about Etsy was the mission of helping [our sellers]. Really, they're entrepreneurs. We refer to them as creative entrepreneurs.

These are individuals who have a gift, a talent, a hobby, a passion for something. The diversity of that is amazing, but we enable them to find their raving fans where they couldn't do this sitting in their kitchen, in their studio, their garage where they're building stuff in middle America, or another country without our platform. That's this amazing thing, is that we give over two million sellers this platform to connect to 40 million buyers with 60 million unique items. I've always loved that mission.

I was also always impressed with [Etsy's] engineering talent because they've been very innovative since the beginning. They were some of the early adopters of DevOps, continuous integration, continuous employment, open sourcing, all of this. Even though you don't necessarily think of the marketplace as being super technically innovative, Etsy engineering always has been.

When I came in in 2017 ... we were so innovative, but we had gotten to a place where a lot of it was homegrown tech. We were at the time, still, in data centers hosting our own servers and everything. I looked at it as an opportunity for us to take the engineers and move them closer to the customers. Instead of doing this very low-level setting up servers and handling that, if we could get out of that business, we could then focus more on the marketplace and build more and more features for the sellers and buyers. We started a migration to the cloud, so we chose Google. We went through a big RFP process ... and we started in January of 2018, the migration.

This is a testament to the strong engineering team here: Nine months later we had the marketplace running on the cloud, which is a super fast migration. We are still wrapping that up this year, so we're about 75% done. We're ahead of schedule, which you don't usually hear on projects.

The engineering team here is just amazing. We'll be shutting down data centers starting this fall. It's been great. It unleashes another topic that might be super interesting: our ability to do really, really sophisticated machine learning on the platform.

Again, you look at the marketplace and say, “How challenging is this technically？” And it ends up that it's super technically challenging. It's one of the reasons that we can get great engineers here: We've got all these cool problems to solve.

Business Insider: You can scale up and down as much as you want.Fisher: Scale up and down. We're typical for e-commerce. We get busy at certain times of the day or certain parts of the year, and yet the servers just consume the energy in a flatline nonstop.

The fruits of technological labor

BI: Since two years ago, everything has been reoriented. What are some of the fruits of that now？Fisher: One of the things that we call “our right to win” is having the best search and discovery. With 60 million items, being able to find that special item when you're looking for it, and maybe even not knowing exactly what you're looking for, and help raise that up and get you the right item, is super important. One of the very first things we did with machine learning is focus on search. The reason search is difficult here as opposed to another e-commerce is [we have] 60 million items, so a really, really large number of items, but it's also very unstructured.

We don't have a catalog that a lot of e-commerce shops have, or SKUs that one person fills in the description. Our sellers write their own descriptions, and these are unique items. If they're not unique, they probably actually don't belong here. The sellers write a description of, maybe, a sweater that they made. They can describe it however they want, they can have the tags that they want, and once it's sold, it's probably gone forever. They might [sell] another one, but it might be described a little bit differently.

Read more: Bonobos CEO reveals why the brand is trying to redefine masculinity, what life is like as a part of Walmart, and what's next

A typical e-commerce site might sell the same blue sweater 5,000 times, and so [customers] get really used to like, “Oh, if you search for blue sweater, this is the best seller,” and all this. Our search becomes infinitely more difficult because of that. What we do is we take our basic search, we use technologies like everyone else, like TFIDF, which is term frequency inverse document frequency. We look at all the terms used, but our sellers also know that if it's coming up on a holiday, if they put that holiday in the title, that'll show up as a term. If we're coming up on Father's Day, even if I'm selling something that might not be a perfect Father's Day gift, if I put Father's Day, it might be shown in search results when people search for that. They try to game search, so we can't rely completely on that. What we do is we take machine learning, we take searches from the past and what people look for, and we analyze that based on what they've clicked, favorites, added to cart, purchased, and so forth.Then in real time, we take whatever your search results are — say you're looking for beer steins, that might return 10,000 items. We take the top 800 and re-rank them because we know that 83% of our purchases are on the first page of the search results. Just like Google, you never click to the second page, right？ It's the same thing with when you're on an e-commerce shop. Most of the purchases are on the first, so getting that first page of results right is super important. In real time, we can re-rank it based on what we call context-specific ranking. We know the context of the query that you're looking for, we know lots of other things like locations and things like that. We can say, “This is what we think would be a better search result for you out of 10,000 items.”

In 2017-2018, [search improvements helped contribute] $260 million in incremental GMS. GMS is our gross [merchandise] sales.

BI: Is it hard to balance the priorities of the consumers and the sellers？ Or do you find that when you're helping one, you're helping the other？Fisher: I think that's the great thing about a two-sided marketplace. When you help one, you help the other. If you make the experience ... easy, then the sellers are happy. They want buyers. Then when we make the sellers' lives easy ... for listings, making it easier for them to see and analyze their results, then of course they have more time. These are artisans and craftspeople who want to be in the shop, they don't want to be managing their marketplace. If we can make all that very easy for them, they can be in a shop and be more creative, which means more items and more creative items. All of that's just a win-win.BI: Are these the kind of innovations that put you in another category versus some of the competitors that are trying to do the artisan, handmade thing？Fisher: It requires a lot of very deep data science and analysis to figure it out. What are the features that really matter in these models？ If you think about machine learning, we had a lot of the concepts around neural nodes and decision trees, and all these algorithms have been known for 40 or 50 years. For decades, people struggled with if this is going to be really valuable stuff someday, and we never had enough data and compute power to make it real. Now, just over the past couple of years, we have had enough data and enough compute power to actually make these super advanced algorithms worthwhile.That's the really cool thing about the space. We collect over a billion events a day of data, so as the buyers come in and they look and they search and they click and they add, all of this creates events, and we can analyze that and then use all of that data into our models. Then we put it back out to the site to get recommendations and re-ranking, and all of these things to make it a better experience. It's just now becoming available to us. We've even seen, moving from our data centers, where we had very fixed compute power, into the cloud, just over the last couple of months we're seeing us jump from models that used two weeks' worth of data, or 30 days' worth of data, to a full year. You see some really interesting incremental benefits from that because we can see seasonality effects, and all this stuff that you couldn't see if I just look at a trending 30 days.We have massive amounts of data that we can put to this, so it is definitely something that a lot of smaller companies can't do.

BI: Does not having the SKUs make it infinitely more complicated？Fisher: Yes, but it's also what makes this unique. If we had SKUs and catalogs and stuff like that, we would be just another e-commerce shop. Because we're not that, what makes us unique, is that you don't come to us for the commodities, you come to us for that special item.When [CEO] Josh [Silverman] came in, he focused on this marketplace that really was lightning in a bottle. Instead of thinking that like, “Oh, let's go focus on other marketplaces out there,” if just look at our top six categories, they're massive in themselves. Home and living is just a huge category, and that's only one of ours that are out there. Jewelry, another massive category, and that's just one of ours.BI: It's more of an everyday experience than a special occasion.Fisher: Yeah. We find that people, whatever they show up for — a year later, they show up for that again. If it's a Father's Day gift, or a Mother's Day gift, they come back a year later for that Mother's Day gift. But we're way more than that. It's literally everything from every piece of furniture in your house to the art on the walls to handmade clothes, shirts, and dresses, and everything you can think of. Your glassware, like these artisans and craftspeople, make almost everything. It's pretty amazing.

BI: For these six top categories, are there unique tech challenges for each one？Fisher: There are interesting challenges around the data. When you come in for clothing, you want to be able to see color variations and sizes. When you come in to shop home and living, like rugs, you need to see dimensions and shapes. Having that and understanding that some of our pieces can fall in multiple categories, to figure out which pieces of data I need to properly show this up in the results for an individual, become incredibly complicated.Because our items are unique, they could sometimes fall in multiple categories. You might need colors and sizes and shapes for some of it.

Reshaping recommendations

BI: So taking all this data, where are you going next with it？ You're improving search — I'm sure you'll probably further improve that.Fisher: Continue to improve search, absolutely.BI: Is there another category that you're looking at？Fisher: Recommendations is another piece of it. Not only when you type in the search query, to give the best results, but also even before you get there — when you land on our homepage or land on a landing page, that we can make recommendations for you. Whether we know you because you've signed in and you're a frequent purchaser, or if we don't know you and we want to show you stuff. As you click on stuff, instantly being able to use that data to give better recommendations is an extreme area that we're focused on.BI: It's based on the same kind of data, right？Fisher: It is. Lots of data. Again, very powerful. We also know that you might be on different journeys in different times. You might be looking for a Father's Day gift this week, but you also might tomorrow be looking for a wedding gift for someone else. They might be two very different journeys, and so understanding which journey you're on is really important for us. Instead of just simply showing like, “Hey, these are your last 12 items,” which all might be related to the wedding search, we could show you more groups. “Oh, you searched for this Father's Day gifts, and weddings, but you also were adding something to your apartment.” You had these separate journeys, and understanding that makes it more powerful.We're even thinking about that from a style perspective. If you're shopping for yourself, you have a particular style, but if you're shopping for someone's wedding gift, you might be thinking about their style. [We are] trying not to mix those.BI: How do you not mix those？Fisher: We try to understand your journeys in separate parts so that we can assign styles and paths and recommendations and all that to different ones.BI: That seems like it would be hard to judge.Fisher: It's very challenging. Then you get added on top of it that sometimes you come to Etsy to be inspired, while other times you come to actually shop because you know what you want to find. Those are two very separate journeys as well. When you're coming to be inspired, you want to see some of the broader categories, or something a little maybe off the beaten path, or something different that you're not thinking of. And so we can use things like the length of your search query, so if you type something that has more than two or three terms, we know you probably have a really good idea of what you're looking for, as opposed to if you just put general terms. We use all these types of notions and ideas.

Then of course, we look at what other people search for and how they behaved, and we try to use that to recommend stuff for you.BI: It almost sounds like psychology at this point.Fisher: Yeah. We, of course, have our mobile apps versus desktop, and the behavior of the same user on each one might be very different.BI: What is the difference？Fisher: Well, our power buyers generally have downloaded the app, and so their frequency is much higher, but time of day might be a factor. Someone during a lunch hour or break on a mobile device might be really inspirational, they're looking to spend their five-minute coffee break or whatever waiting for someone, just to be inspired. Yet, when they're at the evening or on a weekend, they might be very targeted in their looking for something. It's not perfectly clean like that, but it's all types of different behaviors that we've got to watch for and try to make the best experience that they can have.BI: Were there any “aha” moments when you found an interesting distinction between how, for example, a different buyer acted at some time of the year？ Fisher: We're working on styles right now, and the data science team published a paper recently about the work on that. I think it's super interesting because there's not a given classification for styles. Our merchandising team came up with 42 different things that they would classify as style. Some of them are your classic style of rustic, and others wouldn't be a classic, like romantic or tropical, which aren't really styles, but they're the way we can see the world.

We're using some really powerful neural networks to process that data. To test that and see how accurate we were, we looked at seasonality. Romantic, of course, had a significant spike during Valentine's Day. Then humor tends to spike on Father's Day, and then a couple other times during the year. Inspirational is around graduation.

Even though again, some of these aren't like a pure style that you would think of like mid-century modern or stuff like that, you can absolutely see that to our buyers and sellers, it resonates. That they use those terms, they look for those types of items. Then it proved itself out based on how you would imagine a calendar to look like in terms of the spikes and the highs and the lows, and it played out that way. That was interesting to see. It was like, “Oh yeah, just as you would expect.”

BI: Are those classifications automatically applied？Fisher: Well, they're trained. What we did was we got our merchandising team to come up with the list. We have something we call a stash. It's basically our editors' picks, and there's about 130,000 items classified in this. This is back to the big data. The problem is 130,000 items isn't enough data to classify. We had to go back to our search results again, and we looked for these terms in search results, giving us over three million more data points to apply to this. Then we can train our model so as listings come in we can apply our machine learning model to automatically classify them.

Crafting special moments

BI: We talked a little bit about competitors, but who do you see as the top competitors in your space, or in e-commerce at large？

Fisher: We all use other e-commerce shops every day, just like other people. There's a huge need for that in the world, that whether it's Amazon or Walmart or Target, whoever, there's a need for that.

We also feel there's a completely other side to all of our lives for special and unique things, and there's a big need for that. As we look back and see the growth over the last two years, you see that really start to play out in our belief that in a world of increased commoditization and sameness, there's also, as we talked about, people who search out the opposite. Whenever it goes to an extreme, people look for an opposite. I think we're seeing e-commerce drive more and more to that commoditization.There's also the other side of it, that people want connections with humans. They want the human connection, which is a big part of my marketplace, that we connect buyers and sellers one at a time. That's really special, that a huge portion of our items can be personalized. I think that's what we see as why no one really competes with us, because we are this marketplace for the very special. Even if someone's trying to focus on some niche, we cover almost everything in every category.BI: As customers expect more from all e-commerce, they expect it immediately and as cheap as possible. How do you make them understand the process behind it？Fisher: The creation and the shipping is super important, so giving our buyers more visibility to that is really important. Again, back in working with sellers, and by working with them, we made changes in the marketplace to make it easier for them, because we obviously can't call two million sellers and teach them how to do something. We make things in the marketplace easier. If we make it easier for them to show when this really would ship, or how long it's really going to take to get there, we can do that on feature functionality on the seller side, and that's a huge win for the buyers because now they can look for stuff and say, “Is it going to make it by my friend's birthday？” We focused on that before the holiday seasons last year. That was a really big win for our buyers and sellers.BI: Do you think that there's a certain kind of customer who want [the special items Etsy offers？] Or do you think that there's a customer in everyone that would desire that sometimes？Fisher: I think there's a piece in everyone that desires it sometimes. Again, if I'm buying paper towels, maybe I don't need it packaged and with a note, but if I'm buying something for my partner, it's even more special that I picked out this thing, but I also worked with the seller to get just the exact right thing, and just the whole story, the background of it. We've got sellers who literally raise the sheep, and they sheer them, and they spin the wool and dye it all on their farm.You give someone a blanket, and it's not just like, “Oh, this is a beautiful blanket.” It's a blanket with a story.The other aspect of shipping that we've found is important to us as a business and our buyers is carbon offset. E-commerce shipping is massive in terms of carbon. One-day shipping doesn't help that.

We actually ... announced that we were offsetting all of our shipping. We pay for carbon offsets for all of our shipping from our sellers to our buyers.

The amazing thing about it is not only that we do it, but when we showed that to our buyers just with a banner that says, “We've offset the shipping,” they purchased more. They made up for the cost of us doing that by buying more. We know that this resonates. I believe people do want fast product. Of course, when I fall in love with something, I need, I want it now, but people also want to be responsible. BI: That's interesting, because e-commerce is so expensive already than traditional retail.Fisher: We thought the same thing. This is part of being the very mission-driven company, that it fits with what we believe in, and by making stuff like that public, we feel we hold ourselves accountable.BI: You brought up the shipping cost. Customers typically hate paying for shipping. That seems not to be the case for Etsy？

Fisher: We did a big push over the holiday season for sellers to offer that and increase significantly their listings. Obviously that helps with buyers. That piece of it is educating the sellers, and it's something again that's about making a change to the platform to help educate them and incentivize them to do that. We understand shipping's not free. In the buyer's mind, even if you write the check, or process the credit card for Prime, you still in your mind, think you're getting it for free, even though you're obviously not.

Getting trendy

BI: I wanted to ask about trends. What kind of trends have you guys seen？

Fisher: From a technology perspective, we enable them through a lot of data. We can give them a ton of data around what is popular, what is trending. If you actually just click on our search, the dropdown shows the most recent hot searches that are happening. We give that to the merchandising team, and then they apply their intellectual knowledge and view of the world on top of that. Then they go on the televisions shows, and all of these places, newspapers and magazines, like to talk about that.

BI: It's more about trend setting than trend following.

Fisher: Yeah, the sellers can be very sophisticated and understand what is the hottest trend. They want to make product that people want and care about, and so within categories, they can watch and see what's hot and jump towards that. We see it a lot.BI: The maker movement, is that still on the upswing？Fisher: Our data would suggest yes, that we're growing — e-commerce is growing. We're growing faster than e-commerce. That's pretty public data. The interesting thing about e-commerce is how small it is of total commerce still. Those of us who live online, you're like, “This is all I ever do.” Especially in New York. “I buy everything online, I get my groceries delivered.” As a part of commerce in total, it's still a small portion of it. E-commerce still has tons of room to grow within general commerce.

Then we see our piece of this, of the commerce in the maker space, as you can grow it even faster. I think, like I said, whenever there's an extreme, you tend to see people want the opposite as well, the balance. That's what we're seeing.BI: Was there anything else that you wanted to talk about？Fisher: I've talked a lot about the mission, but that comes through in what we call our culture of how much we care about diversity and inclusion. From the very early days of the company, they've cared about it. Within engineering — engineering is almost half the company — 33% of our engineering staff are non-male. If you look at the industry, that's almost double than most. That's engineering-specific. If you include the broader technology, which is product management, design, all that, it's even more. We have gender parity on our board, our executive team. All of those things are really important to us, and it's just part of the culture resonating with our mission.