Criteo recommendation team is charge of choosing the best products to put in each of our ads. As everyone at Criteo, we work at a very large scale. We manage a catalog containing billions of different products and do recommendation for several billions of displays per day. And the number are growing every month.
As an ENG in recommendation team, I have to maintain the existing recommendation engine components (hadoop jobs and online servers) and to make them scale to sustain our growth. I also have to constantly improve of the quality of our recommendations.
That being said, you still do not know much about what it is like working at Criteo. So, let me describe you one day at Criteo.
9h30 : I arrive at the office. The first thing I do is to check the ABTest to see :
- a) whether it is positive,
- b) whether it makes us loose too much money.
A bit of context may help you understand why I do it.
For a few weeks now, we’re computing embeddings for the products sold by our advertisers. We use those embeddings to compute similarities between products. We have high hopes these scores will help our engine deliver better recommendations. Especially as offline metrics on our dedicated test platform have validated this hypothesis. So last week we launched an ABTest on 50% of the traffic to check it.
Actually even if the ABTest generates more clicks, it is still inconclusive in terms of added value for our partners. The confidence intervals are still large and it’s too early to draw conclusions.
Let it run for one more week.
10h20: Daily standup. It is a good time to hear about how the team projects are running. A colleague is having troubles launching a job on the cluster. I had the same issue last week. I take 5 min after the standup to help him configure his job correctly.
10h30: Time to code. Two of our researchers have invented a way to improve embeddings by using metadata.
They have published a paper describing their method. The offline results are encouraging.().
We want to test it on Criteo data for a potential ABTest. So, I need to implement the algorithm. As we have billions of products to process, we need to distribute the computation. Spark MLlib seems to be the perfect tool for this task.
As any Spark newbie, I fight with “_”, Spark serialization errors and implicit black magic. Fortunately, people from the Prediction team are pretty experienced with Spark and give me precious advices.
11h50: The menu pops on Slack. Only 5 minutes left to choose between salmon and burger.
11h55: Departure for lunch. My team takes lunch time very seriously.
13h00: Back to the implicits.
15h00: Team meeting around the monitoring of our Hadoop jobs. Two weeks ago, an inefficient monitoring almost lead us to a prod an incident. We have to improve ourselves on the topic. Everybody brings ideas on how to:
- improve our tooling,
- make the alerts easier to understand, especially for newcomers
- make sure the status emails are checked once a day.
30 min later we leave the meeting with a set of actionable items. That’s how we like meetings: short and efficient.
When I get back to my desk, a notification says that my offline analyses are ready.
I am designing a prediction using image similarities to improve our recommendations. It is based on product similarities computed from image features. The image feature are CNN features from AlexNet computed with Tensorflow. According to these results, one of my experimental model is clearly better than the others. NetherthelessI think I can improve it by tuning the similarity score bucketization. I launch a new series of offline tests to benchmark these parameters. I’ll check the results tomorrow. If they’re good enough we’ll plan an ABTest.
16h45 : I launched my Spark job on the cluster. I have to wait for it to finish. I use this time to review my colleagues code on Gerrit. As I need clarifications on one of the review, I go to the desk of the author. Direct discussions are often more efficient than endless argumentations on Gerrit.
18h00 : It is already time to go to our monthly R&D all hands: a meeting where all the R&D meets to share the latest updates:
HR news, latest successful projects and an open question session with our CTO.
After the meeting, I grab a beer and some food from the cocktail and take the opportunity to talk with other teams and meet newcomers.
The day is over. Time to go home. Tomorrow I’ll be also quite busy but in a totally different way. I have a recruiting interview planned and I have to find speaker for the next session of the RecsysFR meetup.
