Imagine you were having a great time at a football game. At the end of the game you took out your phone to call for an Uber. Unfortunately, so did everyone else around you. The price had surged 300%. However, you noticed that Uber gave you another option. You can call a different driver for the normal price. The only catch here was that this driver did not have a driver’s license.
So what would you do here? You were a bit tipsy and definitely didn’t want to pay the ridiculous surge price. Plus this licenseless driver had been vetted by Uber, so he must be legit too, right? So how many of you would want to pay the premium and have a licensed driver to take your home? You might know which driver you would choose, but do you know what people looking at the App around you are thinking?
Now you can see how a pro license could impact customers’ decision in the Company T Marketplace, a platform for hiring local services, is tricky. Would our customers hire a licenseless handyman to fix their HVAC for cheaper? Would our customers even notice the license badge, or would they just assume, like when calling an Uber, that all Company T handymen had been verified? In addition, do customers even know which jobs are risky, that they should hire a licensed pro?
It is crucial to know how much customers care about licensed pros in Company T’s home construction category. In other words, how licenses impact customers’ choice. And do the benefits outweigh the costs?
For Company T, the license mapping verifying campaign across 50 states and over 50 risky categories is costly. If we know for sure that customers care about licensed pros, we could not only scale our license verifying campaign, but also guide the customers towards safer pros through our redesigned service page.
The Experiment:
To measure the license impact on customer’s CTR (click through rate), we launched an experiment tweaking a pro’s service page. In baseline, the license information was very limited. In treatment, we added a credential section to display license and background check information. The randomization is done at the visitor session level. It means that two visitors could see different designs for the same pro service page, one with credential section and one without. The same pro could appear in both treatment and control, dependent on the visitor assignment.
In an aggregate level, we saw that the licensed pros in treatment had 2-7% increase in CTR. However, this difference was not statistically significant. Moreover, since we could not randomize pros, the difference in CTR could simply be a difference in pro make--the pros in treatment were better pros.
Causal Inference: Fixed Effect (Panel Data) Model:
In our dataset, each row is a customer session’s visit to a pro’s service page, which results in either a click to contact or not a click. If we simply fit a binary regression (equation 1) with the treatment variable (licensing), we wouldn’t be able to account for the differences among pro service pages (since one pro service page could be viewed by hundreds of visitors). The pro service page effect is called the Fixed Effect (FE), the unvarying characteristic for each pro service page cluster, and this type of data is called panel data. To control for FE, we add one more term, alpha j, in the regression. Alpha j is a list of one hot encoding dummy variables for each pro service page j. We can consider alpha j as the individual intercept for each pro service page j.
Equation 1: Vanilla Binary Regression
Equation 2: Fixed Effect Binary Regression
Result:
After running the model on five different threshold cuts of the data, we detect minimal to none effect from license treatment. Rather than drawing a conclusion that license has no impact on a customer’s CTR, the hypothesis above leads us to recommend a stronger experiment to kill off the lack of saliency assumption. This more focused experiment would put the license badge directly at the face of the visitors, too prominently to be ignored.
Variant 1 would display a pro’s license badge, variant 2 would display a pro’s badge for something like ‘respond quickly’ or ‘high demand’. The ATE1 ( average treatment effect) of variant 2 to baseline would be the effect of any platform endorsement effect on pro. The ATE2 of variant 1 to variant 2 would be the license effect. If ATE1 is significant, we know that we could give licensed pros a boost if we choose to, changing the contact make of home construction categories to more safer. If ATE2 is significant, we know that the license badge itself has a positive impact and we should further push for this.
Conclusion:
To answer a complex problem such as the license impact, we need a well-designed experiment coupled with causal inference modelling. We carry forward the lesson the next time we set up experiment design.
Appendix
This DAG illustrates the need for a fixed effect model. Each Yij here is the outcome variable (if visitor j contacts pro i). We are interested in measuring the correlation between Dij (treatment: licensed badge display) and Yij, which gives us the treatment effect. However, Yij are also correlated with ui, the unobserved variables for the pro service page quality (such as picture quality, review sentiment, etc), making Dij endogenous. To eliminate such confoundedness, we control for Xi, the dummy variable for each pro i. Thus, the only variation in Yij would come from Dij, the treatment effect.
