From algorithm development to experimental design
In the era of big data, says Jinglong Zhao, data is a crucial commodity, like oil.
“In my first line of work in algorithm design, I build the pump on top of the ‘oil’ to extract value from it,” he explains.
But algorithms can be flawed, and even useful algorithms can often be improved. “In my second line of work in experimental design, I try to evaluate: have I extracted enough value from the data or not?”
Zhao is a recent graduate of the IDSS Social & Engineering Systems PhD program whose research has focused primarily on online marketplaces, where algorithms play an essential role. Here, an algorithm’s effectiveness is typically defined by a company’s key performance indicators: profit maximization, market share, cost reduction. Measuring this effectiveness is complicated by the nature of the platform, where products interact and cause and effect can be difficult to determine.
“Think of testing if a vaccine is effective,” says Zhao. “You’ve got two cages of mice, and one cage gets the vaccine. That’s your intervention. Maybe one cage has significantly better outcomes. This is the simplest experimental design.”
Online marketplaces are not simple environments to measure, however. The products online can’t as readily be split into two groups to test certain interventions, and there are other complicating factors. Some products in the marketplace compete, but some are complimentary.
“The mice are in one large cage and they can bite each other,” jokes Zhao. “You can’t easily know if an effect is because of your intervention. You need careful thinking about how to design experiments that properly handle such interference.”
With a background in industrial engineering and operations research, Zhao now uses methods from optimization and econometrics in his research. “I’ve always loved math, but I’m interested in applying it to human problems. The economics and econometrics classes I took at IDSS significantly opened up my horizon. Theoretical research is important, but can we validate some of the insights that we build out of the theory?”
Zhao is one of only three to earn the relatively new SES PhD — so far. In the fall, he will join the faculty of the Boston University Questrom School of Business.
IDSS has left a lasting impression on Zhao’s future research interests as well. “I’m interested in the societal implications of algorithms as well,” he says, citing metrics such as customer satisfaction and quality of life. “I’m interested in evaluating whether policy interventions will create a positive or negative impact on society. The SES program has trained me well to consider such problems.”
