By Twisha Mehta and Brennan Shearer

[Editor's note: This is a cross-posting with Samasource.] 

When we talk about randomized controlled trials (RCT) in development, we’re often talking about the findings of an academic paper. But another key part is how partners think through the role that rigorous impact evaluations can play in achieving their learning goals. An understated part of a successful RCT is the internal organizational buy-in that our implementation partners make to deliver their services in a way that complies with the design of a rigorous evaluation. I was able to catch up with Twisha Mehta from Samasource, currently in the middle of an evaluation of their program in Kenya, to hear a partner’s perspective of the goals and challenges they met along the way. Samasouce provides secure, high-quality training data and validation for AI. They count the most prestigious technology firms as their clients and serve technology teams driving humanity forward. As an early adopter of Impact Sourcing, the core of the Samasource business model is rooted in social impact. Their staff is driven by a mission to expand opportunity for low-income people through the digital economy.
 

What is Samasource, and what’s the program being evaluated?

Samasource is a social business that provides training data strategy, annotation, and validation services to technology companies. We have global delivery centers in Uganda, Kenya, and India that provide language and computer vision data services for use in machine learning algorithms, such as those used in self-driving cars and virtual reality applications. Our business model is predicated on the idea that, when combined with the appropriate technology platform, we can effectively train a workforce comprised exclusively of workers from low income, low education backgrounds to deliver high-quality digital services at market competitive quality and prices. This is the driving idea behind our Learning and Development (L&D) training program. We recruit students to join 10-day boot camps from low-income communities throughout Nairobi, Kampala, and Gulu. Upon completing the training course, they are eligible to apply for jobs at our global delivery centers. For most trainees, this is the first formal job they’ve had. The curriculum covers basic digital skills, occupational skills relevant to AI work.

This RCT aims to understand the value of training as a stand-alone service along with the value of training in combination with the ability to apply for a job at our global delivery centers.

(See IPA’s project summary for an easily digestible explanation of the impact evaluation.)
 

Why was it the right time for a randomized evaluation? What do you hope to learn?

As a social business, we often face the dual challenge of generating social impact, while also balancing business needs of growth and scale. We’ve learned a lot along the way on how to do that effectively. Investing in our impact is important to us for a few reasons. First, we want to create more transparency about what we’ve learned. We want to change the mindset of private sector corporations that believe that you have to choose between social impact and profit. You can have both! We’ve seen firsthand how transformative the power of a paycheck has been on the lives of the people who we hire. From that perspective we want to use this impact evaluation as a proof point.

Second, we’ve reached a stage of organizational maturity where we have unprecedented insights into our entire supply chain. Data is a core part of our DNA as we grow and scale—and we are committed to using it for continual learning. Operationally, this positions us to not only ask questions about our impact but to make changes to our targeting and delivery model based on what we learn.
 

Can you describe the process of running an RCT within a profitable business?

Internally, we had support from the senior management team which recognized the business case for the RCT—we needed to undertake this study in order to prove the efficacy of our business model and prove that social businesses can thrive in the world (and back it up with rigorous impact data!). Operationally, the challenge was to deliver within the constraints of our demand-driven business without compromising service quality. This meant getting buy-in from teams at the local level and working very closely with the local teams.

What were the challenges?

Our foremost concern was about our reputation and maintaining the trust we’ve built in the communities we operate in. It was very important to us to be forthright and transparent about the study process with the people we invited for our Learning and Development training boot camps. Still, it was a challenging shift for our trainers, who work closely with the communities we recruit from, to inform people that they were placed in the control group—and will not receive any training or opportunity to apply for jobs with Samasource. To mitigate any perceptions of unfairness, the team decided to run the randomization process with a physical lottery during the open enrollment periods. This helped interested students that were placed in the control group understand that it was simply the luck of the draw.

To mitigate any perceptions of unfairness, the team decided to run the randomization process with a physical lottery during the open enrollment periods.

The second major challenge was to operationalize the research design. After all, our services are dependent on client demand and we can’t hire for new positions without securing work first. We worked closely with our various business units to forecast hiring based on client demand, then worked with the researchers to plan the research within those constraints.

Good, bad, or ugly, we're going to share these results and learn from them.

What’s next for this evaluation?

At Samasource, we have a strong data-driven culture across all of our business units. Data provides us with concrete evidence to pilot initiatives, understand their efficacy, and invest in ideas that work. The RCT is a significant investment for our company. Good, bad, or ugly, we're going to share these results and learn from them. We know that we can use the results of this RCT to improve our training program to deepen our positive impact by creating more digital jobs.

When we started, Samasource was a seemingly far-fetched idea: Training and hiring marginalized East African youth to do digital work for some of the largest technology companies. However, several philanthropic funders, including Rockefeller Foundation and Mastercard Foundation, took a chance on us and helped us test and scale our idea until we reached financial sustainability. Seven years after we started, we hit financial sustainability—an elusive north star for many startups!

Now, as we complete our tenth year as a company, we look forward to the RCT results to validate our impact, create new academic literature on our model, and most importantly, provide evidence to our early supporters that well thought-out out-of-the-box ideas are worth investing in! (With impact data of course!)
 

Twisha Mehta is the Senior Impact Manager at Samasource. 

Brennan Shearer is a policy and communications manager at IPA.