Do You Need To Manage a Team of Data Scientists? — Understanding Control Theory Might Change Everything

Originally published here

How I would use it to manage junior, mid-level, and senior data scientist

Data Scientist is a highly sought-after position in the world of business. There are many who want to enter this field, and there are also those who want to stay in it for as long as possible. This can be difficult especially as you move up in the world, if you don’t know how to manage data scientists properly, so we’re going to discuss different ways to manage junior, mid-level, and senior data scientists based on the Control theory.

1. The problem with managing data scientists

The problem with managing data scientists is that there isn’t a set way to do it. Every business has different needs and requirements for its data science team, so the process of managing them varies from company to company. It’s not just about managing people but also managing projects and outcomes of those projects. You might want to manage junior-level data scientists one way while managing senior-level data scientists another way. Obviously, you would assign more challenging and larger-scale projects to senior data scientists, which is why managing them differently makes sense. However managing junior-level data scientists one way and midlevel the other may not be as efficient or effective for either group of people without understanding why and how…. let alone being clear about it upfront.

For example, managing a team that has both junior and middle levels but assigning more challenging projects to only the seniors could result in frustration from the juniors who might feel like they are being held back or not growing in their careers. You wouldn’t want to do that, right?

Considering all of this, managing a data science team is tricky business and requires a lot of thought and planning on your part as the data science manager/leader/boss… whatever title you give yourself 🙂

2. What is Control theory and how does it work in the workplace

OK. Have you heard of Control theory? This theory has been around for a while in the management area. Basically, Control theory says depending on the level of uncertainly around the likely outcome and the path which you can take to get there, you have to control different aspects of the process.

The illustration above describes what I mean. Let’s say there is a clearly defined outcome and you know exactly what you have to do to achieve the outcome. The theory suggests that your management focus should on the behavior of your staff. For those projects that you know the likely outcome but there are multiple ways to get to the outcome perhaps due to changing environment, then you are recommended to focus on the outcome and give the flexibility to your staff to work out how to get to the outcome. For those complex and/or “never done before” types of projects, you should focus your attention on hiring the most capable staff to run the project.

3. How to use Control theory for junior, mid-level, and senior-level data scientists

You can use Control theory in managing junior data scientists by defining clear parameters. For example, if you are a manager given a task to work on selling widgets to small businesses and you hire three data scientists with different experience levels then give each person one element that they will be responsible for like making sure there’s enough supply to meet demand or managing the cost of the campaign or managing demand.

-If you hire a junior data scientist then define what was successful for the last project and set clear parameters to follow that pattern. For this role, you would emphasize their technical ability and willingness to follow directions when you are hiring for this role. He/she is more likely to do exploratory data analysis and documentation.

-Midlevel data scientists can figure out ways to get the desired results within those boundaries after understanding objectives. They are more likely to try something new because they have enough experience with previous projects. They are likely to test different machine learning algorithms for defined outcomes. When hiring for this role, your interview questions should be designed to capture their ability to solve problems collaboratively with other team members and focus on driving their efforts for achieving their goals.

-Senior data scientists will have a better understanding of how your company works and what parameters are important to set so they can find out ways that achieve it in the best way possible. You might want to start with an idea or an innovative way that you haven’t really done before. You should get senior data scientists more free reign to explore and challenge assumptions. As such, it’s critical to look for their previous experience of managing projects from idealization to production should be your hiring criteria.

4. Why you should try out this new system of management

We know this is typically how hospital staff are managed. When you walk into a hospital, you are likely to get a nurse who will check your temperature and blood pressure as a routine. Let’s say you need a neurosurgeon, you know he/she is not bounded by predefined procedures and closely monitored/managed by the hospital executives.

It’s easy to assign tasks and approach your management focus sorely based on how difficult some of the tasks are (i.e. give the easy tasks to the junior data scientists). However, I recommend that you should shift your management focus based on what you can control.

– Junior data scientists need to be guided by more experienced data scientists; they may feel inadequate in their abilities and skills if left on their own.

– Midlevel staff will benefit from having control over projects which allows them to make decisions for themselves (or with their junior data scientists).

– Senior staff will be able to pool their experience and knowledge while managing the work of both junior and mid-level staff.

Data science is a field that requires an understanding of statistics, machine learning, and data engineering. It’s not easy to learn all these skills in one go AND apply them to real-life projects. It would take time and experience to build your skillsets. As you progress your career towards being a project lead, manager, or executive, you will need to start thinking about developing your management skills (i.e. soft skills). When you become a manager, leader, or boss, perhaps you could utilize the Control theory to rethink how to manage your data scientists while managing multiple projects and stakeholders. Typically, assigning easy tasks to junior staff and difficult tasks to senior staff is a good idea but by using this approach, you might be missing out on opportunities for both types of employees to grow their skillsets.