Consider past projects that involved a team effort. Think of projects that have failed to meet deadlines or gone over budget. What is the common denominator? Is this too little hyperparameter tuning? Bad model for logging artifact?
Probably not, right? One of the most common causes of project failures is poor project management. Project management has the responsibility of breaking down a project into manageable steps. Each step must then be estimated continuously for the amount of work remaining.
There is a lot more that a given project manager is responsible for, from sprint execution to retrospectives. But I don’t want to focus on project management as a Role, I want to focus on project management as a Skill, Just as anyone on a team can demonstrate leadership as a skill, Anyone on the team can demonstrate project management as a skill. And boy, is it a useful skill for a data scientist.
Let’s focus on estimating the single phase for brevity. The fact of the matter is that most data science tasks are very hard to estimate:
- How long will the data cleaning phase take? Totally depends on the data you are working with.
- How long will the exploratory data analysis phase take? Totally depends on what you find out along the way.
you get my point This has led many people to think that estimating the duration of phrases in a data science project is pointless.
I think this is a wrong conclusion. What is more accurate is that it is difficult to accurately estimate the duration of the data science phase before starting the phase, but working with project management continuous estimation. Or, at least, that’s what good project management should do.
Instead of anticipating the data cleaning task in advance, imagine you are one week into the data cleaning task. Now you know that there are three data sources stored in different databases. Two of the databases lack proper documentation, while the latter lacks a data model but is very well documented. Some data is missing in all three data sources, but not as much as you feared. What can you say about this?
Sure, you don’t have zero information. You know that you will not finish the data cleaning work tomorrow. On the other hand, you’re pretty sure three months is way too long for this job. So you have a variant distribution giving the probability of the phase ending. This distribution has a “mean” (an estimate for the duration of the phase) and a “standard deviation” (the amount of uncertainty in the estimate).
the important thing is This ideological distribution changes every day. You get as much information as possible about the work that needs to be done. Naturally, the “standard deviation” will decrease over time as you become more and more certain of when the phase will end. It is your job to communicate this information to the stakeholders. And don’t use the distribution language I used when explaining it to stakeholders, that may live among us.
It is extremely valuable for a data scientist to be able to say something like:
“I think this phase will take 3 to 6 weeks. I can give you an updated estimate in a week which will be more accurate.











