Are “complex” ML Models Always Better?

After mentoring data professionals for a few years, I notice that there is always a handful of recent graduates (from universities or boot camps) who are caught by the mental trap of “always go for the most COMPLEX machine learning model!”

For instance, some of them would always go for stepwise regression (bi-directional elimination) to estimate customer lifetime value. When asked the reason why they are not testing with forward selection and backward elimination as well, the usual answer is, “I did not do so because I thought that stepwise regression is sufficient given its complexity.”

So let me set this straight. When we are doing supervised learning, we have a target (Y) and several features (Xs). Our job as data scientists is to find the “hidden” relationship between the target – what we want to predict – from the individual features. Now I do not deny that complex machine learning models do help us to model complex relationships (a mixture of quadratic, cubic, setting “curly” decision boundaries etc) but at the end of the day, we must know that it is a “curve fitting” exercise, as mentioned by Judea Pearl in this article. Now the “curve” need not necessary be curved, but can be a linear relationship (a.k.a when X changes, Y also proportionally changes as well).

What I am saying is, we cannot totally discount a machine learning model just because it is not “complex” enough. We should only discount a machine learning model when it has been tested and we do not get better “predictions” from it (not modelling the relationship between the target and features accurately) or it does not meet the business requirements, for instance not meeting the transparency needed, or the cost of implementation is prohibitive.

In conclusion, as data scientists, we need to test all possible machine learning algorithms, regardless of its “complexity” (usually defined by how complicated the underlying math is) as each class of machine learning model provides a different way to “curve fit” the hidden relationship between the features and the target.

Do keep this in mind at all times! Have fun in your data science learning journey! If the post has been useful, do share it.