Preprint
Machine Learning

CatBoost for big data: an interdisciplinary review

John Hancock(Florida Atlantic University), Taghi M. Khoshgoftaar(Florida Atlantic University)
November 4, 2020Journal Of Big Data1,712 citations
PDFDOI

1.7k

Citations

42

Influential Citations

Journal Of Big Data

Venue

2020

Year

Abstract

Gradient Boosted Decision Trees (GBDT's) are a powerful tool for classification and regression tasks in Big Data. Researchers should be familiar with the strengths and weaknesses of current implementations of GBDT's in order to use them effectively and make successful contributions. CatBoost is a member of the family of GBDT machine learning ensemble techniques. Since its debut in late 2018, researchers have successfully used CatBoost for machine learning studies involving Big Data. We take this opportunity to review recent research on CatBoost as it relates to Big Data, and learn best practices from studies that cast CatBoost in a positive light, as well as studies where CatBoost does not outshine other techniques, since we can learn lessons from both types of scenarios. Furthermore, as a Decision Tree based algorithm, CatBoost is well-suited to machine learning tasks involving categorical, heterogeneous data. Recent work across multiple disciplines illustrates CatBoost's effectiveness and shortcomings in classification and regression tasks. Another important issue we expose in literature on CatBoost is its sensitivity to hyper-parameters and the importance of hyper-parameter tuning. One contribution we make is to take an interdisciplinary approach to cover studies related to CatBoost in a single work. This provides researchers an in-depth understanding to help clarify proper application of CatBoost in solving problems. To the best of our knowledge, this is the first survey that studies all works related to CatBoost in a single publication.

Analysis

Why This Paper Matters

This paper is significant because it fills a gap in the literature by providing the first comprehensive, interdisciplinary survey of CatBoost, a relatively new but powerful gradient boosted decision tree (GBDT) algorithm. As big data continues to grow, practitioners need clear guidance on which tools to use and how to tune them effectively. By covering both successes and failures, the authors offer balanced insights that help researchers avoid costly mistakes and make informed decisions.

The interdisciplinary approach is particularly valuable because it aggregates lessons from fields as diverse as healthcare, finance, and engineering, showing that CatBoost's strengths (handling categorical features, robustness to noise) and weaknesses (sensitivity to hyperparameters) are consistent across domains. This makes the paper a practical reference for any AI practitioner considering CatBoost for their work.

Technical Contributions

  • First unified survey: Collects and synthesizes all CatBoost-related research up to 2020 in one place, saving researchers time and effort.
  • Balanced perspective: Analyzes both positive and negative case studies, providing a realistic view of CatBoost's capabilities.
  • Hyperparameter sensitivity analysis: Emphasizes the critical role of tuning, which is often overlooked in applied studies.
  • Categorical data handling: Highlights CatBoost's built-in support for categorical features as a key differentiator from other GBDT implementations.

Results

The paper does not present new experimental results but synthesizes findings from 1712 cited works. Key takeaways include: CatBoost often outperforms XGBoost and LightGBM on datasets with many categorical features; however, without proper hyperparameter tuning, its performance can degrade significantly. The review identifies common pitfalls such as using default parameters without validation and failing to account for data heterogeneity.

Significance

This survey serves as a foundational reference for the CatBoost community, helping to standardize best practices and accelerate adoption in big data applications. By exposing the importance of hyperparameter tuning and the contexts where CatBoost excels, it enables researchers to apply the algorithm more effectively. The interdisciplinary scope also encourages cross-pollination of ideas, potentially leading to novel applications in emerging fields like IoT analytics and real-time prediction systems.

catboostgradient-boosted-decision-treesbig-datamachine-learningensemble-methodshyperparameter-tuningcategorical-data

Related Papers