10 Setting The Scene
You are not expected to be a ML engineer or have years of experience to be able to follow this section. However, knowledge of the basics of machine learning and a keen desire to supplement what is spoken about here with personal reading is strongly advised. For example,while we’ll provide a high-level explanation of terms like ‘epoch,’ exploring external resources for a deeper understanding of these concepts is strongly recommended.
Machine learning models (specifically language models) play an important role in extracting meaningful insights from the vast amounts of data we work with on a daily basis.
Whilst our job mainly focussed on using these ML models to perform classification tasks (consider sentiment analysis, named entity recognition, emotion detection, peaks and pits etc), ML in text analytics goes beyond classification and includes text generation, translation, summarisation, question answering to name a few implementations.
For us data scientists, our goal is often to decide the best approach to answer a research questions- be it using predefined OOTB (out of the box) models or bespoke models that we need to train or fine-tune ourselves to produce a suitable approach to a client’s needs.
This section aims to provide a comprehensive overview of the process involved in training and fine-tuning NLP-based machine learning models for classification tasks. It covers everything from data acquisition and labelling to model evaluation and determining when a model is suitable for inference. While technical in nature, this guide remains code-agnostic to focus on concepts and best practices that are universally applicable.
10.1 Understanding the Problem: Defining the Research Question
Before diving into data collection or whirring up a GPU for model training, it’s crucial to have a clear and well-defined research question from the client.The question provided by the client will shape every aspect of the project, from the selection of data sources to the design of labelling schemes and, ultimately, the type of model that is used
Cassie Kozyrkov (former Chief Decision Scientist at Google) brilliantly describes machine learning as “A thing labeller” - a tool that helps make a series of small decisions within the data. For example: Is this email spam? Should we invest more in this campaign? If our task involves labelling, categorizing, or making a decision based on patterns in data, then machine learning may be a good fit. However, if the correct answer can be looked up each time, or if decisions don’t involve subtle patterns in large datasets, ML is likely unnecessary.
10.2 Does the Task Need Machine Learning?
One of the most common misconceptions is that ML is a universal solution, often seen as “magical” by those unfamiliar with its limitations. However, ML isn’t always the right tool for the job. Before we consider building models, we need to be able to clearly articulate what success looks like. If the goals and metrics are not defined, or if there isn’t a need for learning from complex patterns in large amounts of data, ML may not be the answer. ML is not magic, though in many businesses (and non data savvy people, unlike yourself) it is considered okay to skip thinking about what it means to do the task well.
For ML to add value, you should be able to answer the following questions before starting:
- What does it mean to do the task correctly? If the task’s outcome isn’t measurable or there’s no clear definition of what success looks like, an ML solution will struggle to be effective.
- Which mistakes are worse than which other mistakes? In many ML tasks, errors are inevitable, but not all errors are equally harmful. Before building a model, it’s essential to rank the mistakes and decide which ones are more tolerable than others.
- How will we measure the performance at scale? Imagine 1,000 units of work are completed, some of them imperfectly. You need a well-defined method to evaluate the overall performance of that work. Without an evaluation metric, it’s impossible to gauge success or failure meaningfully.
Best Practices for Deciding Whether ML is Needed
1. Is the problem deterministic?If a task can be solved by looking up known answers or using rule-based logic, you don’t need ML. ML is beneficial when patterns need to be learned from data, particularly when those patterns are complex, subtle, or change over time. 2. Can a non-ML solution address the problem efficiently? Consider whether simpler approaches such as basic statistics, heuristics, or existing automation tools can solve the problem. ML should only be used when it offers a clear advantage over these simpler methods. 3. Do we have enough data, and is it labelled correctly? ML models require data, and lots of it, especially for tasks like classification. Moreover, labelled data (with clear examples of the desired output) is crucial for supervised learning. If the data is scarce or poorly labelled, you may need to reconsider whether ML is a viable approach. 4. Can we measure success? Defining evaluation metrics is a must before starting any ML project. If it’s impossible to clearly measure how well a model performs (e.g., accuracy, F1 score, precision-recall, etc.), reconsider whether ML is necessary or if the problem is well-formed.
automating the ineffable