Blog Series Overview

Writing test cases in plain English is second nature for most QA professionals—but what if machines could understand and act on those instructions directly? This is where Named Entity Recognition (NER), a powerful Natural Language Processing (NLP) technique, comes into play. By identifying key entities like actions, screens, and objects from everyday language, NER helps bridge the gap between human-readable test steps and machine-executable automation scripts. In this post, we’ll explore how NER in QA can transform natural language test cases into structured data that powers AI-driven test automation.

What is Named Entity Recognition (NER)?

Named Entity Recognition (NER) is a subtask of Natural Language Processing (NLP) that identifies and classifies words or phrases in a sentence into predefined categories. In the context of NLP test automation, NER plays a crucial role in breaking down plain English test steps into actionable components. For example:

• Screens – UI areas like "Login screen" or "Job Order page"
• Actions – Operations like "insert", "submit", "delete"
• Objects – Functional items such as "Job order", "form", or "user"

For more details about NER, check this Wikipedia Link

How I trained my NER model?

To build a truly custom test automation solution, I didn’t rely on generic models. Instead, I fine-tuned a BERT-based Named Entity Recognition (NER) model specifically trained to understand test case language written in natural English.

Why BERT?

I chose bert-base-cased from Hugging Face Transformers as the base model for a few reasons:

• It’s pre-trained on a massive corpus of English (Wikipedia + BookCorpus), giving it a solid grasp of language structure.
• The “cased” version preserves capitalization, which is often useful in UI terms like “Login Screen” vs “login screen”.
• It’s proven to perform well on token-level classification tasks like NER, especially after fine-tuning on a domain-specific dataset.

My Fine-Tuning Approach

I created a small, task-focused dataset where each sentence is a simple test step like:

“Create Job Order Screen”

Each sentence is tokenized and labeled like this:

["Create", "Job", "Order", "Screen"]
["0", "B-ORDER", "I-ORDER", "B-TOOL"]

This tagging format is based on the IOB (Inside-Outside-Beginning) scheme commonly used for NER tasks.

The Training Pipeline

Here’s the overall workflow I followed:

1. Tokenization:
   Used BertTokenizerFast to tokenize inputs while aligning labels with tokenized outputs.
2. Dataset Preparation:
   • Created synthetic but diverse train and test datasets.
   • Used Hugging Face datasets.Dataset class for formatting and transformation.
3. Label Alignment:
   Carefully handled alignment between original tokens and subwords, marking non-aligned subwords with -100 (to be ignored during loss computation).
4. Fine-Tuning:
   Used the Trainer API from Hugging Face with:
   • Learning rate: 2e-5
   • Batch size: 16
   • Epochs: 5 for better convergence
   • Evaluation metrics: Precision, Recall, F1-score using sklearn
5. Model Saving:
   Finally, saved both the fine-tuned model and tokenizer locally for reuse:

model.save_pretrained("./fine_tuned_ner")
tokenizer.save_pretrained("./fine_tuned_ner")

This saved model is now capable of understanding test steps like:

“Open Job Order Screen”
“Open Service Request Screen”
“Initiate Quality Check Screen”

and accurately tagging entities such as:

• B-SCREEN, I-SCREEN – for multi-word actions
• B-TOOL – for tools or execution components

Now that the model is trained and saved, it’s time to put it to work—converting plain English test steps into structured, automation-ready formats.

In the next post, we’ll dive into intent detection—how it complements NER and helps drive even smarter automation pipelines.