Why UK-trained spam filters fail on Indian scam SMS

Most SMS spam research still trains on the UCI SMS Spam Collection: English messages collected in the UK around 2012. Indian phones get a different kind of fraud: fake KYC expiry notices, UPI collect requests dressed up as refunds, courier customs fees, and "digital arrest" calls from fake police.

ScamShield is a detector built for that context, and it measures three separate problems instead of assuming them away.

A model that has never seen words like UPI, KYC or Aadhaar misses Indian scams. When I tested UK-trained models on Indian messages, every model trained from scratch lost 33–52% of its accuracy. Pretrained DistilBERT lost only 4%.

Scammers write 0TP, b@nk or "v e r i f y" precisely because word-based filters stop recognising the word. ScamShield runs a normaliser that reverses six obfuscation families (leet, look-alike Cyrillic and Greek letters, spaced and dotted letters, invisible characters) before classifying, and shows every rewrite to the user.

The effect is measurable. Under a letter-spacing attack, scam recall for a simple RNN fell to 0.798, and the normaliser brought it back to 0.988. A words-only TF-IDF model went from 0.937 under attack back to 0.987.

A bare "spam" label gives no reason to trust it. ScamShield opens a case file for each message: the verdict with calibrated confidence, a word-level heatmap showing how much each word pushed the verdict, the scam type, a lexical check of any link (links are never opened), and advice on what to do and where to report it.

Messages with a confidence between 0.35 and 0.65 are marked as needing human review rather than forced into a verdict.

I compared eight models: Naive Bayes, logistic regression, a linear SVM, RNN and BiLSTM variants with and without attention, and DistilBERT. On an Indian test set of 6,061 messages, scoring legitimate versus scam, the deployed calibrated linear SVM reached 0.984 accuracy, 0.970 F1 and 0.993 scam recall. DistilBERT reached 0.982 accuracy and 0.968 F1.

I deployed the linear SVM because it matches DistilBERT here, needs no GPU and answers in milliseconds.

Sources were de-duplicated against each other before splitting (5,656 duplicates removed), so no test message also appears in training. Model choice used cross-validation on training data, never the test set. Personal data such as phone numbers, Aadhaar, PAN and account numbers is masked automatically.

• UCI SMS Spam Collection: 4,836 UK messages after de-duplication, used for training and the UK test.
• Mishra and Soni SMS Phishing: 1,053 messages (the rest duplicated UCI).
• Indian Telecom SMS Spam Collection: 1,854 messages for the Indian test and dev sets.
• ScamShield dataset: 6,985 real rows, with machine-generated rows excluded and checked absent.
• 28 published examples from news and PIB Fact Check, and 1,117 templated synthetic messages used only for training.

The public Indian sources only label spam or ham, and their spam mixes marketing with fraud. A documented keyword pass splits it into promotional spam and smishing, and records how each label was set. The Indian test set has 6,061 messages: 1,017 legitimate, 3,758 promotional spam and 1,286 smishing.

• Normalise: reverse six obfuscation families while protecting URLs, amounts, OTP codes, phone numbers and texting slang such as gr8 and 7pm.
• Features: word and character TF-IDF plus 16 engineered features, including urgency words, requests for an OTP or PIN, obfuscation count, a URL look-alike score and Indian scam terms.
• Classify: a calibrated linear SVM, chosen by cross-validation on training data only.
• Explain: remove each word in turn and measure how much the model margin changes, which gives the word-level heatmap.
• Check links: edit distance and substring matching against about 30 official Indian bank and brand domains, plus shorteners, suspicious top-level domains, raw IP hosts and @ tricks.

Each message is classified as legitimate, promotional spam or smishing (SMS phishing). Smishing is labelled with its type: KYC, OTP or account takeover, UPI, courier, prize or lottery, job or task, digital arrest, and phishing. The case file also shows the kill chain behind the message: hook, urgency, claimed authority, then the demand.

Promotions are capped at medium risk: unwanted, but not fraud. Advice is specific to the scam type and points to cybercrime.gov.in and the 1930 helpline.

ScamShield installs as a web app with a share target, so a suspicious SMS can be shared straight from the messaging app into the detector. An awareness quiz shows how the model reads each message, which helps people learn the patterns themselves.