Some time ago, I designed a WordPress plugin to track WordPress bot traffic, with a strong focus on search engine crawlers.
At the beginning, the design followed a very traditional mindset. Specifically, I wanted accuracy above everything else.
The Original Idea: Trust Nothing but DNS
Instead of relying on User-Agent strings, the plugin performed reverse DNS lookups on every visitor IP.
The goal was simple: verify whether a request truly came from Google, Bing, or other official search engines.
In theory, this logic is solid.
First, User-Agent strings are easy to spoof.
Meanwhile, an IP resolving to an official domain is much harder to fake.
Because of that, the design looked reliable on paper. However, reality turned out to be far more complicated.
Where Reality Broke the Model
The real issues appeared only after deploying the plugin in production.
My site runs behind Cloudflare, which changes the picture entirely.
By the time most requests reach WordPress, the IP address WordPress sees is already a Cloudflare edge node.
As a result, even when a request genuinely comes from Googlebot, the reverse DNS lookup points to Cloudflare — not *.googlebot.com.
Because of this, something critical happened.
A large portion of legitimate search engine traffic was classified as “invalid.”
From a data perspective, that outcome is unacceptable.
After all, the goal was to understand real crawler behavior, not to enforce a verification method that collapses once Cloudflare sits in the middle.
Why Patching Didn’t Make Sense
At that point, I seriously considered patching the original design.
For example, I thought about:
Combining Cloudflare headers
Adding special-case detection rules
Maintaining growing IP and domain whitelists
However, the more I evaluated those ideas, the clearer one conclusion became.
I wasn’t solving the problem — I was fighting the architecture.
Eventually, that realization led me to stop improving that plugin entirely.
Reframing the Question
Instead of asking how to fix the implementation, I went back to a more basic question.
What do I actually want to see?
Surprisingly, the answer was simple and very practical.
First, I want to see search engine bot traffic that truly reaches WordPress.
Second, I want to see mid- and high-risk traffic that has already bypassed Cloudflare defenses.
Most importantly, these two types of traffic share a key characteristic.
They have already entered WordPress.
Because of that, re-verifying identity inside WordPress suddenly felt unnecessary.
A Simpler Division of Responsibility
That’s when the perspective changed.
Instead of duplicating work, responsibilities became clearer.
Cloudflare focuses on:
Blocking obvious abuse
Challenging automation
Filtering large volumes of noise
WordPress focuses on one thing only:
Recording the final outcome.
This shift became the foundation of the new plugin design.
The New Approach: Observe What Gets Through
The redesigned dwith Bot Insight (Lite) no longer tries to prove who is a “real” crawler.
Instead, it follows two very restrained rules.
It records search engine bots only when the User-Agent clearly matches.
It records only mid- and high-risk requests that actually enter WordPress.
As a result, several sources of complexity disappear.
No access.log parsing
No server-level permission requirements
No reverse DNS lookups
No Cloudflare-related misclassification
All logic runs entirely inside the WordPress request lifecycle.
Clearer Data, Better Decisions
Conceptually, the plugin behaves like an inbound result observer.
Cloudflare removes most of the noise. The plugin answers one question only: After all that protection, what still gets through?
In practice, this produces far cleaner results.
Search engine data remains reliable
Risk traffic becomes concentrated and predictable
Firewall rules can be derived from real behavior, not assumptions
This isn’t a “smarter” plugin.
It’s simply a clearer one.
Project source code is available on GitHub: dwith-wp-bot-insight
Leave a Reply