My blog (Dwith.com) runs on a pretty common setup: Baota (BT Panel) + WordPress. Checking server logs isn’t hard at all.
Over time, though, I noticed something: I don’t actually want to read all access logs. What I really care about is this:
When do legitimate search engine crawlers show up,
what do they crawl,
and how often do they crawl?The problem is, raw server logs are noisy. Regular visitors, scanners, and bots that fake “Googlebot” all get mixed together. In that mess, the real Google and Bing crawlers are surprisingly hard to track. That makes it tough to monitor crawl health over the long run.
So I came up with a simple idea: I wanted a dedicated place in the WordPress admin to log and review verified, official search engine crawlers only.
I’m not a developer, and I don’t know PHP or WordPress plugin development. I did one thing: I described what I wanted as clearly as possible. We then iterated on the design and implementation until it worked reliably.
The plugin follows one core principle:
Log only verified official search engine crawlers,
with minimal server overhead.To make that happen, the plugin doesn’t rely on User-Agent alone. It verifies crawlers using reverse DNS checks plus forward verification, so the logs reflect real official bots—not imposters.
It currently supports these search engines:
Google
Bing (also covers Yahoo)
Yandex
DuckDuckGoOn the performance side, the plugin caches verification results. That way, the same IP won’t trigger repeated DNS lookups, which keeps resource usage low.
What the Plugin Does
Logs verified official search engine crawler visits only
Keeps the most recent 30 days of logs and auto-cleans older entries
Adds a dedicated log page in the WordPress admin
Filters logs by search engine
Supports page size options: 100 / 200 / 500 / 1000 entries
Shows a 30-day summary (counts + share of total crawls)
Displays a 30-day trend chart with expand/collapse
I didn’t build this to chase features. I built it to turn scattered server-log noise into one clear, easy-to-check view of crawler activity.
For me, the biggest win is simple: I can understand crawl behavior directly in WordPress—no more digging through raw server logs every time.
This project also reminded me of something practical: even if you don’t write code, you can still ship useful tools when your requirements are clear and you keep iterating.
Plugin download: Log Real Search Engine Crawlers Accurately - Wordpress Plugin Download (Deleted)
Version Update - Log-Based Detection Abandoned
Status: This plugin is no longer being actively developed.
The original design identifies bots using server-side request logs, User-Agent inspection, and reverse DNS validation. This approach becomes unreliable in modern deployments-especially when Cloudflare (or similar reverse proxies) is enabled.
Why it’s not a good fit
Edge caching bypasses WordPress: many requests never reach PHP, so the plugin records nothing for those hits.
IP semantics break: the origin often sees Cloudflare edge IPs, making reverse DNS validation invalid.
User-Agent is not a source of truth: UA can be modified or normalized upstream and cannot prove identity.
Security rules block before origin: WAF/managed rules/challenges can stop traffic before WordPress runs.
Conclusion: accurate and complete bot traffic measurement cannot be achieved reliably at the WordPress layer with log-based detection. This is a structural limitation-not something that can be fixed with incremental plugin patches.
Decision: Stop further fixes and enhancements for this plugin.I’ve moved to a simpler and more reliable approach that treats bot detection as an edge-layer responsibility. Read the full rationale here: WordPress Bot Traffic: Why I Abandoned Log-Based Detection
Leave a Reply