Good morning everyone! I was reading this from Glenn Gabe (my new favourite expert) over the weekend and started worrying that half the traffic in my logs is just robots. AI spiders trawling through my content to fatten their databases (and their wallets).
Should I look into blocking AI bots to conserve my bandwidth?
https://www.gsqi.com/marketing-blog/ai-crawling-visualized-cloudflare/
@olivias I wouldn’t lose too much sleep over it unless your host starts charging you extra for the traffic. Bots are annoying, sure, but they’re everywhere - trying to block every one of them is like playing whack a mole with invisible hammers.
Still, it’s worth knowing who’s visiting. You can check out the GitHub AI bots list, the one Momentic Marketing updates every few months, or Human Security’s big master list if you want to see just how crowded the web really is. It’s pretty wild.
Personally, I keep an eye on my logs, make sure nothing’s eating bandwidth for breakfast, and call it good. Awareness helps but it's easy to become obsessed. Spend that time writing instead ... the bots will be back tomorrow.
Hey Olivia. It's a valid concern but I’m with Rohan on this one. Unless your host is billing you for the extra traffic, I wouldn’t stress too much. I checked my own logs after seeing all the AI bot talk, and yeah, there were plenty of them… but my site didn’t exactly explode.
I've been using Cloudfare for a while now so I set up a few simple rules and called it a day. If they’re not slowing things down or costing money, I’d rather spend the time adding new content. Half these bots come and go faster than my kids’ attention spans anyway.
The web’s always had freeloaders. The trick is to know they’re there, tighten the basics, and then get back to work.
Ooops, only just saw your responses! Thanks both, sounds sensible :-)
I may have a little play around with Cloudfare but I must admit, something always nags at me slightly having to wait for the Cloudfare screen to verify before I am taken to the website I clicked through to.
I wonder how many people click away before it verifies? Rhetorical question!