As someone who’s been on the web since the 90s I hate this.
The web was designed to be user agent agnostic. Desktop, phone, fridge, ai agents, curl, python script - whatever agent you are using shouldn’t matter for access. That’s the whole point of open internet, period.
When the Web was first designed, some of the concerns we have today were nonexistent.
I believe in freedom of information, and would love for the information I share to be accessed in any way a given user wants.
But I have to stand defensive and support the author here, too. The modern LLM boom aims to essentially replace original resources with AI-generated summaries step by step. This is detrimental to the Internet, and to knowledge as we know and preserve it.
First, there is an event commonly called Google Zero, which is briefly mentioned in the article. If you don’t know what it is, it is the not-so-hypothetical-anymore moment when Google (or, really, any other large player) essentially accumulates all information on the Web, feeds it to AI, and since then doesn’t serve links anymore, going straight to answers based on training data. Users will jump to this - they already do - because it offers convenience. But for any independent creators it means having no audience, no money, and no means to produce new quality content, trapping users in a self-containing loop that loses nuance, actuality, and truthfulness, and stays under corporate control. This goes beyond cooking recipes and personal notes - it permeates science, political discussion, and much more.
Second, LLMs multiply traffic coming to sites, which becomes an infrastructural problem. Bots access sites at much higher rates than humans do, and when their intent is to scrape your entire website every now and again and there are dozens of them, this becomes huge.
Third, having proprietary models train on the data I provide without any attribution, copyright etc. makes giant corporation profit off my back, while at the same thing making it so that less genuine users will see what I produce. This means careers of authors, journalists etc. are dying, and this also means they are left free to abuse each and every one of us without any consent.
Fourth, and I wonder if you see it by now, LLMs and the way they represent data, along with SEO tools meant to drive information through the search bots, begin to shape how we talk. All I say doesn’t have to be a list of points, yet it is. It could be less verbose, more readable, yet it is the way it is. Because when we interact with the products of such developments too much, we begin shaping our own language in a way that is less human-readable and more meant for machines, without us often being aware of it. This is a real issue of communication.
So, as much as I hate it, I’m gonna protect a lot of the data I share.
I fundamentally disagree with all of your claims. The web was already ruined by SEO farms and I know that cause I worked in organic growth for years, AI is not making this worse.
The traffic argument is non-sensical in 2025. serving 200kb html file costs literally nothing. So if you want to write and share something you can do it without spending a single penny, ever.
I understand the frustration and confusion here but all of this whining is lacking any real vision. Information should be free and accessible to all and the rest can be solved without changing this core principle.
So while you “protect” what you share we all will continue to grow and share information freely and actualy contribute real change not start breaking the looms.
Locking information into corporate-controlled loops is antithetical to freedom and accessibility.
Having singular proprietary point of entry, or even few of them, into the entire knowledge of mankind is not sharing.
This is the part people are willing to protect. Actual peer-to-peer sharing of information, with as little private choke points as possible.
And having the web ruined by SEO is not an argument to keep going. It’s already worse than it should be, and search engines already provide worse quality results than before. This needs to be reversed, not reinforced.
Same. And just because page size is “low” doesn’t mean shit when they’re flooding requests. Try having public research data and watch how much your costs go up just due to load balancer throughput.
I disagree, someone has to pay for the internet, server and electricity to serve your supposedly “free” page. If you are talking about something like GitHub pages? Not really a solution if Microsoft, just…i dont know, got rid of freebies once they have you locked into hosting your content with them? Corpos would never ensure thay have you locked in as a consumer before turning the screws
There are thousands of alternatives to github pages that will host your content for free. Taking your static website to a different host literally takes 5 minutes. At the scale of Github or Netlify your 1gb/mo bandwidth blog website is a rounding error and well worth the potential conversion price. The ignorance in this thread is astounding.
You can “disagree” all you want but reality is that information hosting made huge gains in the past 20 years to the point where the only cost to share a blog these days is the domain name. Even if you don’t want to feed the “free corporate machines” then you can easily host it for 5$/mo and if you can’t afford that I don’t know what to tell you.
Okay sure the internet is free and corpos are always going to give away web hosting…or i do self host and know that someone has to pay the bill despite it being “200 kb”…bruh, they aint giving it away for free, your content and everything you “own” are the price you pay for admission. Start up or mega corp, if you are not paying for it, you are the product
They did have a lot of concerns with abuse though and you can see that in the way the cookies debate went before they were supported in their current form. I think AI crawlers tanking bandwidths for websites and misusing the data they scrape would 100% be something the Mozilla from back then would’ve had concerns over allowing or encouraging.
You’re conflating two different issues. The topic is “for whom the web is for?” not banwidth distribution and optimization.
If LLM bot is being abusive then that’s no different from any other user agent behaving like this and we should expand these protections from intentional/unintentional ddos irrelevant of user agent.
I think your starting point (allowing bot user agents to crawl the web has overlooked benefits) is a good one, but things aren’t black and white–there are clear drawbacks, too. Bots obviously have an orders of magnitude higher potential for abuse; to the point where bot traffic–as it currently stands in the real world–is qualitatively different from human traffic.
we should expand these protections from intentional/unintentional ddos irrelevant of user agent.
Sure, but targeted regulation based on heuristics (in this case, user agent) is also a widely accepted practice. DUI laws exist, even though the goals (fewer murders and safer roads) are already separately regulated.
Would it be nice if we didn’t have to do this? Or there were some other solution? Sure, but I have no idea where to even start, unfortunately.
As someone who’s been on the web since the 90s I hate this.
The web was designed to be user agent agnostic. Desktop, phone, fridge, ai agents, curl, python script - whatever agent you are using shouldn’t matter for access. That’s the whole point of open internet, period.
When the Web was first designed, some of the concerns we have today were nonexistent.
I believe in freedom of information, and would love for the information I share to be accessed in any way a given user wants.
But I have to stand defensive and support the author here, too. The modern LLM boom aims to essentially replace original resources with AI-generated summaries step by step. This is detrimental to the Internet, and to knowledge as we know and preserve it.
First, there is an event commonly called Google Zero, which is briefly mentioned in the article. If you don’t know what it is, it is the not-so-hypothetical-anymore moment when Google (or, really, any other large player) essentially accumulates all information on the Web, feeds it to AI, and since then doesn’t serve links anymore, going straight to answers based on training data. Users will jump to this - they already do - because it offers convenience. But for any independent creators it means having no audience, no money, and no means to produce new quality content, trapping users in a self-containing loop that loses nuance, actuality, and truthfulness, and stays under corporate control. This goes beyond cooking recipes and personal notes - it permeates science, political discussion, and much more.
Second, LLMs multiply traffic coming to sites, which becomes an infrastructural problem. Bots access sites at much higher rates than humans do, and when their intent is to scrape your entire website every now and again and there are dozens of them, this becomes huge.
Third, having proprietary models train on the data I provide without any attribution, copyright etc. makes giant corporation profit off my back, while at the same thing making it so that less genuine users will see what I produce. This means careers of authors, journalists etc. are dying, and this also means they are left free to abuse each and every one of us without any consent.
Fourth, and I wonder if you see it by now, LLMs and the way they represent data, along with SEO tools meant to drive information through the search bots, begin to shape how we talk. All I say doesn’t have to be a list of points, yet it is. It could be less verbose, more readable, yet it is the way it is. Because when we interact with the products of such developments too much, we begin shaping our own language in a way that is less human-readable and more meant for machines, without us often being aware of it. This is a real issue of communication.
So, as much as I hate it, I’m gonna protect a lot of the data I share.
I fundamentally disagree with all of your claims. The web was already ruined by SEO farms and I know that cause I worked in organic growth for years, AI is not making this worse.
The traffic argument is non-sensical in 2025. serving 200kb html file costs literally nothing. So if you want to write and share something you can do it without spending a single penny, ever.
I understand the frustration and confusion here but all of this whining is lacking any real vision. Information should be free and accessible to all and the rest can be solved without changing this core principle.
So while you “protect” what you share we all will continue to grow and share information freely and actualy contribute real change not start breaking the looms.
Locking information into corporate-controlled loops is antithetical to freedom and accessibility.
Having singular proprietary point of entry, or even few of them, into the entire knowledge of mankind is not sharing.
This is the part people are willing to protect. Actual peer-to-peer sharing of information, with as little private choke points as possible.
And having the web ruined by SEO is not an argument to keep going. It’s already worse than it should be, and search engines already provide worse quality results than before. This needs to be reversed, not reinforced.
Instructions unclear, built whole site with nested tables.
Each one had better be in its own iframe.
Open until your server is down because LLM are overloading it
At my company, we had to implement all sorts of WAF rules precisely for that reason. Those things are fucking aggressive.
Same. And just because page size is “low” doesn’t mean shit when they’re flooding requests. Try having public research data and watch how much your costs go up just due to load balancer throughput.
overloading from 200kb of html? We’re not in dialup era anymore
It’s one request Michael, what could it cost? 200kb?
Literally free. So many tech illiterate people in this threat that still think website hosting costs money.
I disagree, someone has to pay for the internet, server and electricity to serve your supposedly “free” page. If you are talking about something like GitHub pages? Not really a solution if Microsoft, just…i dont know, got rid of freebies once they have you locked into hosting your content with them? Corpos would never ensure thay have you locked in as a consumer before turning the screws
There are thousands of alternatives to github pages that will host your content for free. Taking your static website to a different host literally takes 5 minutes. At the scale of Github or Netlify your 1gb/mo bandwidth blog website is a rounding error and well worth the potential conversion price. The ignorance in this thread is astounding.
You can “disagree” all you want but reality is that information hosting made huge gains in the past 20 years to the point where the only cost to share a blog these days is the domain name. Even if you don’t want to feed the “free corporate machines” then you can easily host it for 5$/mo and if you can’t afford that I don’t know what to tell you.
Okay sure the internet is free and corpos are always going to give away web hosting…or i do self host and know that someone has to pay the bill despite it being “200 kb”…bruh, they aint giving it away for free, your content and everything you “own” are the price you pay for admission. Start up or mega corp, if you are not paying for it, you are the product
They did have a lot of concerns with abuse though and you can see that in the way the cookies debate went before they were supported in their current form. I think AI crawlers tanking bandwidths for websites and misusing the data they scrape would 100% be something the Mozilla from back then would’ve had concerns over allowing or encouraging.
You’re conflating two different issues. The topic is “for whom the web is for?” not banwidth distribution and optimization.
If LLM bot is being abusive then that’s no different from any other user agent behaving like this and we should expand these protections from intentional/unintentional ddos irrelevant of user agent.
I think your starting point (allowing bot user agents to crawl the web has overlooked benefits) is a good one, but things aren’t black and white–there are clear drawbacks, too. Bots obviously have an orders of magnitude higher potential for abuse; to the point where bot traffic–as it currently stands in the real world–is qualitatively different from human traffic.
Sure, but targeted regulation based on heuristics (in this case, user agent) is also a widely accepted practice. DUI laws exist, even though the goals (fewer murders and safer roads) are already separately regulated.
Would it be nice if we didn’t have to do this? Or there were some other solution? Sure, but I have no idea where to even start, unfortunately.