People giving unsolicited advice are not being helpful, just annoying.

They are a scourge.

Looks like Dennis went viral in the activitPub space thx to friendica …

no. it was primarily someone taking a screenshot and posting it. someone who took a screenshot of.. diaspora. while being logged into their geraspora account.

but of course it's a friendica user who also sees nothing wrong about posting unsolicited advice who is making wrong claims.

@utopiArte Your grasp of human psychology, internet culture, and science in general, is weak.

Consider staying off the internet.

(How'd you like that unsolicited advice?)

@ Andreas Geisler - eat more chicken ;)

I am sated, honest.

But some people exist in a mode of constant omnidirectional condescension, like little almighties, looking down in all directions.

Mostly lost causes. Deflating their egos sometimes helps, but usually just makes them worse.

It was fun, but it's time to stop. This post is about LLM-bots being assholes, but that doesn't mean we have to go down to the same levels.

This is the reason why our FOSS project restricted viewing the diffs to logged in accounts. For us some chinese bots have been the main problem - not google or bing.

Just started using this repo for exactly this reason.

@Dennis Schubert Was kann / sollte man dagegen unternehmen?

"don't host web properties for foss projects" seems to be a good advice.

Is it unique to wikis for foss projects?
The silly way they crawl it makes me think this is a general thing happening to every service on the web.
Is there a way to find out/compare whether the crawlers are trying to target specific kinds of things?

so, I should provide some more context to that, I guess. my web server setup isn't "small" by most casual hosters definition. the total traffic usually is always above 5req/s, and this is not an issue for me.

also, crawlers are everywhere. not just those that I mentioned, but also search engine crawlers, and others. a big chunk of my traffic is actually from "bytespider", which is the LLM training bot from the TikTok company. It wasn't mentioned in this post, because although they make a lot of traffic (in terms of traffic size), that's primarily because they also ingest images, and their request volume is generally low.

some spiders are more aggressive than others. a long, long time ago - I've seen a crawler try to enumerate diaspora*s numeric post IDs to crawl everything, but cases like this are rare.

in this case, what made me angry was the fact that they were specifically crawling the edit history of the diaspora wiki. that's odd, because search engines don't care about old content generally. it was also odd, because the request volume was so high, it caused actual issues. MediaWiki isn't the best performance-wise, and especially the history pages are really, really slow. and if you have a crawler firing requests are multiple requests per second, this is bad - and noteworthy.

I've talked privately to others with affected web properties, and it indeed looks like some of those companies have "generic web crawlers", but also specific ones for certain types of software. MediaWiki is frequently affected, and so are phpBB/smf forums, apparently. those crawlers seem to be way more aggressive than their "generic web" counterparts - which might actually just be a bug, who knows.

a few people here, on mastodon, and on other places, have made "suggestions". I've ignored all of them, and I'll continue to ignore all of them. first, suggesting blocking user agent strings or IPs is not a feasible solution, which should be evident to everyone who read my initial post.

I'm also no a huge fan of the idea to feed them trash-content. while there are ways to make that work in a scalable and sustainable fashion, the majority of suggestions I got were along the lines of "use an LLM to generate trash content and feed it to them". this is, sorry for the phrase, quite stupid. I'm not going to engage in a pissing contest with LLM-companies about who can waste more electricity and effort. ultimately, all you do by feeding them trash content is to make stuff slightly more inconvenient - there are easy ways to detect that and to get around that.

for people who post stuff to the internet and who are concerned that their content will be used to train LLMs, I only have one suggestion: use platforms that allow you to distribute content non-publicly, and carefully pick who you share content with. I've gotten a lot of hate a few years ago for categorically rejecting a diaspora feature that would implement a "this post should be visible to every diaspora user, but not search engines" feature, and while that post was written before the age of shitty LLMs, the core remains true: if your stuff is publicly on the internet, there's little you can do. the best thing you can do is be politically engaged and push for clear legislative regulation.

for people who host their own stuff, I also can only offer generic advice. set up rate limits (although be careful, rate limits can easily hurt real users, which is why the wiki had super relaxed rate limits previously). and the biggest advice: don't host things. you'll always be exposed to some kind of abuse - if it's not LLM training bots, it's some chinese or russian botnet trying to DDoS or crawl for vulnerabilities, or some spammer network who want to post viagra ads on your services.

I have deleted one comment in this post because I will not be offering a platform to distribute legal hot takes. If you want legal advice, talk to a lawyer, don't just Google things.

Roger for the advice for people who post stuff to the internet and who are concerned that their content will be used to train LLMs, I only have one suggestion: use platforms that allow you to distribute content non-publicly, and carefully pick who you share content with. and thanks @Dennis Schubert

Maybe worth a try: Nepenthes

https://www.404media.co/email/7a39d947-4a4a-42bc-bbcf-3379f112c999/