Skip to main content

Search

Items tagged with: robotsTXT


#fediAdmin #fediTips #fediVerse

To begin with I wonder what happens if our sites and profiles display CC-BY-SA-NC as #copyright notice. Any use by #AI scrapers should become illegal and indemnisation inforcable.
Also if you search for #robotsTXT in google, this is what you get.

> Ignoring robots.txt instructions can result in your scraping activities being considered unethical or even illegal.

@maxschrems
@markus_netzpolitik
@ankedb
A screen shot from today of a google search for robots.txt.
The informational text for standard questions by google itself reads:
"Is it illegal to scrape a site if there is no robots.txt present?
Ignoring robots. txt instructions can result in your scraping activities being considered unethical or even illegal.3 nov 2023"