Stop AI crawlers: Difference between revisions
mNo edit summary |
No edit summary |
||
| Line 6: | Line 6: | ||
* Nepenthes | * Nepenthes | ||
* Varnish and caching | * Varnish and caching | ||
So, what ''can'' we do? | |||
== Defenses before MediaWiki == | == Defenses before MediaWiki == | ||
| Line 17: | Line 18: | ||
== Defenses in MediaWiki == | == Defenses in MediaWiki == | ||
* [[mediawikiwiki:Extension:Lockdown|Lockdown extension]] - suitable for other purposes in the category of "User Rights". For example, you can block certain swaths of URLs | * [[mediawikiwiki:Extension:Lockdown|Lockdown extension]] - most suitable for other purposes in the category of "User Rights". It '''is useful''' for disallowing anonymous reads of "heavy" pages. For example, you can block certain swaths of URLs in an entire namespace such as all Special pages. It is just not designed for complex filtering. | ||
* StopForumSpam - as the name suggests, suitable for preventing write access (not reads/views). | * StopForumSpam - as the name suggests, suitable for preventing write access (not reads/views). | ||
* [[mediawikiwiki:Extension:AbuseFilter|AbuseFilter extension]]- suitable for setting rules about content editing such as preventing links to specific domains, but not for traffic. | * [[mediawikiwiki:Extension:AbuseFilter|AbuseFilter extension]]- suitable for setting rules about content editing such as preventing links to specific domains, but not for traffic. | ||
| Line 38: | Line 39: | ||
== Discussion == | == Discussion == | ||
[[mw:Handling web crawlers]] | [[mw:Handling web crawlers|Handling web crawlers]] provides details on various solutions, like [[mediawikiwiki:Handling_web_crawlers#Lockdown|how to use Lockdown]] to at least prevent anonymous reads on heavy pages. | ||
== Solution == | == Solution == | ||
We | We tracked this work in [https://github.com/freephile/meza/issues/156 issue 156] | ||
{{References}} | {{References}} | ||