Stop AI crawlers: Difference between revisions

Revision as of 02:02, 25 November 2025

AI crawlers from all over the world have become a huge problem. They don't play by the (Robots.txt) rules, so it's even worse than old-school indexing by Bing, Google, and Yahoo which were bad enough.

In his 2025 MediaWiki User and Developer Workshop presentation^[1] Jeffrey Wang mentions some approaches as inadequate:

Fail2ban
Nepenthes
Varnish and caching

So, what can we do?

Defenses before MediaWiki[edit]

WAF e.g. Cloudflare - the Content Distribution Network (CDN) company offers a Web Application Firewall (WAF) product^[2] to stop network attacks. )
Filtering reverse proxies
Anubis - their README claims the solution to be over-zealous, but then offers default configurations that would appear to expressly allow the good guys like Internet Archive, bing and google^[3].

Defenses in MediaWiki[edit]

Lockdown extension - most suitable for other purposes in the category of "User Rights". It is useful for disallowing anonymous reads of "heavy" pages. For example, you can block certain swaths of URLs in an entire namespace such as all Special pages - although you need to list each SpecialPage individually in your configuration (see example below). It is just not designed for complex filtering.
StopForumSpam - as the name suggests, suitable for preventing write access (not reads/views).
AbuseFilter extension- suitable for setting rules about content editing such as preventing links to specific domains, but not for traffic.
CrawlerProtection extension- by MyWikis' Jeffrey Wang. Currently has a PR by vedmaka that would enable a list of SpecialPages to deny anonymous access to. I submitted another PR incorporating the suggestions therein.

Problematic pages in MediaWiki[edit]

SpecialPages
- WhatLinksHere
- RecentChangesLinked
History
Arbitrary Diffs
The 'ABCD' special pages
- SMW
  - Ask
  - BrowseData
- Cargo
  - CargoQuery
  - Drilldown

Discussion[edit]

Handling web crawlers provides details on various solutions, like how to use Lockdown to at least prevent anonymous reads on heavy pages.

Solution[edit]

We tracked this work in issue 156

Listing of all Special Pages on this wiki. We obtain the list by querying the API with curl -s "[YOURWIKI]api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json" | jq -r '.query.specialpagealiases[].aliases[]' | sort

By querying your own API, you get SpecialPages that are added by the extensions unique to your wiki versus just MediaWiki core.

Items marked with ✅ are suggested to be allowed for anonymous users rather than blocked.

AbuseFilter
AbuseLog
AccountSecurity
ActiveUsers
AddData
AddPage
AdminLinks
AllMessages
AllMyFiles
AllMyUploads
AllPages
AncientPages
ApiHelp
ApiSandbox
Ask
AuthenticationPopupSuccess
AutoblockList
BatchUpload
BetaFeatures
BlankPage
Block
BlockedExternalDomains
BlockIP
BlockList
BlockUser
Book
BookSources
BotPasswords
BrokenLinks
BrokenRedirects
✅ Browse
BrowseData
Captcha
✅ Categories
CategoryTree
ChangeContentModel
ChangeCredentials
ChangeEmail
ChangePassword
CheckUser
CheckUserLog
Cite
CiteThisPage
ClearPendingReviews
Collab_Pad
CollabPad
Collection
ComparePages
Concepts
✅ ConfirmEmail
ConstraintErrorList
Contribs
Contribute
Contributions
ContributionScores
CreateAccount
CreateCategory
CreateClass
CreateForm
CreateProperty
CreateRedirect
CreateTemplate
DeadendPages
Delete
DeleteBatch
DeletedContributions
DeletePage
Diff
DisableOATHForUser
DiscussionToolsDebug
DisplayNotificationsConfiguration
DoubleRedirects
Edit
EditData
EditDiagram
EditPage
EditSchema
EditTags
EditWatchlist
Email
EmailUser
EnableFlow
EnableStructuredDiscussions
ExpandTemplates
Export
ExportRDF
FacetedSearch
FewestRevisions
FileDuplicateSearch
FileList
FilePath
FindComment
Flow
FormEdit
Forms
FormStart
Gadgets
GadgetUsage
GeneratePages
GetData
GoToComment
GoToInterwiki
GraphSandbox
History
History
ImageList
Import
ImportCSV
ImportSpreadsheet
ImportXML
Info
Interwiki
InvalidateEmail
Investigate
InvestigateBlock
IPBlockList
LinkAccounts
LinkSearch
LintErrors
ListAdmins
ListAutoblocks
ListBlocks
ListBots
ListDuplicatedFiles
ListFileDuplicates
ListFiles
ListGrants
ListGroupRights
ListRedirects
ListUsers
LockDB
Log
✅ Login
Logout
Logs
LonelyPages
LongPages
MakeBot
MakeSysop
Manage_Two-factor_authentication
MathShowImage
MathStatus
MediaStatistics
MergeHistory
MIMESearch
MissingRedirectAnnotations
MobileDiff
MobileLanguages
MobileOptions
MostCategories
MostFiles
MostImages
MostInterwikis
MostLinked
MostLinkedCategories
MostLinkedFiles
MostLinkedPages
MostLinkedTemplates
MostRevisions
MostTranscludedPages
MostUsedCategories
MostUsedTemplates
MovePage
MultiPageEdit
MyContribs
MyContributions
MyFiles
MyLanguage
MyLog
MyPage
MyTalk
MyUploads
NamespaceInfo
NewFiles
NewImages
NewPages
Newsection
NewSection
Notifications
NotificationsMarkRead
Nuke
OATH
OATHAuth
OATH_Manage
OATHManage
OrphanedPages
Packages
PageData
PageHistory
PageInfo
PageProperty
Pagesbyprop
PagesByProp
PageStatistics
Pageswithprop
PagesWithProp
PasswordPolicies
✅ PasswordReset
PendingReviews
PendingTaskList
PermaLink
PermanentLink
✅ PluggableAuthLogin
PluggableAuthLogout
Preferences
✅ PrefixIndex
ProcessingErrorList
Properties
PropertyLabelSimilarity
Protect
ProtectedPages
ProtectedTitles
ProtectPage
Purge
✅ Random
RandomInCategory
✅ RandomPage
RandomRedirect
RandomRootpage
RecentChanges
RecentChangesLinked
Redirect
RelatedChanges
RemoveCredentials
RenameUser
ReplaceText
✅ ResetPass
✅ ResetPassword
ResetTokens
RevisionDelete
RunJobs
RunQuery
✅ Search
✅ SearchByProperty
SemanticMediaWiki
ShortPages
SMWAdmin
✅ SpecialPages
✅ Statistics
✅ Stats
StructuredDiscussions
Tags
TalkPage
Templates
Thanks
TopicSubscriptions
TrackingCategories
Two-factor_authentication
Types
Unblock
UncategorizedCategories
UncategorizedFiles
UncategorizedImages
UncategorizedPages
UncategorizedTemplates
Undelete
UnlinkAccounts
UnlockDB
UnusedCategories
UnusedFiles
UnusedImages
UnusedProperties
UnusedTemplates
UnwatchedPages
Upload
Uploads
UploadStash
UploadWindow
URIResolver
UserGroupRights
UserList
✅ UserLogin
UserLogout
UserMerge
UserRights
Users
VerifyOATHForUser
✅ Version
ViewXML
WantedCategories
WantedFiles
WantedPages
WantedProperties
WantedTemplates
WatchAnalytics
Watchlist
WebChat
WhatLinksHere
WhoIsWatching
WithoutInterwiki

Assuming CrawlerProtection is updated with my PR, you can just configure that extension with the list above.

You can achieve this restriction using the Lockdown extension with this configuration:

$wgSpecialPageLockdown['AbuseFilter'] = ['user'];
$wgSpecialPageLockdown['AbuseLog'] = ['user'];
$wgSpecialPageLockdown['AccountSecurity'] = ['user'];
$wgSpecialPageLockdown['ActiveUsers'] = ['user'];
$wgSpecialPageLockdown['AddData'] = ['user'];
$wgSpecialPageLockdown['AddPage'] = ['user'];
$wgSpecialPageLockdown['AdminLinks'] = ['user'];
$wgSpecialPageLockdown['AllMessages'] = ['user'];
$wgSpecialPageLockdown['AllMyFiles'] = ['user'];
$wgSpecialPageLockdown['AllMyUploads'] = ['user'];
$wgSpecialPageLockdown['AllPages'] = ['user'];
$wgSpecialPageLockdown['AncientPages'] = ['user'];
$wgSpecialPageLockdown['ApiHelp'] = ['user'];
$wgSpecialPageLockdown['ApiSandbox'] = ['user'];
$wgSpecialPageLockdown['Ask'] = ['user'];
$wgSpecialPageLockdown['AuthenticationPopupSuccess'] = ['user'];
$wgSpecialPageLockdown['AutoblockList'] = ['user'];
$wgSpecialPageLockdown['BatchUpload'] = ['user'];
$wgSpecialPageLockdown['BetaFeatures'] = ['user'];
$wgSpecialPageLockdown['BlankPage'] = ['user'];
$wgSpecialPageLockdown['Block'] = ['user'];
$wgSpecialPageLockdown['BlockedExternalDomains'] = ['user'];
$wgSpecialPageLockdown['BlockIP'] = ['user'];
$wgSpecialPageLockdown['BlockList'] = ['user'];
$wgSpecialPageLockdown['BlockUser'] = ['user'];
$wgSpecialPageLockdown['Book'] = ['user'];
$wgSpecialPageLockdown['BookSources'] = ['user'];
$wgSpecialPageLockdown['BotPasswords'] = ['user'];
$wgSpecialPageLockdown['BrokenLinks'] = ['user'];
$wgSpecialPageLockdown['BrokenRedirects'] = ['user'];
$wgSpecialPageLockdown['BrowseData'] = ['user'];
$wgSpecialPageLockdown['Captcha'] = ['user'];
$wgSpecialPageLockdown['CategoryTree'] = ['user'];
$wgSpecialPageLockdown['ChangeContentModel'] = ['user'];
$wgSpecialPageLockdown['ChangeCredentials'] = ['user'];
$wgSpecialPageLockdown['ChangeEmail'] = ['user'];
$wgSpecialPageLockdown['ChangePassword'] = ['user'];
$wgSpecialPageLockdown['CheckUser'] = ['user'];
$wgSpecialPageLockdown['CheckUserLog'] = ['user'];
$wgSpecialPageLockdown['Cite'] = ['user'];
$wgSpecialPageLockdown['CiteThisPage'] = ['user'];
$wgSpecialPageLockdown['ClearPendingReviews'] = ['user'];
$wgSpecialPageLockdown['Collab_Pad'] = ['user'];
$wgSpecialPageLockdown['CollabPad'] = ['user'];
$wgSpecialPageLockdown['Collection'] = ['user'];
$wgSpecialPageLockdown['ComparePages'] = ['user'];
$wgSpecialPageLockdown['Concepts'] = ['user'];
$wgSpecialPageLockdown['ConstraintErrorList'] = ['user'];
$wgSpecialPageLockdown['Contribs'] = ['user'];
$wgSpecialPageLockdown['Contribute'] = ['user'];
$wgSpecialPageLockdown['Contributions'] = ['user'];
$wgSpecialPageLockdown['ContributionScores'] = ['user'];
$wgSpecialPageLockdown['CreateAccount'] = ['user'];
$wgSpecialPageLockdown['CreateCategory'] = ['user'];
$wgSpecialPageLockdown['CreateClass'] = ['user'];
$wgSpecialPageLockdown['CreateForm'] = ['user'];
$wgSpecialPageLockdown['CreateProperty'] = ['user'];
$wgSpecialPageLockdown['CreateRedirect'] = ['user'];
$wgSpecialPageLockdown['CreateTemplate'] = ['user'];
$wgSpecialPageLockdown['DeadendPages'] = ['user'];
$wgSpecialPageLockdown['Delete'] = ['user'];
$wgSpecialPageLockdown['DeleteBatch'] = ['user'];
$wgSpecialPageLockdown['DeletedContributions'] = ['user'];
$wgSpecialPageLockdown['DeletePage'] = ['user'];
$wgSpecialPageLockdown['Diff'] = ['user'];
$wgSpecialPageLockdown['DisableOATHForUser'] = ['user'];
$wgSpecialPageLockdown['DiscussionToolsDebug'] = ['user'];
$wgSpecialPageLockdown['DisplayNotificationsConfiguration'] = ['user'];
$wgSpecialPageLockdown['DoubleRedirects'] = ['user'];
$wgSpecialPageLockdown['Edit'] = ['user'];
$wgSpecialPageLockdown['EditData'] = ['user'];
$wgSpecialPageLockdown['EditDiagram'] = ['user'];
$wgSpecialPageLockdown['EditPage'] = ['user'];
$wgSpecialPageLockdown['EditSchema'] = ['user'];
$wgSpecialPageLockdown['EditTags'] = ['user'];
$wgSpecialPageLockdown['EditWatchlist'] = ['user'];
$wgSpecialPageLockdown['Email'] = ['user'];
$wgSpecialPageLockdown['EmailUser'] = ['user'];
$wgSpecialPageLockdown['EnableFlow'] = ['user'];
$wgSpecialPageLockdown['EnableStructuredDiscussions'] = ['user'];
$wgSpecialPageLockdown['ExpandTemplates'] = ['user'];
$wgSpecialPageLockdown['Export'] = ['user'];
$wgSpecialPageLockdown['ExportRDF'] = ['user'];
$wgSpecialPageLockdown['FacetedSearch'] = ['user'];
$wgSpecialPageLockdown['FewestRevisions'] = ['user'];
$wgSpecialPageLockdown['FileDuplicateSearch'] = ['user'];
$wgSpecialPageLockdown['FileList'] = ['user'];
$wgSpecialPageLockdown['FilePath'] = ['user'];
$wgSpecialPageLockdown['FindComment'] = ['user'];
$wgSpecialPageLockdown['Flow'] = ['user'];
$wgSpecialPageLockdown['FormEdit'] = ['user'];
$wgSpecialPageLockdown['Forms'] = ['user'];
$wgSpecialPageLockdown['FormStart'] = ['user'];
$wgSpecialPageLockdown['Gadgets'] = ['user'];
$wgSpecialPageLockdown['GadgetUsage'] = ['user'];
$wgSpecialPageLockdown['GeneratePages'] = ['user'];
$wgSpecialPageLockdown['GetData'] = ['user'];
$wgSpecialPageLockdown['GoToComment'] = ['user'];
$wgSpecialPageLockdown['GoToInterwiki'] = ['user'];
$wgSpecialPageLockdown['GraphSandbox'] = ['user'];
$wgSpecialPageLockdown['History'] = ['user'];
$wgSpecialPageLockdown['History'] = ['user'];
$wgSpecialPageLockdown['ImageList'] = ['user'];
$wgSpecialPageLockdown['Import'] = ['user'];
$wgSpecialPageLockdown['ImportCSV'] = ['user'];
$wgSpecialPageLockdown['ImportSpreadsheet'] = ['user'];
$wgSpecialPageLockdown['ImportXML'] = ['user'];
$wgSpecialPageLockdown['Info'] = ['user'];
$wgSpecialPageLockdown['Interwiki'] = ['user'];
$wgSpecialPageLockdown['InvalidateEmail'] = ['user'];
$wgSpecialPageLockdown['Investigate'] = ['user'];
$wgSpecialPageLockdown['InvestigateBlock'] = ['user'];
$wgSpecialPageLockdown['IPBlockList'] = ['user'];
$wgSpecialPageLockdown['LinkAccounts'] = ['user'];
$wgSpecialPageLockdown['LinkSearch'] = ['user'];
$wgSpecialPageLockdown['LintErrors'] = ['user'];
$wgSpecialPageLockdown['ListAdmins'] = ['user'];
$wgSpecialPageLockdown['ListAutoblocks'] = ['user'];
$wgSpecialPageLockdown['ListBlocks'] = ['user'];
$wgSpecialPageLockdown['ListBots'] = ['user'];
$wgSpecialPageLockdown['ListDuplicatedFiles'] = ['user'];
$wgSpecialPageLockdown['ListFileDuplicates'] = ['user'];
$wgSpecialPageLockdown['ListFiles'] = ['user'];
$wgSpecialPageLockdown['ListGrants'] = ['user'];
$wgSpecialPageLockdown['ListGroupRights'] = ['user'];
$wgSpecialPageLockdown['ListRedirects'] = ['user'];
$wgSpecialPageLockdown['ListUsers'] = ['user'];
$wgSpecialPageLockdown['LockDB'] = ['user'];
$wgSpecialPageLockdown['Log'] = ['user'];
$wgSpecialPageLockdown['Logout'] = ['user'];
$wgSpecialPageLockdown['Logs'] = ['user'];
$wgSpecialPageLockdown['LonelyPages'] = ['user'];
$wgSpecialPageLockdown['LongPages'] = ['user'];
$wgSpecialPageLockdown['MakeBot'] = ['user'];
$wgSpecialPageLockdown['MakeSysop'] = ['user'];
$wgSpecialPageLockdown['Manage_Two-factor_authentication'] = ['user'];
$wgSpecialPageLockdown['MathShowImage'] = ['user'];
$wgSpecialPageLockdown['MathStatus'] = ['user'];
$wgSpecialPageLockdown['MediaStatistics'] = ['user'];
$wgSpecialPageLockdown['MergeHistory'] = ['user'];
$wgSpecialPageLockdown['MIMESearch'] = ['user'];
$wgSpecialPageLockdown['MissingRedirectAnnotations'] = ['user'];
$wgSpecialPageLockdown['MobileDiff'] = ['user'];
$wgSpecialPageLockdown['MobileLanguages'] = ['user'];
$wgSpecialPageLockdown['MobileOptions'] = ['user'];
$wgSpecialPageLockdown['MostCategories'] = ['user'];
$wgSpecialPageLockdown['MostFiles'] = ['user'];
$wgSpecialPageLockdown['MostImages'] = ['user'];
$wgSpecialPageLockdown['MostInterwikis'] = ['user'];
$wgSpecialPageLockdown['MostLinked'] = ['user'];
$wgSpecialPageLockdown['MostLinkedCategories'] = ['user'];
$wgSpecialPageLockdown['MostLinkedFiles'] = ['user'];
$wgSpecialPageLockdown['MostLinkedPages'] = ['user'];
$wgSpecialPageLockdown['MostLinkedTemplates'] = ['user'];
$wgSpecialPageLockdown['MostRevisions'] = ['user'];
$wgSpecialPageLockdown['MostTranscludedPages'] = ['user'];
$wgSpecialPageLockdown['MostUsedCategories'] = ['user'];
$wgSpecialPageLockdown['MostUsedTemplates'] = ['user'];
$wgSpecialPageLockdown['MovePage'] = ['user'];
$wgSpecialPageLockdown['MultiPageEdit'] = ['user'];
$wgSpecialPageLockdown['MyContribs'] = ['user'];
$wgSpecialPageLockdown['MyContributions'] = ['user'];
$wgSpecialPageLockdown['MyFiles'] = ['user'];
$wgSpecialPageLockdown['MyLanguage'] = ['user'];
$wgSpecialPageLockdown['MyLog'] = ['user'];
$wgSpecialPageLockdown['MyPage'] = ['user'];
$wgSpecialPageLockdown['MyTalk'] = ['user'];
$wgSpecialPageLockdown['MyUploads'] = ['user'];
$wgSpecialPageLockdown['NamespaceInfo'] = ['user'];
$wgSpecialPageLockdown['NewFiles'] = ['user'];
$wgSpecialPageLockdown['NewImages'] = ['user'];
$wgSpecialPageLockdown['NewPages'] = ['user'];
$wgSpecialPageLockdown['Newsection'] = ['user'];
$wgSpecialPageLockdown['NewSection'] = ['user'];
$wgSpecialPageLockdown['Notifications'] = ['user'];
$wgSpecialPageLockdown['NotificationsMarkRead'] = ['user'];
$wgSpecialPageLockdown['Nuke'] = ['user'];
$wgSpecialPageLockdown['OATH'] = ['user'];
$wgSpecialPageLockdown['OATHAuth'] = ['user'];
$wgSpecialPageLockdown['OATH_Manage'] = ['user'];
$wgSpecialPageLockdown['OATHManage'] = ['user'];
$wgSpecialPageLockdown['OrphanedPages'] = ['user'];
$wgSpecialPageLockdown['Packages'] = ['user'];
$wgSpecialPageLockdown['PageData'] = ['user'];
$wgSpecialPageLockdown['PageHistory'] = ['user'];
$wgSpecialPageLockdown['PageInfo'] = ['user'];
$wgSpecialPageLockdown['PageProperty'] = ['user'];
$wgSpecialPageLockdown['Pagesbyprop'] = ['user'];
$wgSpecialPageLockdown['PagesByProp'] = ['user'];
$wgSpecialPageLockdown['PageStatistics'] = ['user'];
$wgSpecialPageLockdown['Pageswithprop'] = ['user'];
$wgSpecialPageLockdown['PagesWithProp'] = ['user'];
$wgSpecialPageLockdown['PasswordPolicies'] = ['user'];
$wgSpecialPageLockdown['PendingReviews'] = ['user'];
$wgSpecialPageLockdown['PendingTaskList'] = ['user'];
$wgSpecialPageLockdown['PermaLink'] = ['user'];
$wgSpecialPageLockdown['PermanentLink'] = ['user'];
$wgSpecialPageLockdown['PluggableAuthLogout'] = ['user'];
$wgSpecialPageLockdown['Preferences'] = ['user'];
$wgSpecialPageLockdown['ProcessingErrorList'] = ['user'];
$wgSpecialPageLockdown['Properties'] = ['user'];
$wgSpecialPageLockdown['PropertyLabelSimilarity'] = ['user'];
$wgSpecialPageLockdown['Protect'] = ['user'];
$wgSpecialPageLockdown['ProtectedPages'] = ['user'];
$wgSpecialPageLockdown['ProtectedTitles'] = ['user'];
$wgSpecialPageLockdown['ProtectPage'] = ['user'];
$wgSpecialPageLockdown['Purge'] = ['user'];
$wgSpecialPageLockdown['RandomInCategory'] = ['user'];
$wgSpecialPageLockdown['RandomRedirect'] = ['user'];
$wgSpecialPageLockdown['RandomRootpage'] = ['user'];
$wgSpecialPageLockdown['RecentChanges'] = ['user'];
$wgSpecialPageLockdown['RecentChangesLinked'] = ['user'];
$wgSpecialPageLockdown['Redirect'] = ['user'];
$wgSpecialPageLockdown['RelatedChanges'] = ['user'];
$wgSpecialPageLockdown['RemoveCredentials'] = ['user'];
$wgSpecialPageLockdown['RenameUser'] = ['user'];
$wgSpecialPageLockdown['ReplaceText'] = ['user'];
$wgSpecialPageLockdown['ResetTokens'] = ['user'];
$wgSpecialPageLockdown['RevisionDelete'] = ['user'];
$wgSpecialPageLockdown['RunJobs'] = ['user'];
$wgSpecialPageLockdown['RunQuery'] = ['user'];
$wgSpecialPageLockdown['SemanticMediaWiki'] = ['user'];
$wgSpecialPageLockdown['ShortPages'] = ['user'];
$wgSpecialPageLockdown['SMWAdmin'] = ['user'];
$wgSpecialPageLockdown['StructuredDiscussions'] = ['user'];
$wgSpecialPageLockdown['Tags'] = ['user'];
$wgSpecialPageLockdown['TalkPage'] = ['user'];
$wgSpecialPageLockdown['Templates'] = ['user'];
$wgSpecialPageLockdown['Thanks'] = ['user'];
$wgSpecialPageLockdown['TopicSubscriptions'] = ['user'];
$wgSpecialPageLockdown['TrackingCategories'] = ['user'];
$wgSpecialPageLockdown['Two-factor_authentication'] = ['user'];
$wgSpecialPageLockdown['Types'] = ['user'];
$wgSpecialPageLockdown['Unblock'] = ['user'];
$wgSpecialPageLockdown['UncategorizedCategories'] = ['user'];
$wgSpecialPageLockdown['UncategorizedFiles'] = ['user'];
$wgSpecialPageLockdown['UncategorizedImages'] = ['user'];
$wgSpecialPageLockdown['UncategorizedPages'] = ['user'];
$wgSpecialPageLockdown['UncategorizedTemplates'] = ['user'];
$wgSpecialPageLockdown['Undelete'] = ['user'];
$wgSpecialPageLockdown['UnlinkAccounts'] = ['user'];
$wgSpecialPageLockdown['UnlockDB'] = ['user'];
$wgSpecialPageLockdown['UnusedCategories'] = ['user'];
$wgSpecialPageLockdown['UnusedFiles'] = ['user'];
$wgSpecialPageLockdown['UnusedImages'] = ['user'];
$wgSpecialPageLockdown['UnusedProperties'] = ['user'];
$wgSpecialPageLockdown['UnusedTemplates'] = ['user'];
$wgSpecialPageLockdown['UnwatchedPages'] = ['user'];
$wgSpecialPageLockdown['Upload'] = ['user'];
$wgSpecialPageLockdown['Uploads'] = ['user'];
$wgSpecialPageLockdown['UploadStash'] = ['user'];
$wgSpecialPageLockdown['UploadWindow'] = ['user'];
$wgSpecialPageLockdown['URIResolver'] = ['user'];
$wgSpecialPageLockdown['UserGroupRights'] = ['user'];
$wgSpecialPageLockdown['UserList'] = ['user'];
$wgSpecialPageLockdown['UserLogout'] = ['user'];
$wgSpecialPageLockdown['UserMerge'] = ['user'];
$wgSpecialPageLockdown['UserRights'] = ['user'];
$wgSpecialPageLockdown['Users'] = ['user'];
$wgSpecialPageLockdown['VerifyOATHForUser'] = ['user'];
$wgSpecialPageLockdown['ViewXML'] = ['user'];
$wgSpecialPageLockdown['WantedCategories'] = ['user'];
$wgSpecialPageLockdown['WantedFiles'] = ['user'];
$wgSpecialPageLockdown['WantedPages'] = ['user'];
$wgSpecialPageLockdown['WantedProperties'] = ['user'];
$wgSpecialPageLockdown['WantedTemplates'] = ['user'];
$wgSpecialPageLockdown['WatchAnalytics'] = ['user'];
$wgSpecialPageLockdown['Watchlist'] = ['user'];
$wgSpecialPageLockdown['WebChat'] = ['user'];
$wgSpecialPageLockdown['WhatLinksHere'] = ['user'];
$wgSpecialPageLockdown['WhoIsWatching'] = ['user'];
$wgSpecialPageLockdown['WithoutInterwiki'] = ['user'];

References[edit]

[1] ttps://www.youtube.com/watch?v=VGS5l3YH2oY

[2] ttps://www.cloudflare.com/application-services/products/waf/ https://www.cloudflare.com/lp/waf-product-brief-xy/

[3] ttps://github.com/TecharoHQ/anubis/tree/main/data/crawlers

[1]

[2]

[3]

@@ Line 21: / Line 21: @@
 * StopForumSpam - as the name suggests, suitable for preventing write access (not reads/views).
 * [[mediawikiwiki:Extension:AbuseFilter|AbuseFilter extension]]- suitable for setting rules about content editing such as preventing links to specific domains, but not for traffic.
-* [[mediawikiwiki:Extension:CrawlerProtection|CrawlerProtection extension]]- by MyWikis' Jeffrey Wang. Currently has [https://github.com/mywikis/CrawlerProtection/pull/10 a PR by vedmaka] that would enable a list of SpecialPages to deny anonymous access to.
+* [[mediawikiwiki:Extension:CrawlerProtection|CrawlerProtection extension]]- by MyWikis' Jeffrey Wang. Currently has [https://github.com/mywikis/CrawlerProtection/pull/10 a PR by vedmaka] that would enable a list of SpecialPages to deny anonymous access to. I submitted [https://github.com/mywikis/CrawlerProtection/pull/12 another PR] incorporating the suggestions therein.
 == Problematic pages in MediaWiki ==
@@ Line 342: / Line 342: @@
 # WithoutInterwiki
+Assuming '''CrawlerProtection''' is updated with [https://github.com/mywikis/CrawlerProtection/pull/12 my PR], you can just configure that extension with the list above.
 {{Collapsible
 |visible_text=You can achieve this restriction using the Lockdown extension with this configuration: