Stop AI crawlers

From Freephile Wiki
Revision as of 14:53, 4 December 2025 by Admin (talk | contribs) (add heading for AllowList)

AI crawlers from all over the world have become a huge problem. They don't play by the (Robots.txt) rules, so it's even worse than old-school indexing by Bing, Google, and Yahoo which were bad enough.

In his 2025 MediaWiki User and Developer Workshop presentation[1] Jeffrey Wang mentions some approaches as inadequate:

  • Fail2ban
  • Nepenthes
  • Varnish and caching

So, what can we do?

Defenses before MediaWiki[edit]

  • WAF e.g. Cloudflare - the Content Distribution Network (CDN) company offers a Web Application Firewall (WAF) product[2] to stop network attacks. )
  • Filtering reverse proxies
  • Anubis - their README claims the solution to be over-zealous, but then offers default configurations that would appear to expressly allow the good guys like Internet Archive, bing and google[3].

Defenses in MediaWiki[edit]

really just the two in bold:

  • Lockdown extension - most suitable for other purposes in the category of "User Rights". It is useful for disallowing anonymous reads of "heavy" pages. For example, you can block certain swaths of URLs in an entire namespace such as all Special pages - although you need to list each SpecialPage individually in your configuration (see example below). It is just not designed for complex filtering.
  • StopForumSpam - as the name suggests, suitable for preventing write access (not reads/views).
  • AbuseFilter extension- suitable for setting rules about content editing such as preventing links to specific domains, but not for traffic.
  • CrawlerProtection extension- by MyWikis' Jeffrey Wang. Currently has a PR by vedmaka that would enable a list of SpecialPages to deny anonymous access to. I submitted another PR incorporating the suggestions therein.

Problematic pages in MediaWiki[edit]

  • SpecialPages
    • WhatLinksHere
    • RecentChangesLinked
  • History
  • Arbitrary Diffs
  • The 'ABCD' special pages
    • SMW
      • Ask
      • BrowseData
    • Cargo
      • CargoQuery
      • Drilldown

Discussion[edit]

Handling web crawlers provides details on various solutions, like how to use Lockdown to at least prevent anonymous reads on heavy pages.

Solution[edit]

We tracked this work in issue 156

Listing of all Special Pages on this wiki. We obtain the list by querying the API with curl -s "[YOURWIKI]api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json" | jq -r '.query.specialpagealiases[].aliases[]' | sort

By querying your own API, you get SpecialPages that are added by the extensions unique to your wiki versus just MediaWiki core.

Other ways to list SpecialPages via the API[edit]

# Get all aliases as a flat array
curl -s "https://wiki.freephile.org/wiki/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json" | \
  jq -r '.query.specialpagealiases[].aliases[]'

# Get aliases grouped by special page (more useful)
curl -s "https://wiki.freephile.org/wiki/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json" | \
  jq '.query.specialpagealiases[] | {realname, aliases}'

# Get a clean JSON array of just the alias strings
curl -s "https://wiki.freephile.org/wiki/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json" | \
  jq '[.query.specialpagealiases[].aliases[]]'

# Get unique aliases (in case there are duplicates)
curl -s "https://wiki.freephile.org/wiki/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json" | \
  jq '[.query.specialpagealiases[].aliases[]] | unique'

Allow List of SpecialPages[edit]

Items marked with ✅ are suggested to be allowed for anonymous users rather than blocked.

  1. AbuseFilter
  2. AbuseLog
  3. AccountSecurity
  4. ActiveUsers
  5. AddData
  6. AddPage
  7. AdminLinks
  8. AllMessages
  9. AllMyFiles
  10. AllMyUploads
  11. AllPages
  12. AncientPages
  13. ApiHelp
  14. ApiSandbox
  15. Ask
  16. AuthenticationPopupSuccess
  17. AutoblockList
  18. BatchUpload
  19. BetaFeatures
  20. BlankPage
  21. Block
  22. BlockedExternalDomains
  23. BlockIP
  24. BlockList
  25. BlockUser
  26. Book
  27. BookSources
  28. BotPasswords
  29. BrokenLinks
  30. BrokenRedirects
  31. ✅ Browse
  32. BrowseData
  33. Captcha
  34. ✅ Categories
  35. CategoryTree
  36. ChangeContentModel
  37. ChangeCredentials
  38. ChangeEmail
  39. ChangePassword
  40. CheckUser
  41. CheckUserLog
  42. Cite
  43. CiteThisPage
  44. ClearPendingReviews
  45. Collab_Pad
  46. CollabPad
  47. Collection
  48. ComparePages
  49. Concepts
  50. ✅ ConfirmEmail
  51. ConstraintErrorList
  52. Contribs
  53. Contribute
  54. Contributions
  55. ContributionScores
  56. CreateAccount
  57. CreateCategory
  58. CreateClass
  59. CreateForm
  60. CreateProperty
  61. CreateRedirect
  62. CreateTemplate
  63. DeadendPages
  64. Delete
  65. DeleteBatch
  66. DeletedContributions
  67. DeletePage
  68. Diff
  69. DisableOATHForUser
  70. DiscussionToolsDebug
  71. DisplayNotificationsConfiguration
  72. DoubleRedirects
  73. Edit
  74. EditData
  75. EditDiagram
  76. EditPage
  77. EditSchema
  78. EditTags
  79. EditWatchlist
  80. Email
  81. EmailUser
  82. EnableFlow
  83. EnableStructuredDiscussions
  84. ExpandTemplates
  85. Export
  86. ExportRDF
  87. FacetedSearch
  88. FewestRevisions
  89. FileDuplicateSearch
  90. FileList
  91. FilePath
  92. FindComment
  93. Flow
  94. FormEdit
  95. Forms
  96. FormStart
  97. Gadgets
  98. GadgetUsage
  99. GeneratePages
  100. GetData
  101. GoToComment
  102. GoToInterwiki
  103. GraphSandbox
  104. History
  105. History
  106. ImageList
  107. Import
  108. ImportCSV
  109. ImportSpreadsheet
  110. ImportXML
  111. Info
  112. Interwiki
  113. InvalidateEmail
  114. Investigate
  115. InvestigateBlock
  116. IPBlockList
  117. LinkAccounts
  118. LinkSearch
  119. LintErrors
  120. ListAdmins
  121. ListAutoblocks
  122. ListBlocks
  123. ListBots
  124. ListDuplicatedFiles
  125. ListFileDuplicates
  126. ListFiles
  127. ListGrants
  128. ListGroupRights
  129. ListRedirects
  130. ListUsers
  131. LockDB
  132. Log
  133. ✅ Login
  134. Logout
  135. Logs
  136. LonelyPages
  137. LongPages
  138. MakeBot
  139. MakeSysop
  140. Manage_Two-factor_authentication
  141. MathShowImage
  142. MathStatus
  143. MediaStatistics
  144. MergeHistory
  145. MIMESearch
  146. MissingRedirectAnnotations
  147. MobileDiff
  148. MobileLanguages
  149. MobileOptions
  150. MostCategories
  151. MostFiles
  152. MostImages
  153. MostInterwikis
  154. MostLinked
  155. MostLinkedCategories
  156. MostLinkedFiles
  157. MostLinkedPages
  158. MostLinkedTemplates
  159. MostRevisions
  160. MostTranscludedPages
  161. MostUsedCategories
  162. MostUsedTemplates
  163. MovePage
  164. MultiPageEdit
  165. MyContribs
  166. MyContributions
  167. MyFiles
  168. MyLanguage
  169. MyLog
  170. MyPage
  171. MyTalk
  172. MyUploads
  173. NamespaceInfo
  174. NewFiles
  175. NewImages
  176. NewPages
  177. Newsection
  178. NewSection
  179. Notifications
  180. NotificationsMarkRead
  181. Nuke
  182. OATH
  183. OATHAuth
  184. OATH_Manage
  185. OATHManage
  186. OrphanedPages
  187. Packages
  188. PageData
  189. PageHistory
  190. PageInfo
  191. PageProperty
  192. Pagesbyprop
  193. PagesByProp
  194. PageStatistics
  195. Pageswithprop
  196. PagesWithProp
  197. PasswordPolicies
  198. ✅ PasswordReset
  199. PendingReviews
  200. PendingTaskList
  201. PermaLink
  202. PermanentLink
  203. ✅ PluggableAuthLogin
  204. PluggableAuthLogout
  205. Preferences
  206. ✅ PrefixIndex
  207. ProcessingErrorList
  208. Properties
  209. PropertyLabelSimilarity
  210. Protect
  211. ProtectedPages
  212. ProtectedTitles
  213. ProtectPage
  214. Purge
  215. ✅ Random
  216. RandomInCategory
  217. ✅ RandomPage
  218. RandomRedirect
  219. RandomRootpage
  220. RecentChanges
  221. RecentChangesLinked
  222. Redirect
  223. RelatedChanges
  224. RemoveCredentials
  225. RenameUser
  226. ReplaceText
  227. ✅ ResetPass
  228. ✅ ResetPassword
  229. ResetTokens
  230. RevisionDelete
  231. RunJobs
  232. RunQuery
  233. ✅ Search
  234. ✅ SearchByProperty
  235. SemanticMediaWiki
  236. ShortPages
  237. SMWAdmin
  238. ✅ SpecialPages
  239. ✅ Statistics
  240. ✅ Stats
  241. StructuredDiscussions
  242. Tags
  243. TalkPage
  244. Templates
  245. Thanks
  246. TopicSubscriptions
  247. TrackingCategories
  248. Two-factor_authentication
  249. Types
  250. Unblock
  251. UncategorizedCategories
  252. UncategorizedFiles
  253. UncategorizedImages
  254. UncategorizedPages
  255. UncategorizedTemplates
  256. Undelete
  257. UnlinkAccounts
  258. UnlockDB
  259. UnusedCategories
  260. UnusedFiles
  261. UnusedImages
  262. UnusedProperties
  263. UnusedTemplates
  264. UnwatchedPages
  265. Upload
  266. Uploads
  267. UploadStash
  268. UploadWindow
  269. URIResolver
  270. UserGroupRights
  271. UserList
  272. ✅ UserLogin
  273. UserLogout
  274. UserMerge
  275. UserRights
  276. Users
  277. VerifyOATHForUser
  278. ✅ Version
  279. ViewXML
  280. WantedCategories
  281. WantedFiles
  282. WantedPages
  283. WantedProperties
  284. WantedTemplates
  285. WatchAnalytics
  286. Watchlist
  287. WebChat
  288. WhatLinksHere
  289. WhoIsWatching
  290. WithoutInterwiki

Assuming CrawlerProtection is updated with my PR, you can just configure that extension with the list above.

You can achieve this restriction using the Lockdown extension with this configuration: 

$wgSpecialPageLockdown['AbuseFilter'] = ['user'];
$wgSpecialPageLockdown['AbuseLog'] = ['user'];
$wgSpecialPageLockdown['AccountSecurity'] = ['user'];
$wgSpecialPageLockdown['ActiveUsers'] = ['user'];
$wgSpecialPageLockdown['AddData'] = ['user'];
$wgSpecialPageLockdown['AddPage'] = ['user'];
$wgSpecialPageLockdown['AdminLinks'] = ['user'];
$wgSpecialPageLockdown['AllMessages'] = ['user'];
$wgSpecialPageLockdown['AllMyFiles'] = ['user'];
$wgSpecialPageLockdown['AllMyUploads'] = ['user'];
$wgSpecialPageLockdown['AllPages'] = ['user'];
$wgSpecialPageLockdown['AncientPages'] = ['user'];
$wgSpecialPageLockdown['ApiHelp'] = ['user'];
$wgSpecialPageLockdown['ApiSandbox'] = ['user'];
$wgSpecialPageLockdown['Ask'] = ['user'];
$wgSpecialPageLockdown['AuthenticationPopupSuccess'] = ['user'];
$wgSpecialPageLockdown['AutoblockList'] = ['user'];
$wgSpecialPageLockdown['BatchUpload'] = ['user'];
$wgSpecialPageLockdown['BetaFeatures'] = ['user'];
$wgSpecialPageLockdown['BlankPage'] = ['user'];
$wgSpecialPageLockdown['Block'] = ['user'];
$wgSpecialPageLockdown['BlockedExternalDomains'] = ['user'];
$wgSpecialPageLockdown['BlockIP'] = ['user'];
$wgSpecialPageLockdown['BlockList'] = ['user'];
$wgSpecialPageLockdown['BlockUser'] = ['user'];
$wgSpecialPageLockdown['Book'] = ['user'];
$wgSpecialPageLockdown['BookSources'] = ['user'];
$wgSpecialPageLockdown['BotPasswords'] = ['user'];
$wgSpecialPageLockdown['BrokenLinks'] = ['user'];
$wgSpecialPageLockdown['BrokenRedirects'] = ['user'];
$wgSpecialPageLockdown['BrowseData'] = ['user'];
$wgSpecialPageLockdown['Captcha'] = ['user'];
$wgSpecialPageLockdown['CategoryTree'] = ['user'];
$wgSpecialPageLockdown['ChangeContentModel'] = ['user'];
$wgSpecialPageLockdown['ChangeCredentials'] = ['user'];
$wgSpecialPageLockdown['ChangeEmail'] = ['user'];
$wgSpecialPageLockdown['ChangePassword'] = ['user'];
$wgSpecialPageLockdown['CheckUser'] = ['user'];
$wgSpecialPageLockdown['CheckUserLog'] = ['user'];
$wgSpecialPageLockdown['Cite'] = ['user'];
$wgSpecialPageLockdown['CiteThisPage'] = ['user'];
$wgSpecialPageLockdown['ClearPendingReviews'] = ['user'];
$wgSpecialPageLockdown['Collab_Pad'] = ['user'];
$wgSpecialPageLockdown['CollabPad'] = ['user'];
$wgSpecialPageLockdown['Collection'] = ['user'];
$wgSpecialPageLockdown['ComparePages'] = ['user'];
$wgSpecialPageLockdown['Concepts'] = ['user'];
$wgSpecialPageLockdown['ConstraintErrorList'] = ['user'];
$wgSpecialPageLockdown['Contribs'] = ['user'];
$wgSpecialPageLockdown['Contribute'] = ['user'];
$wgSpecialPageLockdown['Contributions'] = ['user'];
$wgSpecialPageLockdown['ContributionScores'] = ['user'];
$wgSpecialPageLockdown['CreateAccount'] = ['user'];
$wgSpecialPageLockdown['CreateCategory'] = ['user'];
$wgSpecialPageLockdown['CreateClass'] = ['user'];
$wgSpecialPageLockdown['CreateForm'] = ['user'];
$wgSpecialPageLockdown['CreateProperty'] = ['user'];
$wgSpecialPageLockdown['CreateRedirect'] = ['user'];
$wgSpecialPageLockdown['CreateTemplate'] = ['user'];
$wgSpecialPageLockdown['DeadendPages'] = ['user'];
$wgSpecialPageLockdown['Delete'] = ['user'];
$wgSpecialPageLockdown['DeleteBatch'] = ['user'];
$wgSpecialPageLockdown['DeletedContributions'] = ['user'];
$wgSpecialPageLockdown['DeletePage'] = ['user'];
$wgSpecialPageLockdown['Diff'] = ['user'];
$wgSpecialPageLockdown['DisableOATHForUser'] = ['user'];
$wgSpecialPageLockdown['DiscussionToolsDebug'] = ['user'];
$wgSpecialPageLockdown['DisplayNotificationsConfiguration'] = ['user'];
$wgSpecialPageLockdown['DoubleRedirects'] = ['user'];
$wgSpecialPageLockdown['Edit'] = ['user'];
$wgSpecialPageLockdown['EditData'] = ['user'];
$wgSpecialPageLockdown['EditDiagram'] = ['user'];
$wgSpecialPageLockdown['EditPage'] = ['user'];
$wgSpecialPageLockdown['EditSchema'] = ['user'];
$wgSpecialPageLockdown['EditTags'] = ['user'];
$wgSpecialPageLockdown['EditWatchlist'] = ['user'];
$wgSpecialPageLockdown['Email'] = ['user'];
$wgSpecialPageLockdown['EmailUser'] = ['user'];
$wgSpecialPageLockdown['EnableFlow'] = ['user'];
$wgSpecialPageLockdown['EnableStructuredDiscussions'] = ['user'];
$wgSpecialPageLockdown['ExpandTemplates'] = ['user'];
$wgSpecialPageLockdown['Export'] = ['user'];
$wgSpecialPageLockdown['ExportRDF'] = ['user'];
$wgSpecialPageLockdown['FacetedSearch'] = ['user'];
$wgSpecialPageLockdown['FewestRevisions'] = ['user'];
$wgSpecialPageLockdown['FileDuplicateSearch'] = ['user'];
$wgSpecialPageLockdown['FileList'] = ['user'];
$wgSpecialPageLockdown['FilePath'] = ['user'];
$wgSpecialPageLockdown['FindComment'] = ['user'];
$wgSpecialPageLockdown['Flow'] = ['user'];
$wgSpecialPageLockdown['FormEdit'] = ['user'];
$wgSpecialPageLockdown['Forms'] = ['user'];
$wgSpecialPageLockdown['FormStart'] = ['user'];
$wgSpecialPageLockdown['Gadgets'] = ['user'];
$wgSpecialPageLockdown['GadgetUsage'] = ['user'];
$wgSpecialPageLockdown['GeneratePages'] = ['user'];
$wgSpecialPageLockdown['GetData'] = ['user'];
$wgSpecialPageLockdown['GoToComment'] = ['user'];
$wgSpecialPageLockdown['GoToInterwiki'] = ['user'];
$wgSpecialPageLockdown['GraphSandbox'] = ['user'];
$wgSpecialPageLockdown['History'] = ['user'];
$wgSpecialPageLockdown['History'] = ['user'];
$wgSpecialPageLockdown['ImageList'] = ['user'];
$wgSpecialPageLockdown['Import'] = ['user'];
$wgSpecialPageLockdown['ImportCSV'] = ['user'];
$wgSpecialPageLockdown['ImportSpreadsheet'] = ['user'];
$wgSpecialPageLockdown['ImportXML'] = ['user'];
$wgSpecialPageLockdown['Info'] = ['user'];
$wgSpecialPageLockdown['Interwiki'] = ['user'];
$wgSpecialPageLockdown['InvalidateEmail'] = ['user'];
$wgSpecialPageLockdown['Investigate'] = ['user'];
$wgSpecialPageLockdown['InvestigateBlock'] = ['user'];
$wgSpecialPageLockdown['IPBlockList'] = ['user'];
$wgSpecialPageLockdown['LinkAccounts'] = ['user'];
$wgSpecialPageLockdown['LinkSearch'] = ['user'];
$wgSpecialPageLockdown['LintErrors'] = ['user'];
$wgSpecialPageLockdown['ListAdmins'] = ['user'];
$wgSpecialPageLockdown['ListAutoblocks'] = ['user'];
$wgSpecialPageLockdown['ListBlocks'] = ['user'];
$wgSpecialPageLockdown['ListBots'] = ['user'];
$wgSpecialPageLockdown['ListDuplicatedFiles'] = ['user'];
$wgSpecialPageLockdown['ListFileDuplicates'] = ['user'];
$wgSpecialPageLockdown['ListFiles'] = ['user'];
$wgSpecialPageLockdown['ListGrants'] = ['user'];
$wgSpecialPageLockdown['ListGroupRights'] = ['user'];
$wgSpecialPageLockdown['ListRedirects'] = ['user'];
$wgSpecialPageLockdown['ListUsers'] = ['user'];
$wgSpecialPageLockdown['LockDB'] = ['user'];
$wgSpecialPageLockdown['Log'] = ['user'];
$wgSpecialPageLockdown['Logout'] = ['user'];
$wgSpecialPageLockdown['Logs'] = ['user'];
$wgSpecialPageLockdown['LonelyPages'] = ['user'];
$wgSpecialPageLockdown['LongPages'] = ['user'];
$wgSpecialPageLockdown['MakeBot'] = ['user'];
$wgSpecialPageLockdown['MakeSysop'] = ['user'];
$wgSpecialPageLockdown['Manage_Two-factor_authentication'] = ['user'];
$wgSpecialPageLockdown['MathShowImage'] = ['user'];
$wgSpecialPageLockdown['MathStatus'] = ['user'];
$wgSpecialPageLockdown['MediaStatistics'] = ['user'];
$wgSpecialPageLockdown['MergeHistory'] = ['user'];
$wgSpecialPageLockdown['MIMESearch'] = ['user'];
$wgSpecialPageLockdown['MissingRedirectAnnotations'] = ['user'];
$wgSpecialPageLockdown['MobileDiff'] = ['user'];
$wgSpecialPageLockdown['MobileLanguages'] = ['user'];
$wgSpecialPageLockdown['MobileOptions'] = ['user'];
$wgSpecialPageLockdown['MostCategories'] = ['user'];
$wgSpecialPageLockdown['MostFiles'] = ['user'];
$wgSpecialPageLockdown['MostImages'] = ['user'];
$wgSpecialPageLockdown['MostInterwikis'] = ['user'];
$wgSpecialPageLockdown['MostLinked'] = ['user'];
$wgSpecialPageLockdown['MostLinkedCategories'] = ['user'];
$wgSpecialPageLockdown['MostLinkedFiles'] = ['user'];
$wgSpecialPageLockdown['MostLinkedPages'] = ['user'];
$wgSpecialPageLockdown['MostLinkedTemplates'] = ['user'];
$wgSpecialPageLockdown['MostRevisions'] = ['user'];
$wgSpecialPageLockdown['MostTranscludedPages'] = ['user'];
$wgSpecialPageLockdown['MostUsedCategories'] = ['user'];
$wgSpecialPageLockdown['MostUsedTemplates'] = ['user'];
$wgSpecialPageLockdown['MovePage'] = ['user'];
$wgSpecialPageLockdown['MultiPageEdit'] = ['user'];
$wgSpecialPageLockdown['MyContribs'] = ['user'];
$wgSpecialPageLockdown['MyContributions'] = ['user'];
$wgSpecialPageLockdown['MyFiles'] = ['user'];
$wgSpecialPageLockdown['MyLanguage'] = ['user'];
$wgSpecialPageLockdown['MyLog'] = ['user'];
$wgSpecialPageLockdown['MyPage'] = ['user'];
$wgSpecialPageLockdown['MyTalk'] = ['user'];
$wgSpecialPageLockdown['MyUploads'] = ['user'];
$wgSpecialPageLockdown['NamespaceInfo'] = ['user'];
$wgSpecialPageLockdown['NewFiles'] = ['user'];
$wgSpecialPageLockdown['NewImages'] = ['user'];
$wgSpecialPageLockdown['NewPages'] = ['user'];
$wgSpecialPageLockdown['Newsection'] = ['user'];
$wgSpecialPageLockdown['NewSection'] = ['user'];
$wgSpecialPageLockdown['Notifications'] = ['user'];
$wgSpecialPageLockdown['NotificationsMarkRead'] = ['user'];
$wgSpecialPageLockdown['Nuke'] = ['user'];
$wgSpecialPageLockdown['OATH'] = ['user'];
$wgSpecialPageLockdown['OATHAuth'] = ['user'];
$wgSpecialPageLockdown['OATH_Manage'] = ['user'];
$wgSpecialPageLockdown['OATHManage'] = ['user'];
$wgSpecialPageLockdown['OrphanedPages'] = ['user'];
$wgSpecialPageLockdown['Packages'] = ['user'];
$wgSpecialPageLockdown['PageData'] = ['user'];
$wgSpecialPageLockdown['PageHistory'] = ['user'];
$wgSpecialPageLockdown['PageInfo'] = ['user'];
$wgSpecialPageLockdown['PageProperty'] = ['user'];
$wgSpecialPageLockdown['Pagesbyprop'] = ['user'];
$wgSpecialPageLockdown['PagesByProp'] = ['user'];
$wgSpecialPageLockdown['PageStatistics'] = ['user'];
$wgSpecialPageLockdown['Pageswithprop'] = ['user'];
$wgSpecialPageLockdown['PagesWithProp'] = ['user'];
$wgSpecialPageLockdown['PasswordPolicies'] = ['user'];
$wgSpecialPageLockdown['PendingReviews'] = ['user'];
$wgSpecialPageLockdown['PendingTaskList'] = ['user'];
$wgSpecialPageLockdown['PermaLink'] = ['user'];
$wgSpecialPageLockdown['PermanentLink'] = ['user'];
$wgSpecialPageLockdown['PluggableAuthLogout'] = ['user'];
$wgSpecialPageLockdown['Preferences'] = ['user'];
$wgSpecialPageLockdown['ProcessingErrorList'] = ['user'];
$wgSpecialPageLockdown['Properties'] = ['user'];
$wgSpecialPageLockdown['PropertyLabelSimilarity'] = ['user'];
$wgSpecialPageLockdown['Protect'] = ['user'];
$wgSpecialPageLockdown['ProtectedPages'] = ['user'];
$wgSpecialPageLockdown['ProtectedTitles'] = ['user'];
$wgSpecialPageLockdown['ProtectPage'] = ['user'];
$wgSpecialPageLockdown['Purge'] = ['user'];
$wgSpecialPageLockdown['RandomInCategory'] = ['user'];
$wgSpecialPageLockdown['RandomRedirect'] = ['user'];
$wgSpecialPageLockdown['RandomRootpage'] = ['user'];
$wgSpecialPageLockdown['RecentChanges'] = ['user'];
$wgSpecialPageLockdown['RecentChangesLinked'] = ['user'];
$wgSpecialPageLockdown['Redirect'] = ['user'];
$wgSpecialPageLockdown['RelatedChanges'] = ['user'];
$wgSpecialPageLockdown['RemoveCredentials'] = ['user'];
$wgSpecialPageLockdown['RenameUser'] = ['user'];
$wgSpecialPageLockdown['ReplaceText'] = ['user'];
$wgSpecialPageLockdown['ResetTokens'] = ['user'];
$wgSpecialPageLockdown['RevisionDelete'] = ['user'];
$wgSpecialPageLockdown['RunJobs'] = ['user'];
$wgSpecialPageLockdown['RunQuery'] = ['user'];
$wgSpecialPageLockdown['SemanticMediaWiki'] = ['user'];
$wgSpecialPageLockdown['ShortPages'] = ['user'];
$wgSpecialPageLockdown['SMWAdmin'] = ['user'];
$wgSpecialPageLockdown['StructuredDiscussions'] = ['user'];
$wgSpecialPageLockdown['Tags'] = ['user'];
$wgSpecialPageLockdown['TalkPage'] = ['user'];
$wgSpecialPageLockdown['Templates'] = ['user'];
$wgSpecialPageLockdown['Thanks'] = ['user'];
$wgSpecialPageLockdown['TopicSubscriptions'] = ['user'];
$wgSpecialPageLockdown['TrackingCategories'] = ['user'];
$wgSpecialPageLockdown['Two-factor_authentication'] = ['user'];
$wgSpecialPageLockdown['Types'] = ['user'];
$wgSpecialPageLockdown['Unblock'] = ['user'];
$wgSpecialPageLockdown['UncategorizedCategories'] = ['user'];
$wgSpecialPageLockdown['UncategorizedFiles'] = ['user'];
$wgSpecialPageLockdown['UncategorizedImages'] = ['user'];
$wgSpecialPageLockdown['UncategorizedPages'] = ['user'];
$wgSpecialPageLockdown['UncategorizedTemplates'] = ['user'];
$wgSpecialPageLockdown['Undelete'] = ['user'];
$wgSpecialPageLockdown['UnlinkAccounts'] = ['user'];
$wgSpecialPageLockdown['UnlockDB'] = ['user'];
$wgSpecialPageLockdown['UnusedCategories'] = ['user'];
$wgSpecialPageLockdown['UnusedFiles'] = ['user'];
$wgSpecialPageLockdown['UnusedImages'] = ['user'];
$wgSpecialPageLockdown['UnusedProperties'] = ['user'];
$wgSpecialPageLockdown['UnusedTemplates'] = ['user'];
$wgSpecialPageLockdown['UnwatchedPages'] = ['user'];
$wgSpecialPageLockdown['Upload'] = ['user'];
$wgSpecialPageLockdown['Uploads'] = ['user'];
$wgSpecialPageLockdown['UploadStash'] = ['user'];
$wgSpecialPageLockdown['UploadWindow'] = ['user'];
$wgSpecialPageLockdown['URIResolver'] = ['user'];
$wgSpecialPageLockdown['UserGroupRights'] = ['user'];
$wgSpecialPageLockdown['UserList'] = ['user'];
$wgSpecialPageLockdown['UserLogout'] = ['user'];
$wgSpecialPageLockdown['UserMerge'] = ['user'];
$wgSpecialPageLockdown['UserRights'] = ['user'];
$wgSpecialPageLockdown['Users'] = ['user'];
$wgSpecialPageLockdown['VerifyOATHForUser'] = ['user'];
$wgSpecialPageLockdown['ViewXML'] = ['user'];
$wgSpecialPageLockdown['WantedCategories'] = ['user'];
$wgSpecialPageLockdown['WantedFiles'] = ['user'];
$wgSpecialPageLockdown['WantedPages'] = ['user'];
$wgSpecialPageLockdown['WantedProperties'] = ['user'];
$wgSpecialPageLockdown['WantedTemplates'] = ['user'];
$wgSpecialPageLockdown['WatchAnalytics'] = ['user'];
$wgSpecialPageLockdown['Watchlist'] = ['user'];
$wgSpecialPageLockdown['WebChat'] = ['user'];
$wgSpecialPageLockdown['WhatLinksHere'] = ['user'];
$wgSpecialPageLockdown['WhoIsWatching'] = ['user'];
$wgSpecialPageLockdown['WithoutInterwiki'] = ['user'];


References[edit]

  1. ↑ https://www.youtube.com/watch?v=VGS5l3YH2oY
  2. ↑ https://www.cloudflare.com/application-services/products/waf/ https://www.cloudflare.com/lp/waf-product-brief-xy/
  3. ↑ https://github.com/TecharoHQ/anubis/tree/main/data/crawlers

Uniform Resource Locator = an address of a web page or other component, including the protocol.

Semantic MediaWiki = See Metadata

Application Programming Interface

JavaScript Object Notation = A data representation format in JavaScript

Retrieved from "https://wiki.freephile.org/wiki/index.php?title=Stop_AI_crawlers&oldid=12576"
Powered by QualityBox
Powered by Semantic MediaWiki