Difference between revisions of "Automated deployment of MediaWiki"
(link to mwstake.org) |
(→MediaWiki with Kubernetes: add a lot of relevant info) |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
As a contributor to [https://github.com/enterprisemediawiki/meza Meza], and developer of [[QualityBox]], we've been closely following the efforts in the [[MediaWiki]] community around easier/automated deployments of MediaWiki. Actually, it's a goal of the [https://mwstake.org MWStakeholders Group]. And, it's a crucial aspect of the whole [https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q3 2017 Wikimedia Technology Department Goals] since at least 2018. Here is a list of some of the efforts and projects within and outside the WMF for automating deployments of MediaWiki: | As a contributor to [https://github.com/enterprisemediawiki/meza Meza], and developer of [[QualityBox]], we've been closely following the efforts in the [[MediaWiki]] community around easier/automated deployments of MediaWiki. Actually, it's a goal of the [https://mwstake.org MWStakeholders Group]. And, it's a crucial aspect of the whole [https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q3 2017 Wikimedia Technology Department Goals] since at least 2018. Here is a list of some of the efforts and projects within and outside the WMF for automating deployments of MediaWiki: | ||
| − | * Our own efforts: [[MediaWiki on Kubernetes]] | + | *Our own efforts: [[MediaWiki on Kubernetes]] |
| − | == WMF == | + | ==WMF== |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | === | + | === Dev Images === |
| − | + | WMF uses "Dev Images" in Release Engineering for their CI/CD pipelines https://gerrit.wikimedia.org/r/q/project:releng/dev-images+ There used to be a Gerrit repo for "dev-images", but it has now moved to GitLab https://gitlab.wikimedia.org/repos/releng/dev-images The repo is about Docker images for local MediaWiki development. Can be used with Docker Compose and Kubernetes. Published to the [https://dockerregistry.toolforge.org/ WikiMedia Docker Registry], e.g. https://dockerregistry.toolforge.org/wikimedia-buster/tags/ | |
| − | |||
| − | |||
| − | + | There was a 2019 issue / pull request to create a MySQL/MariaDB image instead of Bitnami image(s) for MySQL/MariaDB https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/38/diffs?commit_id=5a437dce4cd1921ce023b223541a962f1a520340 | |
| − | |||
| − | === | + | ===For Distributors=== |
| − | |||
| − | |||
| + | *[https://phabricator.wikimedia.org/T182020 Per RFC], distributors can override DefaultSettings.php with <code>includes/PlatformSettings.php</code> | ||
| + | *OPEN task: [https://phabricator.wikimedia.org/T173955 Allow DefaultSettings.php be overridden by environment variables] - Important for Kubernetes | ||
| − | == Configuration Management and Deployment at WMF == | + | ===MediaWiki with Kubernetes=== |
| + | |||
| + | *https://gerrit.wikimedia.org/r/plugins/gitiles/releng/local-charts/ Gerrit repo for using [[Minikube]] and '''Helm''' charts to define MediaWiki local development environments. '''This was just an experiment?''' | ||
| + | |||
| + | See https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes in Wikitech | ||
| + | |||
| + | The production code/repo for WMF's MediaWiki on Kubernetes is in the '''operations''' [https://gerrit.wikimedia.org/g/operations/deployment-charts Deployment Charts] repo. Mirror on GitHub https://github.com/wikimedia/operations-deployment-charts | ||
| + | |||
| + | One relevant project is the '''[https://phabricator.wikimedia.org/project/profile/2290/ Prod Kubernetes]''' Component and '''[https://phabricator.wikimedia.org/project/view/2290/ Prod Kubernetes]''' workboard in Phabricator. These are organized under the [https://phabricator.wikimedia.org/tag/sre/ SRE] team. [[wp:Wikimedia Site Reliability Engineering|Wikimedia Site Reliability Engineering]] | ||
| + | |||
| + | ===MediaWiki on Docker=== | ||
| + | [https://github.com/docker-library/docs Docker Library Docs] specifies how Docker official images are made (ie. ''all'' the official images on Docker Hub). There is specific information for MediaWiki at https://github.com/docker-library/docs/tree/master/mediawiki These images are what users get when using MediaWiki Docker (see section below for further details). | ||
| + | |||
| + | Compared to a "developer" process, you can see what is going into WMF production images at the [https://github.com/wikimedia/operations-docker-images-production-images Wikimedia base docker images for production] repository by [https://github.com/lavagetto lavagetto] (Giuseppe Lavagetto) | ||
| + | |||
| + | ==== MediaWiki Docker ==== | ||
| + | [https://www.mediawiki.org/wiki/MediaWiki-Docker MediaWiki-Docker] is distributed with MediaWiki core. Just follow the directions in [https://gerrit.wikimedia.org/g/mediawiki/core/+/HEAD/DEVELOPERS.md DEVELOPERS.md] to create your Docker-based development environment. The images, and more information can be found on the [https://hub.docker.com/_/mediawiki/ MediaWiki] page in DockerHub. The actual upstream repository (where images are built) comes from a WMF repo mirror on GitHub [https://github.com/wikimedia/mediawiki-docker Wikimedia Git repo of the Docker official image for MediaWiki] <ref>The repo was originally started as (forked from) Ben Hutchins' repo - which is now deprecated https://github.com/benhutchins/docker-mediawiki</ref> Note that there are at least 3 variants you can choose from: Apache, FPM and FPM-Alpine. | ||
| + | |||
| + | There is a compose file https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/HEAD/docker-compose.yml which can be overridden to specify variants, "recipes" and options. While the Docker Hub image just derives from PHP, the Compose file specifically derives from the '''Debian''' '''Linux PHP''' image<ref>e.g. buster-php81-fpm:1.0.1-s2 which derives from [https://hub.docker.com/layers/library/php/8.1.1-fpm-buster/images/sha256-e2af36f826fec07166dd390193fb0cd5957e361cdd9e1eb6e72de0100591441a Debian:10-slim]</ref>. | ||
| + | |||
| + | ==== MediaWiki Docker Dev ==== | ||
| + | [https://github.com/addshore/mediawiki-docker-dev MediaWiki-Docker-Dev] (Addshore's GitHub repo) for using Docker and MediaWiki (Developers) See also: [https://github.com/addwiki/mediawiki-api MediaWiki API library] It's a library for making it easier to work with the MediaWiki API. Good to know/use/contribute. One member of a group of libraries to interact with WikiData, etc. ([https://addwiki.readthedocs.io/en/latest/ docs]) Addwiki repos are by Addshore, Jeroen DeDauw and Sam Wilson. | ||
| + | |||
| + | ===Node Services=== | ||
| + | |||
| + | *[https://github.com/wikimedia/mediawiki-node-services MediaWiki Node Services] Run several MediaWiki nodejs services (RESTBase, Parsoid) in a single docker container. Useful for small / low-memory installs. By WMF's [https://github.com/d00rman d00rman] (Marko Obrovac - Croatia) and [https://github.com/gwicke gwicke] (Gabriel Wicke - Menlo Park) | ||
| + | |||
| + | ===Lex Sulzer=== | ||
| + | |||
| + | *[https://smw-cindykate.com/main/Component_947846 Set up BlueSpice MediaWiki Enterprise Distribution] by [https://github.com/lexsulzer lexsulzer] (Lex Sulzer at Dataspects - Zurich, Switzerland) | ||
| + | *[https://github.com/dataspects Dataspects]' [https://smw-cindykate.com/main/Component0272684347 Easy System Cloning] | ||
| + | |||
| + | |||
| + | ==Configuration Management and Deployment at WMF== | ||
It should be noted that the WMF uses Puppet for configuration management (See the [https://phabricator.wikimedia.org/diffusion/OMWC/ operations/mediawiki-config repo]), and <strike>[https://wikitech.wikimedia.org/wiki/Scap3 Scap3] (which is used in [https://wikitech.wikimedia.org/wiki/SWAT_deploys SWAT])</strike> (obsolete) for [https://wikitech.wikimedia.org/wiki/Deployments deployments]. There is the [https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS Cloud VPS] which is powered by [[wp:OpenStack]] for projects related to the [https://meta.wikimedia.org/wiki/Wikimedia_movement Wikimedia movement]. The environment includes access to a variety of [https://wikitech.wikimedia.org/wiki/Portal:Data_Services data services]. Cloud VPS is meant to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the [https://wikitech.wikimedia.org/wiki/Portal:Toolforge Toolforge] environment. Virtual machines running on Cloud VPS are subject to the [https://wikitech.wikimedia.org/wiki/Portal:Wikimedia_VPS/instance_lifecycle instance lifecycle]. | It should be noted that the WMF uses Puppet for configuration management (See the [https://phabricator.wikimedia.org/diffusion/OMWC/ operations/mediawiki-config repo]), and <strike>[https://wikitech.wikimedia.org/wiki/Scap3 Scap3] (which is used in [https://wikitech.wikimedia.org/wiki/SWAT_deploys SWAT])</strike> (obsolete) for [https://wikitech.wikimedia.org/wiki/Deployments deployments]. There is the [https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS Cloud VPS] which is powered by [[wp:OpenStack]] for projects related to the [https://meta.wikimedia.org/wiki/Wikimedia_movement Wikimedia movement]. The environment includes access to a variety of [https://wikitech.wikimedia.org/wiki/Portal:Data_Services data services]. Cloud VPS is meant to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the [https://wikitech.wikimedia.org/wiki/Portal:Toolforge Toolforge] environment. Virtual machines running on Cloud VPS are subject to the [https://wikitech.wikimedia.org/wiki/Portal:Wikimedia_VPS/instance_lifecycle instance lifecycle]. | ||
| − | === Puppet === | + | ===Puppet=== |
| − | |||
| − | |||
| − | |||
| + | *https://wikitech.wikimedia.org/wiki/Puppet | ||
| + | *https://wikitech.wikimedia.org/wiki/Puppet_coding | ||
| − | == Config Settings == | + | ==Config Settings== |
With any automation tooling and deployment, you will inevitably need to refer to the long list of [https://www.mediawiki.org/wiki/Manual:Configuration_settings MediaWiki Configuration settings]. | With any automation tooling and deployment, you will inevitably need to refer to the long list of [https://www.mediawiki.org/wiki/Manual:Configuration_settings MediaWiki Configuration settings]. | ||
Latest revision as of 13:02, 13 May 2024
As a contributor to Meza, and developer of QualityBox, we've been closely following the efforts in the MediaWiki community around easier/automated deployments of MediaWiki. Actually, it's a goal of the MWStakeholders Group. And, it's a crucial aspect of the whole 2017 Wikimedia Technology Department Goals since at least 2018. Here is a list of some of the efforts and projects within and outside the WMF for automating deployments of MediaWiki:
- Our own efforts: MediaWiki on Kubernetes
Contents
WMF[edit | edit source]
Dev Images[edit | edit source]
WMF uses "Dev Images" in Release Engineering for their CI/CD pipelines https://gerrit.wikimedia.org/r/q/project:releng/dev-images+ There used to be a Gerrit repo for "dev-images", but it has now moved to GitLab https://gitlab.wikimedia.org/repos/releng/dev-images The repo is about Docker images for local MediaWiki development. Can be used with Docker Compose and Kubernetes. Published to the WikiMedia Docker Registry, e.g. https://dockerregistry.toolforge.org/wikimedia-buster/tags/
There was a 2019 issue / pull request to create a MySQL/MariaDB image instead of Bitnami image(s) for MySQL/MariaDB https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/38/diffs?commit_id=5a437dce4cd1921ce023b223541a962f1a520340
For Distributors[edit | edit source]
- Per RFC, distributors can override DefaultSettings.php with
includes/PlatformSettings.php - OPEN task: Allow DefaultSettings.php be overridden by environment variables - Important for Kubernetes
MediaWiki with Kubernetes[edit | edit source]
- https://gerrit.wikimedia.org/r/plugins/gitiles/releng/local-charts/ Gerrit repo for using Minikube and Helm charts to define MediaWiki local development environments. This was just an experiment?
See https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes in Wikitech
The production code/repo for WMF's MediaWiki on Kubernetes is in the operations Deployment Charts repo. Mirror on GitHub https://github.com/wikimedia/operations-deployment-charts
One relevant project is the Prod Kubernetes Component and Prod Kubernetes workboard in Phabricator. These are organized under the SRE team. Wikimedia Site Reliability Engineering
MediaWiki on Docker[edit | edit source]
Docker Library Docs specifies how Docker official images are made (ie. all the official images on Docker Hub). There is specific information for MediaWiki at https://github.com/docker-library/docs/tree/master/mediawiki These images are what users get when using MediaWiki Docker (see section below for further details).
Compared to a "developer" process, you can see what is going into WMF production images at the Wikimedia base docker images for production repository by lavagetto (Giuseppe Lavagetto)
MediaWiki Docker[edit | edit source]
MediaWiki-Docker is distributed with MediaWiki core. Just follow the directions in DEVELOPERS.md to create your Docker-based development environment. The images, and more information can be found on the MediaWiki page in DockerHub. The actual upstream repository (where images are built) comes from a WMF repo mirror on GitHub Wikimedia Git repo of the Docker official image for MediaWiki [1] Note that there are at least 3 variants you can choose from: Apache, FPM and FPM-Alpine.
There is a compose file https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/HEAD/docker-compose.yml which can be overridden to specify variants, "recipes" and options. While the Docker Hub image just derives from PHP, the Compose file specifically derives from the Debian Linux PHP image[2].
MediaWiki Docker Dev[edit | edit source]
MediaWiki-Docker-Dev (Addshore's GitHub repo) for using Docker and MediaWiki (Developers) See also: MediaWiki API library It's a library for making it easier to work with the MediaWiki API. Good to know/use/contribute. One member of a group of libraries to interact with WikiData, etc. (docs) Addwiki repos are by Addshore, Jeroen DeDauw and Sam Wilson.
Node Services[edit | edit source]
- MediaWiki Node Services Run several MediaWiki nodejs services (RESTBase, Parsoid) in a single docker container. Useful for small / low-memory installs. By WMF's d00rman (Marko Obrovac - Croatia) and gwicke (Gabriel Wicke - Menlo Park)
Lex Sulzer[edit | edit source]
- Set up BlueSpice MediaWiki Enterprise Distribution by lexsulzer (Lex Sulzer at Dataspects - Zurich, Switzerland)
- Dataspects' Easy System Cloning
Configuration Management and Deployment at WMF[edit | edit source]
It should be noted that the WMF uses Puppet for configuration management (See the operations/mediawiki-config repo), and Scap3 (which is used in SWAT) (obsolete) for deployments. There is the Cloud VPS which is powered by wp:OpenStack for projects related to the Wikimedia movement. The environment includes access to a variety of data services. Cloud VPS is meant to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the Toolforge environment. Virtual machines running on Cloud VPS are subject to the instance lifecycle.
Puppet[edit | edit source]
Config Settings[edit | edit source]
With any automation tooling and deployment, you will inevitably need to refer to the long list of MediaWiki Configuration settings.
- ↑ The repo was originally started as (forked from) Ben Hutchins' repo - which is now deprecated https://github.com/benhutchins/docker-mediawiki
- ↑ e.g. buster-php81-fpm:1.0.1-s2 which derives from Debian:10-slim
