Difference between revisions of "Cloning"
Jump to navigation
Jump to search
(more on cloning) |
(introduce an article for cloning DO droplet) |
||
| (7 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
| + | When first written, this article was about copying a physical hardware system. Nowadays, few systems are physical, and "cloud" is pervasive. Read about [[Cloning/Droplet]] for an updated version. | ||
| + | |||
Once you have [[backups]], you can and should test the ability to restore those backups to a new hardware environment. The restore procedure is going to depend on what you used to create your "image" in the first place. Independent of the notion of backups, a system administrator (or even an experienced user) will want to "clone" a computer system. That is make an identical copy or nearly identical copy of a computer on different hardware. The most widespread use-case for this is when a person gets a new computer and wants to migrate the old computer software and data to the new hardware. From the system administrator's perspective, the common use case is different: it's when you have to build out a cluster, provide redundancy, or otherwise scale a hardware/software platform. | Once you have [[backups]], you can and should test the ability to restore those backups to a new hardware environment. The restore procedure is going to depend on what you used to create your "image" in the first place. Independent of the notion of backups, a system administrator (or even an experienced user) will want to "clone" a computer system. That is make an identical copy or nearly identical copy of a computer on different hardware. The most widespread use-case for this is when a person gets a new computer and wants to migrate the old computer software and data to the new hardware. From the system administrator's perspective, the common use case is different: it's when you have to build out a cluster, provide redundancy, or otherwise scale a hardware/software platform. | ||
| − | Consider all the times that you could produce a system that you wish to replicate. For example, a standard "Developer's Workstation" or "Designer's Studio" or "Student Desktop" and many more technical varieties like "Web Front-end", "Mail Relay" etc. | + | Consider all the times that you could produce a system that you wish to replicate. For example, a standard "Developer's Workstation" or "Designer's Studio" or "Student Desktop" and many more technical varieties like "Web Front-end", "Mail Relay" etc. |
| − | == | + | While cloning is never a real-time operation, synchronization can be. For system architectures that require high availability, or scalability, you will need some form of [[synchronization]] for various components of the system (disk, databases, etc.) |
| + | |||
| + | ==dpkg --set-selections== | ||
One way to do this on Debian-based systems (including Ubuntu) is to use <code>dpkg</code> get/set-selections in combination with <code>apt-get</code> using it's dselect-upgrade option: | One way to do this on Debian-based systems (including Ubuntu) is to use <code>dpkg</code> get/set-selections in combination with <code>apt-get</code> using it's dselect-upgrade option: | ||
| Line 9: | Line 13: | ||
the following: | the following: | ||
| − | === on System A === | + | ===on System A=== |
<source lang="bash"> | <source lang="bash"> | ||
sudo dpkg --get-selections > package-list.txt | sudo dpkg --get-selections > package-list.txt | ||
| Line 15: | Line 19: | ||
copy package-list.txt from System A to System B | copy package-list.txt from System A to System B | ||
| − | === on System B === | + | ===on System B=== |
<source lang="bash"> | <source lang="bash"> | ||
sudo dpkg --set-selections < package-list.txt | sudo dpkg --set-selections < package-list.txt | ||
| Line 31: | Line 35: | ||
There are other tools to do the job too, and the tools you want to use for this depend on how automated you aim for this task to be, and how much you need/want to be able to customize each machine as it's being installed. | There are other tools to do the job too, and the tools you want to use for this depend on how automated you aim for this task to be, and how much you need/want to be able to customize each machine as it's being installed. | ||
| − | == KickStart == | + | ==KickStart== |
'''[http://fedoraproject.org/wiki/Anaconda/Kickstart Kickstart]''' is a Red Hat package that deploys Red Hat to multiple installation targets with minimal customisation. Aside from the wiki, it is also covered in [http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.3/html/Installation_Guide/index.html the Installation Guide for RedHat Enterprise Linux] | '''[http://fedoraproject.org/wiki/Anaconda/Kickstart Kickstart]''' is a Red Hat package that deploys Red Hat to multiple installation targets with minimal customisation. Aside from the wiki, it is also covered in [http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.3/html/Installation_Guide/index.html the Installation Guide for RedHat Enterprise Linux] | ||
| − | == System Imager == | + | ==System Imager== |
SystemImager is a third-party tool that does a better job. http://systemimager.org/ | SystemImager is a third-party tool that does a better job. http://systemimager.org/ | ||
| − | == Fully Automated Install | + | ==Fully Automated Install== |
'''[http://www.informatik.uni-koeln.de/fai/ fai]''' (fully automated install) is a Debian-based tool to do likewise. Like System Imager, it's suitable for building clusters. See the | '''[http://www.informatik.uni-koeln.de/fai/ fai]''' (fully automated install) is a Debian-based tool to do likewise. Like System Imager, it's suitable for building clusters. See the | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | == Replicator == | + | *[http://www.informatik.uni-koeln.de/fai/ project website] and the |
| + | *[http://faiwiki.informatik.uni-koeln.de/index.php/Main_Page wiki]. | ||
| + | *[http://www.slideshare.net/henningsprang/automated-installations-and-infrastructure-management-with-fai-presentation presentation] | ||
| + | *[[File:Fai_poster_a4.pdf|600px|thumb|right|Fully Automated Install (FAI) Poster]] | ||
| + | |||
| + | ==Replicator== | ||
'''Replicator''' seems like it could use some volunteers to expand. http://replicator.sourceforge.net/ It tries to do some customization for differences in hard disk sizes, video cards, etc. You may want to check it out. | '''Replicator''' seems like it could use some volunteers to expand. http://replicator.sourceforge.net/ It tries to do some customization for differences in hard disk sizes, video cards, etc. You may want to check it out. | ||
| − | == Partition Image == | + | ==Partition Image== |
| − | '''[http://www.partimage.org/ | + | '''[http://www.partimage.org/ Partition Image]''' is a semi-automated tool for replicating a Linux partition to multiple targets. It is particularly useful as a system cloning tool, and has many uses for data backup and archiving of whole disks or partitions. |
| + | |||
| + | ==Replicating a PHP installation== | ||
| + | With the PHP CLI installed, you can use <code>php -m</code> to get a list of the compiled-in modules for your PHP installation. This is the same as you would find by using the <code>phpinfo();</code> function in a script, except that you get exactly the info you need, in a list format that you can use. That list can show you what you might need to install on another host to ensure that each host is compatible for the software you're running. However, it won't show you the name of the actual packages that you need to install. So, if you want to know what PHP packages are installed, you can do the following | ||
| + | <source lang="bash"> | ||
| + | # On system A, get a list of the php packages that are installed | ||
| + | ## apt-based package manager: Debian, Ubuntu | ||
| + | dpkg --get-selections |grep -v deinstall | grep php5- | awk 'ORS=" " { print $1 }' | ||
| + | ## yum-based package manager: RedHat, CentOS, Fedora | ||
| + | |||
| + | yum list installed | grep php5 | awk 'ORS=" " { print $1 }' | ||
| + | # On system B, install those packages | ||
| + | sudo apt-get install [LIST FROM A] | ||
| + | </source> | ||
| + | If system A and system B use different package managers, then the best you can do is get the list of the modules from each system and compare them. In our <code>diff</code>, we'll use the system we're moving from on the left, because <code>diff</code> is quirky and will show lines that 'differ' with a pipe symbol; thus it's less error-prone to print the first column with awk. | ||
| + | <source lang="bash"> | ||
| + | A=webfaction.php.modules.txt | ||
| + | |||
| + | B=digital.ocean.php.modules.txt | ||
| + | diff --suppress-common-lines --side-by-side $A $B | ||
| + | |||
| + | </source> | ||
| + | It's not perfect, but now we have a good list of the modules to search and install. | ||
| + | <pre> | ||
| + | > | ||
| + | curl | ||
| + | > | ||
| + | gd | ||
| + | gmp | ||
| + | imap | ||
| + | intl | ||
| + | ionCube | ||
| + | ldap | ||
| + | mailparse | ||
| + | mcrypt | ||
| + | memcache | ||
| + | mysqlnd | ||
| + | > | ||
| + | pdo_pgsql | ||
| + | pdo_sqlite | ||
| + | pgsql | ||
| + | pspell | ||
| + | > | ||
| + | sqlite3 | ||
| + | tidy | ||
| + | > | ||
| + | > | ||
| + | > | ||
| + | xmlrpc | ||
| + | xsl | ||
| + | the | ||
| + | > | ||
| + | |||
| + | </pre> | ||
| + | |||
| + | |||
| + | See what's available on our new host. (repositories are different) | ||
| + | <source lang="bash"> | ||
| + | apt-cache search php5- | ||
| + | </source> | ||
| + | |||
| + | <pre> | ||
| + | php5-cgi - server-side, HTML-embedded scripting language (CGI binary) | ||
| + | php5-cli - command-line interpreter for the php5 scripting language | ||
| + | php5-common - Common files for packages built from the php5 source | ||
| + | php5-curl - CURL module for php5 | ||
| + | php5-dbg - Debug symbols for PHP5 | ||
| + | php5-dev - Files for PHP5 module development | ||
| + | php5-gd - GD module for php5 | ||
| + | php5-gmp - GMP module for php5 | ||
| + | php5-json - JSON module for php5 | ||
| + | php5-ldap - LDAP module for php5 | ||
| + | php5-mysql - MySQL module for php5 | ||
| + | php5-odbc - ODBC module for php5 | ||
| + | php5-pgsql - PostgreSQL module for php5 | ||
| + | php5-pspell - pspell module for php5 | ||
| + | php5-readline - Readline module for php5 | ||
| + | php5-recode - recode module for php5 | ||
| + | php5-snmp - SNMP module for php5 | ||
| + | php5-sqlite - SQLite module for php5 | ||
| + | php5-tidy - tidy module for php5 | ||
| + | php5-xmlrpc - XML-RPC module for php5 | ||
| + | php5-xsl - XSL module for php5 | ||
| + | libphp5-embed - HTML-embedded scripting language (Embedded SAPI library) | ||
| + | php5-adodb - Extension optimising the ADOdb database abstraction library | ||
| + | php5-apcu - APC User Cache for PHP 5 | ||
| + | php5-enchant - Enchant module for php5 | ||
| + | php5-exactimage - fast image manipulation library (PHP bindings) | ||
| + | php5-fpm - server-side, HTML-embedded scripting language (FPM-CGI binary) | ||
| + | php5-gdcm - Grassroots DICOM PHP5 bindings | ||
| + | php5-gearman - PHP wrapper to libgearman | ||
| + | php5-geoip - GeoIP module for php5 | ||
| + | php5-gnupg - wrapper around the gpgme library | ||
| + | php5-imagick - ImageMagick module for php5 | ||
| + | php5-imap - IMAP module for php5 | ||
| + | php5-interbase - interbase/firebird module for php5 | ||
| + | php5-intl - internationalisation module for php5 | ||
| + | php5-lasso - Library for Liberty Alliance and SAML protocols - PHP 5 bindings | ||
| + | php5-librdf - PHP5 language bindings for the Redland RDF library | ||
| + | php5-mapscript - php5-cgi module for MapServer | ||
| + | php5-mcrypt - MCrypt module for php5 | ||
| + | php5-memcache - memcache extension module for PHP5 | ||
| + | php5-memcached - memcached extension module for PHP5, uses libmemcached | ||
| + | php5-midgard2 - Midgard2 Content Repository - PHP5 language bindings and module | ||
| + | php5-ming - Ming module for php5 | ||
| + | php5-mongo - MongoDB database driver | ||
| + | php5-msgpack - PHP extension for interfacing with MessagePack | ||
| + | php5-mysqlnd - MySQL module for php5 (Native Driver) | ||
| + | php5-mysqlnd-ms - MySQL replication and load balancing module for PHP | ||
| + | php5-oauth - OAuth 1.0 consumer and provider extension | ||
| + | php5-pinba - Pinba module for PHP 5 | ||
| + | php5-ps - ps module for PHP 5 | ||
| + | php5-radius - PECL radius module for PHP 5 | ||
| + | php5-redis - PHP extension for interfacing with Redis | ||
| + | php5-remctl - PECL module for Kerberos-authenticated command execution | ||
| + | php5-rrd - PHP bindings to rrd tool system | ||
| + | php5-sasl - Cyrus SASL Extension | ||
| + | php5-stomp - Streaming Text Oriented Messaging Protocol (STOMP) client module for PHP 5 | ||
| + | php5-svn - PHP Bindings for the Subversion Revision control system | ||
| + | php5-sybase - Sybase / MS SQL Server module for php5 | ||
| + | php5-tokyo-tyrant - PHP interface to Tokyo Cabinet's network interface, Tokyo Tyrant | ||
| + | php5-vtkgdcm - Grassroots DICOM VTK PHP bindings | ||
| + | php5-xcache - Fast, stable PHP opcode cacher | ||
| + | php5-xdebug - Xdebug Module for PHP 5 | ||
| + | php5-xhprof - Hierarchical Profiler for PHP5 | ||
| + | </pre> | ||
| + | |||
| + | And thus, our install | ||
| + | <source lang="bash"> | ||
| + | sudo apt-get install php5-curl php5-gd php5-gmp php5-imap php5-intl php5-ldap php5-mcrypt php5-memcache php5-mysqlnd php5-pgsql php5-pspell php5-sqlite php5-tidy php5-xmlrpc php5-xsl | ||
| + | </source> | ||
[[Category:Howto]] | [[Category:Howto]] | ||
Latest revision as of 12:19, 7 June 2023
When first written, this article was about copying a physical hardware system. Nowadays, few systems are physical, and "cloud" is pervasive. Read about Cloning/Droplet for an updated version.
Once you have backups, you can and should test the ability to restore those backups to a new hardware environment. The restore procedure is going to depend on what you used to create your "image" in the first place. Independent of the notion of backups, a system administrator (or even an experienced user) will want to "clone" a computer system. That is make an identical copy or nearly identical copy of a computer on different hardware. The most widespread use-case for this is when a person gets a new computer and wants to migrate the old computer software and data to the new hardware. From the system administrator's perspective, the common use case is different: it's when you have to build out a cluster, provide redundancy, or otherwise scale a hardware/software platform.
Consider all the times that you could produce a system that you wish to replicate. For example, a standard "Developer's Workstation" or "Designer's Studio" or "Student Desktop" and many more technical varieties like "Web Front-end", "Mail Relay" etc.
While cloning is never a real-time operation, synchronization can be. For system architectures that require high availability, or scalability, you will need some form of synchronization for various components of the system (disk, databases, etc.)
Contents
dpkg --set-selections[edit | edit source]
One way to do this on Debian-based systems (including Ubuntu) is to use dpkg get/set-selections in combination with apt-get using it's dselect-upgrade option:
To clone the package installation state of System A to System B do the following:
on System A[edit | edit source]
sudo dpkg --get-selections > package-list.txt
copy package-list.txt from System A to System B
on System B[edit | edit source]
sudo dpkg --set-selections < package-list.txt
sudo apt-get -u dselect-upgrade
You might get 'database' errors, in which case you should install dselect and update that as well (all as root, or with sudo):
apt-get update
apt-get install dselect
dselect update
apt-get -u dselect-upgrade
There are other tools to do the job too, and the tools you want to use for this depend on how automated you aim for this task to be, and how much you need/want to be able to customize each machine as it's being installed.
KickStart[edit | edit source]
Kickstart is a Red Hat package that deploys Red Hat to multiple installation targets with minimal customisation. Aside from the wiki, it is also covered in the Installation Guide for RedHat Enterprise Linux
System Imager[edit | edit source]
SystemImager is a third-party tool that does a better job. http://systemimager.org/
Fully Automated Install[edit | edit source]
fai (fully automated install) is a Debian-based tool to do likewise. Like System Imager, it's suitable for building clusters. See the
- project website and the
- wiki.
- presentation
Replicator[edit | edit source]
Replicator seems like it could use some volunteers to expand. http://replicator.sourceforge.net/ It tries to do some customization for differences in hard disk sizes, video cards, etc. You may want to check it out.
Partition Image[edit | edit source]
Partition Image is a semi-automated tool for replicating a Linux partition to multiple targets. It is particularly useful as a system cloning tool, and has many uses for data backup and archiving of whole disks or partitions.
Replicating a PHP installation[edit | edit source]
With the PHP CLI installed, you can use php -m to get a list of the compiled-in modules for your PHP installation. This is the same as you would find by using the phpinfo(); function in a script, except that you get exactly the info you need, in a list format that you can use. That list can show you what you might need to install on another host to ensure that each host is compatible for the software you're running. However, it won't show you the name of the actual packages that you need to install. So, if you want to know what PHP packages are installed, you can do the following
# On system A, get a list of the php packages that are installed
## apt-based package manager: Debian, Ubuntu
dpkg --get-selections |grep -v deinstall | grep php5- | awk 'ORS=" " { print $1 }'
## yum-based package manager: RedHat, CentOS, Fedora
yum list installed | grep php5 | awk 'ORS=" " { print $1 }'
# On system B, install those packages
sudo apt-get install [LIST FROM A]
If system A and system B use different package managers, then the best you can do is get the list of the modules from each system and compare them. In our diff, we'll use the system we're moving from on the left, because diff is quirky and will show lines that 'differ' with a pipe symbol; thus it's less error-prone to print the first column with awk.
A=webfaction.php.modules.txt
B=digital.ocean.php.modules.txt
diff --suppress-common-lines --side-by-side $A $B
It's not perfect, but now we have a good list of the modules to search and install.
> curl > gd gmp imap intl ionCube ldap mailparse mcrypt memcache mysqlnd > pdo_pgsql pdo_sqlite pgsql pspell > sqlite3 tidy > > > xmlrpc xsl the >
See what's available on our new host. (repositories are different)
apt-cache search php5-
php5-cgi - server-side, HTML-embedded scripting language (CGI binary) php5-cli - command-line interpreter for the php5 scripting language php5-common - Common files for packages built from the php5 source php5-curl - CURL module for php5 php5-dbg - Debug symbols for PHP5 php5-dev - Files for PHP5 module development php5-gd - GD module for php5 php5-gmp - GMP module for php5 php5-json - JSON module for php5 php5-ldap - LDAP module for php5 php5-mysql - MySQL module for php5 php5-odbc - ODBC module for php5 php5-pgsql - PostgreSQL module for php5 php5-pspell - pspell module for php5 php5-readline - Readline module for php5 php5-recode - recode module for php5 php5-snmp - SNMP module for php5 php5-sqlite - SQLite module for php5 php5-tidy - tidy module for php5 php5-xmlrpc - XML-RPC module for php5 php5-xsl - XSL module for php5 libphp5-embed - HTML-embedded scripting language (Embedded SAPI library) php5-adodb - Extension optimising the ADOdb database abstraction library php5-apcu - APC User Cache for PHP 5 php5-enchant - Enchant module for php5 php5-exactimage - fast image manipulation library (PHP bindings) php5-fpm - server-side, HTML-embedded scripting language (FPM-CGI binary) php5-gdcm - Grassroots DICOM PHP5 bindings php5-gearman - PHP wrapper to libgearman php5-geoip - GeoIP module for php5 php5-gnupg - wrapper around the gpgme library php5-imagick - ImageMagick module for php5 php5-imap - IMAP module for php5 php5-interbase - interbase/firebird module for php5 php5-intl - internationalisation module for php5 php5-lasso - Library for Liberty Alliance and SAML protocols - PHP 5 bindings php5-librdf - PHP5 language bindings for the Redland RDF library php5-mapscript - php5-cgi module for MapServer php5-mcrypt - MCrypt module for php5 php5-memcache - memcache extension module for PHP5 php5-memcached - memcached extension module for PHP5, uses libmemcached php5-midgard2 - Midgard2 Content Repository - PHP5 language bindings and module php5-ming - Ming module for php5 php5-mongo - MongoDB database driver php5-msgpack - PHP extension for interfacing with MessagePack php5-mysqlnd - MySQL module for php5 (Native Driver) php5-mysqlnd-ms - MySQL replication and load balancing module for PHP php5-oauth - OAuth 1.0 consumer and provider extension php5-pinba - Pinba module for PHP 5 php5-ps - ps module for PHP 5 php5-radius - PECL radius module for PHP 5 php5-redis - PHP extension for interfacing with Redis php5-remctl - PECL module for Kerberos-authenticated command execution php5-rrd - PHP bindings to rrd tool system php5-sasl - Cyrus SASL Extension php5-stomp - Streaming Text Oriented Messaging Protocol (STOMP) client module for PHP 5 php5-svn - PHP Bindings for the Subversion Revision control system php5-sybase - Sybase / MS SQL Server module for php5 php5-tokyo-tyrant - PHP interface to Tokyo Cabinet's network interface, Tokyo Tyrant php5-vtkgdcm - Grassroots DICOM VTK PHP bindings php5-xcache - Fast, stable PHP opcode cacher php5-xdebug - Xdebug Module for PHP 5 php5-xhprof - Hierarchical Profiler for PHP5
And thus, our install
sudo apt-get install php5-curl php5-gd php5-gmp php5-imap php5-intl php5-ldap php5-mcrypt php5-memcache php5-mysqlnd php5-pgsql php5-pspell php5-sqlite php5-tidy php5-xmlrpc php5-xsl
