Difference between revisions of "Cloning"

From Freephile Wiki
Jump to navigation Jump to search
(introduce an article for cloning DO droplet)
(New page: Once you have backups, you can and should test the ability to restore those backups to a new hardware environment. The restore procedure is going to depend on what you used to create ...)
(8 intermediate revisions by one other user not shown)
Line 1: Line 1:
When first written, this article was about copying a physical hardware system. Nowadays, few systems are physical, and "cloud" is pervasive.  Read about [[Cloning/Droplet]] for an updated version.
 
 
 
Once you have [[backups]], you can and should test the ability to restore those backups to a new hardware environment.  The restore procedure is going to depend on what you used to create your "image" in the first place.  Independent of the notion of backups, a system administrator (or even an experienced user) will want to "clone" a computer system.  That is make an identical copy or nearly identical copy of a computer on different hardware.  The most widespread use-case for this is when a person gets a new computer and wants to migrate the old computer software and data to the new hardware.  From the system administrator's perspective, the common use case is different: it's when you have to build out a cluster, provide redundancy, or otherwise scale a hardware/software platform.
 
Once you have [[backups]], you can and should test the ability to restore those backups to a new hardware environment.  The restore procedure is going to depend on what you used to create your "image" in the first place.  Independent of the notion of backups, a system administrator (or even an experienced user) will want to "clone" a computer system.  That is make an identical copy or nearly identical copy of a computer on different hardware.  The most widespread use-case for this is when a person gets a new computer and wants to migrate the old computer software and data to the new hardware.  From the system administrator's perspective, the common use case is different: it's when you have to build out a cluster, provide redundancy, or otherwise scale a hardware/software platform.
  
Consider all the times that you could produce a system that you wish to replicate.  For example, a standard "Developer's Workstation" or "Designer's Studio" or "Student Desktop" and many more technical varieties like "Web Front-end", "Mail Relay" etc.
+
Consider all the times that you could produce a system that you wish to replicate.  For example, a standard "Developer's Workstation" or "Designer's Studio" or "Student Desktop" and many more technical varieties like "Web Front-end", "Mail Relay" etc.
  
While cloning is never a real-time operation, synchronization can be. For system architectures that require high availability, or scalability, you will need some form of [[synchronization]] for various components of the system (disk, databases, etc.)
+
== Apt-get --set-selections ==
 
 
==dpkg --set-selections==
 
 
One way to do this on Debian-based systems (including Ubuntu) is to use <code>dpkg</code> get/set-selections in combination with <code>apt-get</code> using it's dselect-upgrade option:
 
One way to do this on Debian-based systems (including Ubuntu) is to use <code>dpkg</code> get/set-selections in combination with <code>apt-get</code> using it's dselect-upgrade option:
  
Line 13: Line 9:
 
the following:
 
the following:
  
===on System A===
+
=== on System A ===
 
<source lang="bash">
 
<source lang="bash">
 
sudo dpkg --get-selections > package-list.txt
 
sudo dpkg --get-selections > package-list.txt
Line 19: Line 15:
 
copy package-list.txt from System A to System B
 
copy package-list.txt from System A to System B
  
===on System B===
+
=== on System B ===
 
<source lang="bash">
 
<source lang="bash">
 
sudo dpkg --set-selections < package-list.txt
 
sudo dpkg --set-selections < package-list.txt
Line 25: Line 21:
 
</source>
 
</source>
  
You might get 'database' errors, in which case you should install <code>dselect</code> and update that as well (all as root, or with sudo):
 
<source lang="bash">
 
apt-get update
 
apt-get install dselect
 
dselect update
 
apt-get -u dselect-upgrade
 
</source>
 
  
 
There are other tools to do the job too, and the tools you want to use for this depend on how automated you aim for this task to be, and how much you need/want to be able to customize each machine as it's being installed.
 
There are other tools to do the job too, and the tools you want to use for this depend on how automated you aim for this task to be, and how much you need/want to be able to customize each machine as it's being installed.
  
==KickStart==
+
== KickStart ==
  
 
'''[http://fedoraproject.org/wiki/Anaconda/Kickstart Kickstart]''' is a Red Hat package that deploys Red Hat to multiple installation targets with minimal customisation.  Aside from the wiki, it is also covered in [http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.3/html/Installation_Guide/index.html the Installation Guide for RedHat Enterprise Linux]
 
'''[http://fedoraproject.org/wiki/Anaconda/Kickstart Kickstart]''' is a Red Hat package that deploys Red Hat to multiple installation targets with minimal customisation.  Aside from the wiki, it is also covered in [http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.3/html/Installation_Guide/index.html the Installation Guide for RedHat Enterprise Linux]
  
==System Imager==
+
== System Imager ==
  
 
SystemImager is a third-party tool that does a better job. http://systemimager.org/
 
SystemImager is a third-party tool that does a better job. http://systemimager.org/
  
==Fully Automated Install==
+
== Fully Automated Install ==
  
 
'''[http://www.informatik.uni-koeln.de/fai/ fai]''' (fully automated install) is a Debian-based tool to do likewise.  Like System Imager, it's suitable for building clusters.  See the  
 
'''[http://www.informatik.uni-koeln.de/fai/ fai]''' (fully automated install) is a Debian-based tool to do likewise.  Like System Imager, it's suitable for building clusters.  See the  
 +
* [http://www.informatik.uni-koeln.de/fai/ project website] and the
 +
* [http://faiwiki.informatik.uni-koeln.de/index.php/Main_Page wiki].
 +
* [http://www.slideshare.net/henningsprang/automated-installations-and-infrastructure-management-with-fai-presentation presentation]
 +
* [[:Media:Fai_poster_a4.pdf|Fully Automated Install (FAI) Poster]]
  
*[http://www.informatik.uni-koeln.de/fai/ project website] and the
+
== Replicator ==  
*[http://faiwiki.informatik.uni-koeln.de/index.php/Main_Page wiki].
 
*[http://www.slideshare.net/henningsprang/automated-installations-and-infrastructure-management-with-fai-presentation presentation]
 
*[[File:Fai_poster_a4.pdf|600px|thumb|right|Fully Automated Install (FAI) Poster]]
 
 
 
==Replicator==  
 
  
 
'''Replicator''' seems like it could use some volunteers to expand. http://replicator.sourceforge.net/ It tries to do some customization for differences in hard disk sizes, video cards, etc.  You may want to check it out.
 
'''Replicator''' seems like it could use some volunteers to expand. http://replicator.sourceforge.net/ It tries to do some customization for differences in hard disk sizes, video cards, etc.  You may want to check it out.
  
==Partition Image==  
+
== Partition Image ==  
'''[http://www.partimage.org/ Partition Image]''' is a semi-automated tool for replicating a Linux partition to multiple targets.  It is particularly useful as a system cloning tool, and has many uses for data backup and archiving of whole disks or partitions.
+
'''[http://www.partimage.org/   Partition Image]''' is a semi-automated tool for replicating a Linux partition to multiple targets.  It is particularly useful as a system cloning tool, and has many uses for data backup and archiving of whole disks or partitions.
 
 
==Replicating a PHP installation==
 
With the PHP CLI installed, you can use <code>php -m</code> to get a list of the compiled-in modules for your PHP installation.  This is the same as you would find by using the <code>phpinfo();</code> function in a script, except that you get exactly the info you need, in a list format that you can use.  That list can show you what you might need to install on another host to ensure that each host is compatible for the software you're running.  However, it won't show you the name of the actual packages that you need to install.  So, if you want to know what PHP packages are installed, you can do the following
 
<source lang="bash">
 
# On system A, get a list of the php packages that are installed
 
## apt-based package manager: Debian, Ubuntu
 
dpkg --get-selections |grep -v deinstall | grep php5- | awk 'ORS=" " { print $1 }'
 
## yum-based package manager: RedHat, CentOS, Fedora
 
 
 
yum list installed | grep php5 | awk 'ORS=" " { print $1 }'
 
# On system B, install those packages
 
sudo apt-get install [LIST FROM A]
 
</source>
 
If system A and system B use different package managers, then the best you can do is get the list of the modules from each system and compare them.  In our <code>diff</code>, we'll use the system we're moving from on the left, because <code>diff</code> is quirky and will show lines that 'differ' with a pipe symbol; thus it's less error-prone to print the first column with awk.
 
<source lang="bash">
 
A=webfaction.php.modules.txt
 
 
 
B=digital.ocean.php.modules.txt
 
diff --suppress-common-lines --side-by-side $A $B
 
 
 
</source>
 
It's not perfect, but now we have a good list of the modules to search and install.
 
<pre>
 
>
 
curl
 
>
 
gd
 
gmp
 
imap
 
intl
 
ionCube
 
ldap
 
mailparse
 
mcrypt
 
memcache
 
mysqlnd
 
>
 
pdo_pgsql
 
pdo_sqlite
 
pgsql
 
pspell
 
>
 
sqlite3
 
tidy
 
>
 
>
 
>
 
xmlrpc
 
xsl
 
the
 
>
 
 
 
</pre>
 
 
 
 
 
See what's available on our new host.  (repositories are different)
 
<source lang="bash">
 
apt-cache search php5-
 
</source>
 
 
 
<pre>
 
php5-cgi - server-side, HTML-embedded scripting language (CGI binary)
 
php5-cli - command-line interpreter for the php5 scripting language
 
php5-common - Common files for packages built from the php5 source
 
php5-curl - CURL module for php5
 
php5-dbg - Debug symbols for PHP5
 
php5-dev - Files for PHP5 module development
 
php5-gd - GD module for php5
 
php5-gmp - GMP module for php5
 
php5-json - JSON module for php5
 
php5-ldap - LDAP module for php5
 
php5-mysql - MySQL module for php5
 
php5-odbc - ODBC module for php5
 
php5-pgsql - PostgreSQL module for php5
 
php5-pspell - pspell module for php5
 
php5-readline - Readline module for php5
 
php5-recode - recode module for php5
 
php5-snmp - SNMP module for php5
 
php5-sqlite - SQLite module for php5
 
php5-tidy - tidy module for php5
 
php5-xmlrpc - XML-RPC module for php5
 
php5-xsl - XSL module for php5
 
libphp5-embed - HTML-embedded scripting language (Embedded SAPI library)
 
php5-adodb - Extension optimising the ADOdb database abstraction library
 
php5-apcu - APC User Cache for PHP 5
 
php5-enchant - Enchant module for php5
 
php5-exactimage - fast image manipulation library (PHP bindings)
 
php5-fpm - server-side, HTML-embedded scripting language (FPM-CGI binary)
 
php5-gdcm - Grassroots DICOM PHP5 bindings
 
php5-gearman - PHP wrapper to libgearman
 
php5-geoip - GeoIP module for php5
 
php5-gnupg - wrapper around the gpgme library
 
php5-imagick - ImageMagick module for php5
 
php5-imap - IMAP module for php5
 
php5-interbase - interbase/firebird module for php5
 
php5-intl - internationalisation module for php5
 
php5-lasso - Library for Liberty Alliance and SAML protocols - PHP 5 bindings
 
php5-librdf - PHP5 language bindings for the Redland RDF library
 
php5-mapscript - php5-cgi module for MapServer
 
php5-mcrypt - MCrypt module for php5
 
php5-memcache - memcache extension module for PHP5
 
php5-memcached - memcached extension module for PHP5, uses libmemcached
 
php5-midgard2 - Midgard2 Content Repository - PHP5 language bindings and module
 
php5-ming - Ming module for php5
 
php5-mongo - MongoDB database driver
 
php5-msgpack - PHP extension for interfacing with MessagePack
 
php5-mysqlnd - MySQL module for php5 (Native Driver)
 
php5-mysqlnd-ms - MySQL replication and load balancing module for PHP
 
php5-oauth - OAuth 1.0 consumer and provider extension
 
php5-pinba - Pinba module for PHP 5
 
php5-ps - ps module for PHP 5
 
php5-radius - PECL radius module for PHP 5
 
php5-redis - PHP extension for interfacing with Redis
 
php5-remctl - PECL module for Kerberos-authenticated command execution
 
php5-rrd - PHP bindings to rrd tool system
 
php5-sasl - Cyrus SASL Extension
 
php5-stomp - Streaming Text Oriented Messaging Protocol (STOMP) client module for PHP 5
 
php5-svn - PHP Bindings for the Subversion Revision control system
 
php5-sybase - Sybase / MS SQL Server module for php5
 
php5-tokyo-tyrant - PHP interface to Tokyo Cabinet's network interface, Tokyo Tyrant
 
php5-vtkgdcm - Grassroots DICOM VTK PHP bindings
 
php5-xcache - Fast, stable PHP opcode cacher
 
php5-xdebug - Xdebug Module for PHP 5
 
php5-xhprof - Hierarchical Profiler for PHP5
 
</pre>
 
 
 
And thus, our install
 
<source lang="bash">
 
sudo apt-get install php5-curl php5-gd php5-gmp php5-imap php5-intl php5-ldap php5-mcrypt php5-memcache php5-mysqlnd php5-pgsql php5-pspell php5-sqlite php5-tidy php5-xmlrpc php5-xsl
 
</source>
 
  
 
[[Category:Howto]]
 
[[Category:Howto]]

Revision as of 22:29, 17 November 2008

Once you have backups, you can and should test the ability to restore those backups to a new hardware environment. The restore procedure is going to depend on what you used to create your "image" in the first place. Independent of the notion of backups, a system administrator (or even an experienced user) will want to "clone" a computer system. That is make an identical copy or nearly identical copy of a computer on different hardware. The most widespread use-case for this is when a person gets a new computer and wants to migrate the old computer software and data to the new hardware. From the system administrator's perspective, the common use case is different: it's when you have to build out a cluster, provide redundancy, or otherwise scale a hardware/software platform.

Consider all the times that you could produce a system that you wish to replicate. For example, a standard "Developer's Workstation" or "Designer's Studio" or "Student Desktop" and many more technical varieties like "Web Front-end", "Mail Relay" etc.

Apt-get --set-selections[edit | edit source]

One way to do this on Debian-based systems (including Ubuntu) is to use dpkg get/set-selections in combination with apt-get using it's dselect-upgrade option:

To clone the package installation state of System A to System B do the following:

on System A[edit | edit source]

sudo dpkg --get-selections > package-list.txt

copy package-list.txt from System A to System B

on System B[edit | edit source]

sudo dpkg --set-selections < package-list.txt
sudo apt-get -u dselect-upgrade


There are other tools to do the job too, and the tools you want to use for this depend on how automated you aim for this task to be, and how much you need/want to be able to customize each machine as it's being installed.

KickStart[edit | edit source]

Kickstart is a Red Hat package that deploys Red Hat to multiple installation targets with minimal customisation. Aside from the wiki, it is also covered in the Installation Guide for RedHat Enterprise Linux

System Imager[edit | edit source]

SystemImager is a third-party tool that does a better job. http://systemimager.org/

Fully Automated Install[edit | edit source]

fai (fully automated install) is a Debian-based tool to do likewise. Like System Imager, it's suitable for building clusters. See the

Replicator[edit | edit source]

Replicator seems like it could use some volunteers to expand. http://replicator.sourceforge.net/ It tries to do some customization for differences in hard disk sizes, video cards, etc. You may want to check it out.

Partition Image[edit | edit source]

Partition Image is a semi-automated tool for replicating a Linux partition to multiple targets. It is particularly useful as a system cloning tool, and has many uses for data backup and archiving of whole disks or partitions.