Importing contacts

Goal

Import records into SugarCRM without using existing functionality, but rather by writing some PHP tool to do the work. Records should go into Accounts and Contacts, which obviously implies a relationship.

Resources Used

Download and Install SugarCRM

I went to the SugarCRM homepage, and clicked the top navbar link to "Sugar Open Source"
I skipped the "Wizard" and went right to the "Download Page" because I know what I'm doing and also have a pre-existing setup of Linux, Apache MySQL and PHP - so I only want the application
I downloaded SugarCE-5.0.0e.zip (production release)
I visited the recommended "Installation" instructions page at http://www.sugarforge.org/content/installation/
Then I installed it according to the instructions, however I immediately ran into trouble because the instructions did say to chmod 766 all files that needed to be writable by the web_user.
This command renders the directories non-executable which manifests in include errors because the web user (www-data) can not see into those directories (to find includes). The following snippet is a fix:

# find directories in the ./crm path and change the mode on them so that all users can execute (see into) the directory 
find ./crm/ -type d |xargs chmod a+x

resolved the include errors

The instructions also fail to mention that the application wants to create a .htaccess file (which doesn't exist in the distribution and would not necessarily be writable to the web user. As a failsafe, the information is printed to the screen in the installer. However it doesn't properly display (lacking newline characters) so the content is not suitable for copy and paste into the file. I resorted to the source which generates the .htaccess content, and used that. An alternative is to simply touch and chmod 777 a .htaccess file prior to running the installer; and then chmod'ing it go=r after install.

Ideas on How to do the import

My self-imposed requirement was to capture all of the data provided -- assuming the owner of the data collected it for a reason and that they wouldn't be too happy with a loss of data in a migration to a new application. I looked briefly at the import functionality exposed by the application, but noticed how the lead capture did not reflect the same data profile as the source that I had.

Existing Work

My first instinct is always to find info about somebody doing this before (e.g. APIs, Documentation, forum questions, Wikis)

I searched the wiki for importing or APIs and quickly found the SOAP Intro and Practical Examples, so I figured I was onto a fast track. However, the soap example worked with a frontend form (which wasn't explained much) leading me to investigate more into what was required by the soap service. When I found the form in "examples", it did not work as an application entry point (because even though it defined ('sugarEntry', true); there was also an IF that pre-empted that definition) because other code was being loaded before the form. I did not immediately see this cause for the example failure, so I put the example form outside the application directory and successfully used the form to insert a single lead This is illustrated at http://www.sugarcrm.com/wiki/index.php?title=Creating_a_lead_capture_form_for_your_website Seeing that the example worked, but did not capture all the details provided in the data exercise, I knew I should look at the existing functionality of modules dealing with Import (dataMaps), the database abstraction layer (SugarBeans and VarDefs) or the database directly to get a clearer picture of what I needed to label everything.

Use the Source Luke

My second instinct was to look at the source. Answers are always found in the source, with the caveat that it can be confusing and/or time-consuming to find those answers.

Looking at the full application import routines and the four-step forms for importing leads, it was obvious that there was a lot of machinery that I didn't necessarily need or want to get involved with to simply import records. For instance, I would not want to (re-)write the form and UI handling when I simply needed to insert records into a database. This made me reconsider using the SOAP example to do the job. In essence, it would only need to read the exercise data and create full records for the 50 provided leads.

Using SOAP

Note: Soap examples seem to have changed with the addition of a version number, so you must be careful about what example you use, and generally rely on the source to be the definitive source ;-)

soap/SoapPortalUsers.php
soap/SoapSugarUsers.php
soap/SoapData.php
soap/SoapDeprecated.php

Getting on with the soap service, it is easy to discover the service profile and import the data

From the soap.php page (which shows the WSDL), we can get a definition of the available SOAP calls. For example, here is the definition of the 'create_account' method

Name: create_account
Binding: sugarsoapBinding
Endpoint: http://freephile.com/crm/soap.php
SoapAction: http://freephile.com/crm/soap.php/create_account
Style: rpc
Input:
  use: encoded
  namespace: http://www.sugarcrm.com/sugarcrm
  encodingStyle: http://schemas.xmlsoap.org/soap/encoding/
  message: create_accountRequest
  parts:
    user_name: xsd:string
    password: xsd:string
    name: xsd:string
    phone: xsd:string
    website: xsd:string
Output:
  use: encoded
  namespace: http://www.sugarcrm.com/sugarcrm
  encodingStyle: http://schemas.xmlsoap.org/soap/encoding/
  message: create_accountResponse
  parts:
    return: xsd:string
Namespace: http://www.sugarcrm.com/sugarcrm
Transport: http://schemas.xmlsoap.org/soap/http
Documentation:

And the create_contact method:

Name: create_contact
Binding: sugarsoapBinding
Endpoint: http://freephile.com/crm/soap.php
SoapAction: http://freephile.com/crm/soap.php/create_contact
Style: rpc
Input:
  use: encoded
  namespace: http://www.sugarcrm.com/sugarcrm
  encodingStyle: http://schemas.xmlsoap.org/soap/encoding/
  message: create_contactRequest
  parts:
    user_name: xsd:string
    password: xsd:string
    first_name: xsd:string
    last_name: xsd:string
    email_address: xsd:string
Output:
  use: encoded
  namespace: http://www.sugarcrm.com/sugarcrm
  encodingStyle: http://schemas.xmlsoap.org/soap/encoding/
  message: create_contactResponse
  parts:
    return: xsd:string
Namespace: http://www.sugarcrm.com/sugarcrm
Transport: http://schemas.xmlsoap.org/soap/http
Documentation:

Even more detail can be had by inspecting the service.

$get_available_modules_params = array (
  'session' => $session_id,
);
$result = $soapclient->call('get_available_modules', $get_available_modules_params);
print '<pre>'; var_export ($result); print '</pre>';

// output:

See Importing_contacts/sugar_modules

$interestedModules = array('Accounts', 'Contacts');

foreach ($interestedModules as $module_name) {
  $get_module_fields_params = array (
    'session' => $session_id,
    'module_name' => $module_name
  );
  $result = $soapclient->call('get_module_fields', $get_module_fields_params);
  print "\n $module_name fields:<br />\n<pre>\n"; var_export ($result); print "\n</pre>\n";
}

Accounts fields:

See Importing_contacts/Accounts

Contacts fields:

See Importing_contacts/Contacts

Setting Records

// read in our data from source CSV file format
$arrData = file('./account_contact_list.csv');
$separator = ',';
// initialize an array to hold structured data; we'll determine what columns we want and map our data
$leadData = array();
// take the first (header) row off the CSV file
$arrFields = explode($separator, array_shift($arrData));
// flipping the array gives us the integer value we need defined by it's name
// in other words, we now have an associative array of integers
$fieldKeys = array_flip($arrFields);

// break up our records
foreach ($arrData as $k => $v) {
  $arrData[$k] = explode($separator, $v);
}
// insert our Account records
foreach ($arrData as $record) {
  $set_entry_params = array(
    'session' => $session_id,
    'module_name' => 'Accounts',
    'name_value_list'=>array(
        array('name'=>'name',                       'value'=>$record[$fieldKeys['Company Name']]), // required
        array('name'=>'industry',                   'value'=>$record[$fieldKeys['last_name']]),
        array('name'=>'phone_fax',                  'value'=>$record[$fieldKeys['Fax']]),
        array('name'=>'billing_address_street',     'value'=>$record[$fieldKeys['Address']]),
        array('name'=>'billing_address_city',       'value'=>$record[$fieldKeys['City']]),
        array('name'=>'billing_address_state',      'value'=>$record[$fieldKeys['State']]),
        array('name'=>'billing_address_postalcode', 'value'=>$record[$fieldKeys['Zip Code']]),
        array('name'=>'billing_address_country',    'value'=>$record[$fieldKeys['Country']]),
        array('name'=>'description',                'value'=>$record[$fieldKeys['Notes']]),
        array('name'=>'phone_office',               'value'=>$record[$fieldKeys['Work Phone']]),
        array('name'=>'phone_alternate',            'value'=>$record[$fieldKeys['Other Phone']]),
        array('name'=>'email',                      'value'=>$record[$fieldKeys['Email']]),
        array('name'=>'website',                    'value'=>$record[$fieldKeys['website']]),
        array('name'=>'employees',                  'value'=>$record[$fieldKeys['employees']]),
        array('name'=>'ticker_symbol',              'value'=>$record[$fieldKeys['ticker_symbol']]),
        array('name'=>'assigned_user_id',           'value'=>$user_guid)
    )
  );
  
  $result = $soapclient->call('set_entry',$set_entry_params);
}
// insert Contacts
foreach ($arrData as $record) {
  $set_entry_params = array(
    'session' => $session_id,
    'module_name' => 'Contacts',
    'name_value_list'=>array(
        array('name'=>'first_name',                 'value'=>$record[$fieldKeys['First Name']]),
        array('name'=>'last_name',                  'value'=>$record[$fieldKeys['Last Name']]),
        array('name'=>'phone_fax',                  'value'=>$record[$fieldKeys['Fax']]),
        array('name'=>'primary_address_street',     'value'=>$record[$fieldKeys['Address']]),
        array('name'=>'primary_address_city',       'value'=>$record[$fieldKeys['City']]),
        array('name'=>'primary_address_state',      'value'=>$record[$fieldKeys['State']]),
        array('name'=>'primary_address_postalcode', 'value'=>$record[$fieldKeys['Zip Code']]),
        array('name'=>'primary_address_country',    'value'=>$record[$fieldKeys['Country']]),
        array('name'=>'phone_office',               'value'=>$record[$fieldKeys['Work Phone']]),
        array('name'=>'phone_other',                'value'=>$record[$fieldKeys['Other Phone']]),
        array('name'=>'email',                      'value'=>$record[$fieldKeys['Email']]),
        array('name'=>'account_name',               'value'=>$record[$fieldKeys['Company Name']]), // relation
        array('name'=>'assigned_user_id',           'value'=>$user_guid)
    )
  );
  
  $result = $soapclient->call('set_entry',$set_entry_params);
}

I didn't just arrive at the code magically. Instead, I just 'drew' a map of the columns of source data that I had, and marked where each data element would potentially fit into the target database table.

print '<pre>'; var_export ($arrFields); print '</pre>';
array (
  0 => 'Company Name',   // a:name                           
  1 => 'Industry',       // a:industry                       
  2 => 'First Name',     //                                  c:first_name
  3 => 'Last Name',      //                                  c:last_name
  4 => 'Fax',            // a:phone_fax                      c:phone_fax
  5 => 'Address',        // a:billing_address_street         c:primary_address_street
  6 => 'City',           // a:billing_address_city           c:primary_address_city
  7 => 'State',          // a:billing_address_state          c:primary_address_state
  8 => 'Zip Code',       // a:billing_address_postalcode     c:primary_address_postalcode
  9 => 'Country',        // a:billing_address_country        c:primary_address_country
  10 => 'Notes',         // a:description                    
  11 => 'Work Phone',    // a:phone_office                   c:phone_work
  12 => 'Other Phone',   // a:phone_alternate                c:phone_other
  13 => 'Email',         // a:email                          c:email1
  14 => 'website',       // a:website                        
  15 => 'employees',     // a:employees                      
  16 => 'ticker_symbol', // a:ticker_symbol
)

And there are a few extra elements like GUID that go into the actual scrip

The first attempt did not establish the record relationships the way that I intended by using the Company Name as the relation. It seemed plausible from the definition, but it didn't happen that way.

Some Issues discovered

"Import Step 2: Upload Export File" is a confusing title on the second step of the "Import Contacts" wizard. How can one use the words 'import', 'upload', 'export' all at once? It could read "Import Step 2: Select Data File to load"
Typographical bug: I saw that the term "custom_delimeted" in the language files. It doesn't actually cause any errors, but b/c the term is used elsewhere (spelled correctly), this typo could potentially lead to a real bug down the line.
1. ./modules/Import/language/en_us.lang.php
2. ./modules/Import/ImportStep1.html
3. ./modules/Import/ImportStep2.php
The source is formatted poorly, or practically not at all in some cases, due to various editors using different space and tab settings, and line endings. PHPBeautifier used in a commit hook would solve this in the repo. Coding standards and configuration files like vim modelines would solve this on the developer desktop.