This article describes the source code management and deployment process for SharePoint 2010 initaitives and provides the current-state of SharePoint 2007 deployment analysis as well as listing the various options and associated best practices for SharePoint 2010. This is a high-level EPC Group process we put in place for our clients but in follow-up articles I will take this to a much granular level.

Overview

The purpose of this article is to:

• Identify the different tools for source code management in SharePoint
• Provide best practices for source code management in SharePoint
• List the different tools for custom solution deployment in SharePoint
• Provide best practices for custom solution deployment in SharePoint  

Note: For both organizations coming from non-SharePoint platforms as well as upgrading from SharePoint 2007 to SharePoint 2010.

The information is taken from various EPC Group best practices and our published online sources and as a result of hands-on testing of the of SharePoint 2010 and Visual Studio 2010.  

Platforms – Microsoft Office 2010, Visual Studio 2010, SharePoint Designer 2010

Features – Document Management/Records Management, User/Developer Communities, shared code/service repositories.

Security – SharePoint Server Farm – Separation of the developer and live environment allows for disassociated and secure coding that does not require prior approval or cause risk conflict.  It is recommended to use this approach to ensure separation of the environments and to allow for more freedom in the development process.  Live environments (QA and Production) will be subject to the SharePoint Governance Plan.

Source Code Management

Source Code Management is considered part of Software Configuration Management (SCM) and encompasses the tools, procedures and actors involved in tracking software changes over time. Myself and my team at EPC Group have seen this be an area were organizations do to always focus their budget on and can sometimes be lacking which can lead to long-term issues.

Visual Studio 2010 is the primary tool for development in SharePoint 2010 and it comes with out of the box integration support for Team Foundation Server (TFS) 2010.  TFS is more than an SCM tool as it also provides a data warehouse to store project schedules, allows for continuous integration and builds, project management and many other features. TFS is currently not an approved SCM tool per the EA SCM standards.  TFS provides the basis for Application Lifecycle Management (ALM) – an area that encompasses more than just SCM.

Note: While TSF has better integration with Visual Studio, it does require licenses to be purchased.

Introduction

Deployment is required to take a SharePoint Customization or Development Solution and put it on a server.  Development and deployment in SharePoint is different from development in .NET (ASP.NET, Silverlight etc).  It involves steps that are unique to SharePoint (creating the DDF file, creating the WSP etc).  These steps apply when deploying SharePoint solution to all environments (QA,Production, Contingency, etc.). 

Deployment Requirements/Objectives

 A SharePoint deployment solution should, in most cases, meet the following objectives:

• Implementation of packaged code in development environment:  The deployment solution should be able to perform all the steps (create, build, package, deploy). for SharePoint deployments right within the development environment (usually Visual Studio).

• Modification of deployment options and parameters: Ideally developers should be able to modify the deployment parameters and options when deploying a SharePoint solution to different environments.  It is not desirable to have a  solution that locks developers down with an obfuscated process which is totally opaque and does not allow the team to control or modify any step of the deployment process if needed.  Having a customizable solution that developers can use to create custom deployments allow flexibility and consistency across environments.

• Extensible:  The SharePoint deployment process contains the following high-level steps: build, package and deploy.  A deployment solution should provide ways to extend a given step to fit the specific policies of the organization or business unit for compliance, administrative/operational consistency and other reasons.

• Retraction of deployment solutions: A developer goes through multiple iterations of development and deployment while developing a solution.  A deployment solution should be able to perform cyclical retraction of SharePoint solutions and support continuous dev->test cycles of SharePoint Development.

• Elimination of repetitive manual processes that reduce productivity

• Single-click build/package/deploy process integrated into Visual Studio IDE:  Ideally  a developer can hit a single key and SharePoint will perform all the steps for deployment within Visual Studio.

Legacy SharePoint Deployments

In SharePoint 2007, developers were introduced to the solution packaging framework which provided the ability to package multiple kinds of features or components like features, assemblies, and files into a single deployable artifact.  The overall deployment process in SharePoint contained the following steps (shown in the picture below):

• Create the solution: All the required elements of the SharePoint solution are created and implemented within this step.  This is usually performed within Visual Studio.

• Build the solution: Use MSBuild or custom scripts to build the solution.

• Package the solution:  This involves creating the WSP for the SharePoint solution.  This step involved different steps: creating DDF file, running the MAKECAB utility etc.

• Deploy the solution: This involves taking a WSP file and running the appropriate STSADM commands to deploy the WSP file into the SharePoint server so the solution is usable.

Visual Studio 2008 did not provide any standard in-built support for building, packaging and deploying a SharePoint solution to a SharePoint farm.  However, there were some point solutions/add-on tools (STSDEV, WSPBuilder, and VSSWe etc) available for Visual Studio 2008 for some of these functions.  All of these tools automated some of the steps in the SharePoint deployment process but fell short of the intended goal. 

Legacy Deployment Limitations

In spite of these tools, the developers still had to write custom scripts to tie all the steps of the deployment process together.   In a nutshell, the following were some of the limitations of the SharePoint 2007 deployment process:

• Poor Integration:  The legacy deployment process required introduction of third-party tools to automate some of the steps in the deployment process.  They were not integrated into Visual Studio and Visual Studio was not SharePoint aware as well.  Due to the lack of integration between SharePoint and Visual Studio, each element of the solution existed in isolation with no direct relationship to each other. 
• Making Communication Confusion:  The legacy deployment process was confusing and did not expose the specific steps of the deployment process.
• Modification:  There was no room to modify individual steps of the deployment process to meet edge case scenarios.
• Automation: There was no way to automate the deployment process.
• Retraction:  There was no way to retract an existing solution other than to write a custom script.
• Upgrades:  There was no way to upgrade an existing solution.
• Artifact Removal
• Conflict Resolution:  The legacy deployment solution did not provide conflict resolution in deployment scenarios where the new solution creates a new version of existing SharePoint artifacts e.g. list definitions etc.

SharePoint 2010 Deployment

Deployment in SharePoint 2010 still uses the same four deployment steps as in the Legacy Deployment but the new deployment process implementation in SharePoint 2010 fixes the limitations in the Legacy Deployment process. 

Visual Studio 2010 Integration

Visual Studio 2010 provides first-class support for working with SharePoint projects in the following ways:

• Single unified interface for the building, packaging and deployment processes:  With the press of the F5 key, Visual Studio 2010 can perform all the necessary steps involved in the SharePoint deployment process.  It can also associate the Visual Studio to the SharePoint process and thus, allow the developer to debug the SharePoint solution right after the deployment.
• Auto-discovery of mapped assets:  Visual Studio 2010 now provides mapped folders that can contain things like images, CSS files etc.  Visual Studio will automatically create module definitions for these artifacts and automatically include them in the WSP when built by Visual Studio 2010.
• Visual Package Explorer: Visual Studio 2010 also provides a nice visual explorer to be able to easily browse through the various components (features and folders) of the SharePoint solution (also called SharePoint Items or SPI).
• Edit-in-place configuration files (manifest.xml):  Visual Studio 2010 now provides very easy-to-understand and use visual designers to work with the underlying manifest definition files.  In addition, one can get to the underlying XML file and hand-edit them for complicated scenarios if needed.
• Extensible framework:  Visual Studio allows creation of custom projects, SharePoint items and thus, provides ways to extend the deployment process in a customized way to fit various requirements.

Creating solutions

Visual Studio 2010 provides pre-built artifacts called SharePoint Items (or SPI).   These SPIs cover most of the commonly used SharePoint solution components that one uses.  These SPIs are pre-configured artifacts for common SharePoint projects and when selected and added to a Visual Studio 2010 SharePoint solution also automate generation of supporting artifacts (features, manifests, .webpart files etc) if such a feature does not already exist.  The SPIs are also aware of the Visual Studio structure for a SharePoint project.  As noted earlier, Visual Studio 2010 provides an extension framework where new custom SPIs can be created, added and used by others.

Building solutions

Visual Studio 2010 exposes properties to control the build process via a Property Sheet.  These options include:

• URL of SharePoint site to which the solution will be deployed
• Type of deployment: Sandbox vs. Farm
• Target of deployment: Global Assembly Cache (GAC) or BIN deployment

Packaging solutions

Packaging in SharePoint still happens between the build and deploy steps. 

It is possible for a SharePoint 2010 solution to contain multiple SharePoint projects.  By default, Visual Studio creates a single WSP per project and so, in case of a solution with multiple projects, Visual Studio will create multiple WSP files.  The default deployment process within SharePoint 2010 builds all the projects within the SharePoint solution but only the ones marked as “start up” will be deployed. Though Visual Studio 2010 provides ways to customize so we could package output from all SharePoint projects with the same solution to write to the same WSP, it is recommended as a best practice to use a single WSP per SharePoint custom "project".

Deploying solutions

The last step in the SharePoint Deployment process: the deployment step has totally changed from the previous legacy deployment process.  Deployment step in SharePoint 2010 consists of many components covered in the next few sections.

Deployment View

Visual Studio 2010 provides deployment designer views that allow customization of the deployment step for a SharePoint project.  Visual Studio 2010 adds a SharePoint tab under SharePoint project properties.  On clicking this tab, the designer will look as follows:

Pre-deployment tasks could include initializing directories or creating directories before a SharePoint deployment while post-deployment tasks could include clean up tasks once the SharePoint deployment is complete.   

Deployment configuration

Visual Studio 2010 provides deployment configurations for use in SharePoint deployment.  A deployment configuration is a set of deployment steps with each step associated with a deployment action.  With each deployment configuration, we can associate pre- and post-deployment tasks as shown in the Deployment View.  Out of the box, Visual Studio 2010 comes with two deployment configurations:  Default and No Activation.  The only difference between the two is the latter does not activate the solution in SharePoint after deployment. 

Typically, one will have to create custom deployment configurations to meet their specific deployment requirements.  A SharePoint project can contain multiple deployment configurations but only one active deployment configuration at a time.  When building a SharePoint project for deployment, the deployment configuration to use can be specified as a parameter.

Deployment Step

A deployment step implements a specific step within a deployment configuration and is associated with an action.  When creating a deployment configuration, one has to specify the deployment steps included in the deployment configuration and the order of execution of the deployment steps.  Each step consists of one or more tasks.  Pre-defined tasks exists for common tasks like recycling the IIS Application Pool, Retracting a solution, Activate a Feature etc. but using the Visual Studio 2010 Extensibility Framework, it is possible to create custom task definitions and plug them into the deployment process by association with a suitable deployment configuration. 

Conflict Resolution

One of the limitations of the legacy deployment solution in SharePoint 2007 was no way to do conflict resolution.  With the integration between Visual Studio 2010 and SharePoint 2010, a developer can perform conflict resolution.  There is built-in conflict resolution for SPIs provided out-of-the box with Visual Studio 2010.  Developers can also customize the conflict resolution and detection logic using custom SPIs.  There are three main ways for conflict resolution for in-built SPIs:

• Automatic
• Prompt
• None

Thus, conflict resolution in SharePoint 2010 deployment is very powerful when used from Visual Studio 2010 as it is specific to each SPI.

Remote Deployment

Using Visual Studio 2010 to build and deploy a solution only works for deployments to the local SharePoint Server or deploying a solution to an integration environment.  Typically in most projects, the developers do not have rights to deploy a custom solution to either QA or the Production environment.  A SharePoint Administrator would be responsible for deploying a custom solution to the farm in the QA or Production environments. 

The development team has to hand the solution binary (WSP file) over to the SharePoint Administrator for deployment.  It is recommended that the development team creates custom deployment configurations for deployments to each environment:  Integration, QA and Production so that there is a consistent, well-defined and extensible set of steps associated with deployments in each environment. 

Tools Involved

As indicated earlier, SharePoint 2010 provides multiple tools for deployment depending on the nature of deployment (local vs. remote).  For local deployments on a developer or integration environment, Visual Studio 2010 is a recommended solution.  For all QA and Production deployments, Power Shell or a custom scripting solution will meet the requirements.

Solution Lifecycle In SharePoint

When deploying a custom solution (a WSP file) to SharePoint, following actions happen:

• Add the given WSP file to the Solution Store
• Solution Deployment and Distribution: Distribute, unpack and install the solution to the front-end Web Servers
• Synchronization if needed

Adding a solution to the Solution Store

Adding a solution to the solution store can be done either using the command line, Web Administrative interface or through the SharePoint Object Model.

Solution Deployment and Distribution

It is possible to use a timer job to do deployments.  In such cases, the deployments create a timer job.  This timer job is picked up by the timer service present in each front-end Web Server.  The timer job uses the SharePoint Foundation Administrative Web Service to access appropriate privileges to deploy solution files to each computer. 

Deployment Best Practices

Some of the best practices for SharePoint Deployment are:

• Use out-of-the box SPIs to implement the various parts of the SharePoint Development solution.  Only create custom SPIs if the out-of-the box SPIs do not meet your requirements.
• Collaborate with the team to discuss and decide the deployment configurations to use for each environment.  Ensure that the deployment configuration is agreed upon with everyone, is well documented and is extensible if necessary.
• For remote deployments, use a PowerShell like standard script language to automate and implement steps of the deployment process outside Visual Studio.
• Implement a Continuous Integration Platform and integrate that into the Deployment Process.  Then, decide on the frequency and schedule of the deployments to the Integration environment and implement an automated deployment solution.  This will enable the the development to quality check the integration environment on a daily basis ensuring that the latest code in SCM is working in an operational environment.
• Create a QA environment which looks exactly like Production. 
• Use proper and adequate logging and exception handling in custom SharePoint solutions.  This will help with troubleshooting problems encountered with a SharePoint solution in QA or Production where one does not have access to Visual Studio to debug the problem.
• Ensure that the deployment scripts and custom deployment extensions are also versioned in the SCM tool along with the source code written for the custom SharePoint solution.
• If possible, use sandbox solutions for testing a solution rather than using a farm solution.