Quantcast
Channel: Microsoft SQL Server Integration Services
Viewing all 149 articles
Browse latest View live

Insert, update and delete records in CRM 2013 with SSIS

$
0
0
Case
I have a business application with clients, accounts and other data and I want to make an interface to Microsoft Dynamics CRM 2013. How do you insert, update or delete records in CRM with SSIS?

Solution
Although the CRM data is stored in a regular SQL Server database, you’re not allowed to insert or change the data in those tables (otherwise you will lose Microsoft support on CRM).

There are a couple of third party CRM destination components available like CozyRoc and BlueSSIS. This solution uses a Script Component that inserst/updates/deletes data in CRM via a webservice and a CRM SDK assembly. There are other .Net solutions with for example a custom assembly or a service reference. All roads lead to Rome, but this is a relatively easy script for those with less .Net skills.

1) Guidlines
The first part of this solution will probably be different for everybody, but there are a few guidelines.
  1. To make sure you don’t insert records twice, you need a key from your business application in CRM. That could be a hidden field in CRM, but it allows you to check whether the record already exists in CRM.
  2. To update or delete records in CRM you need the guid (the unique key or entity id) from the CRM record.
So you want to make a lookup or a join to the CRM view to get the required data.
  1. You probably don’t want to update records unnecessarily. This is slow and pollutes the CRM history.
In this solution I selected the exact same columns from CRM and added the GUID. I joined the two sources on the business key with a left outer join. If the business key on the CRM part is null then it’s an insert else it could be an update.
You could compare all column the check whether you need an update or not, but because I have about 20 columns to check this could end up in a huge, unreadable and unmaintainable expression. I used a checksum transformation to calculate a hash from all columns and then I only have to compare those two hashes. You could also do this in the source query with some TSQL code.
example package























Below I will describe those Script Components.

2) Download CRM SDK
For this example I used CRM 2013 and I downloaded the free Microsoft Dynamics CRM 2013 Software Development Kit (SDK). Execute the downloaded file to extract all the files. We only need Microsoft.Xrm.Sdk.dll assembly which can be found in the SDK\Bin folder.
SDK download















3) DLL to SSIS machine
To use the assembly (DLL) from step 2 in SSIS, you need to add the DLL to the Global Assembly Cache (GAC) on your SSIS machine. Here is an example for adding to the gac on Win Server 2008 R2. You also need to copy it to the Binn folder of SSIS: D:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\

4) Parameters
To avoid hardcoded usernames, domainnames, passwords and webservices, I created four project parameters (SSIS 2012). These parameters will be used in the Script Components.
4 parameters, password is marked sensitive.







5) Script Component - Parameters
Add a Script Component (type destination) for Insert, Update or Delete. When you edit the Script Component make sure to add the four parameters from the previous step as read only variables.
Add parameters as read only variables


























6) Script Component - Input Columns
Add the columns that you need as Input Columns. For the Insert you only need the columns from your business application. For the update you also need the entity ID from CRM. This is the Technical Id (a guid) from the CRM entity that you want to update. For a delete you only need that entity ID.
Input Columns for insert























7) The Script - Add assembly
Hit the Edit Script button to start the VSTA editor. In the solution explorer you need to add three references:
  • microsoft.xrm.sdk.dll (from the SSIS bin folder mentioned in step 3, use browse)
  • System.Runtime.Serialization.dll (from .Net tab)
  • System.ServiceModel.dll (from .Net tab)
Right click references and choose add reference

















Now very important: press the Save All button to save the entire internal vsta project (including references)
Save All













8a) The Script - Insert
Here is an C# example (for VB.Net use this translator) for inserting CRM records with SSIS 2012.
// C# Code
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.Xrm.Sdk; // Added
using Microsoft.Xrm.Sdk.Client; // Added
using Microsoft.Xrm.Sdk.Query; // Added
using System.ServiceModel.Description; // Added

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
// Webservice
IOrganizationService organizationservice;

// Variables for the CRM webservice credentials
// You could also declare them in the PreExecute
// if you don't use it anywhere else
string CrmUrl = "";
string CrmDomainName = "";
string CrmUserName = "";
string CrmPassWord = "";

// This method is called once, before rows begin to be processed in the data flow.
public override void PreExecute()
{
base.PreExecute();

// Fill variables with values from project parameters
CrmUrl = this.Variables.CrmWebservice.ToString();
CrmDomainName = this.Variables.CrmDomain.ToString();
CrmUserName = this.Variables.CrmUser.ToString();
CrmPassWord = this.Variables.CrmPassword.ToString();

// Connect to webservice with credentials
ClientCredentials credentials = new ClientCredentials();
credentials.UserName.UserName = string.Format("{0}\\{1}", CrmDomainName, CrmUserName);
credentials.UserName.Password = CrmPassWord;
organizationservice = new OrganizationServiceProxy(new Uri(CrmUrl), null, credentials, null);
}

// This method is called once for every row that passes through the component from Input0.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
// Create a Entity object of type 'account'
Entity newAccount = new Entity("account");

// Store the business key of the source in CRM
// This makes it easier to compare and filter records for update
newAccount["cst_caressid"] = Row.CaressId;

// fill crm fields. Note fieldnames are case sensitive!
newAccount["name"] = Row.AccountName;
newAccount["emailaddress1"] = Row.Email;
newAccount["telephone1"] = Row.Phone;

// Address, but check if the columns are filled
if (!Row.Street_IsNull)
{
newAccount["address1_line1"] = Row.Street;
}

if (!Row.Housenumber_IsNull)
{
newAccount["address1_line2"] = Row.Housenumber;
}

if (!Row.Zipcode_IsNull)
{
newAccount["address1_postalcode"] = Row.Zipcode;
}

if (!Row.Residence_IsNull)
{
newAccount["address1_city"] = Row.Residence;
}

if (!Row.Country_IsNull)
{
newAccount["address1_country"] = Row.Country;
}

// Filling a OptionSet (dropdownbox) is a little different
// You need to know the codes defined in CRM. You need
// CRM knowledge to find those so ask the CRM consultant.
OptionSetValue accountType = new OptionSetValue();
if (!Row.AccountType_IsNull)
{
switch (Row.AccountType)
{
case "Large":
accountType.Value = 1;
break;
case "Medium":
accountType.Value = 2;
break;
case "Small":
accountType.Value = 3;
break;
default:
accountType.Value = 2;
break;
}
newAccount.Attributes.Add("accounttype", (OptionSetValue)accountType);
}

// Create account
organizationservice.Create(newAccount);
}
}


8b) The Script - Update
Here is an C# example (for VB.Net use this translator) for updating CRM records with SSIS 2012.
// C# Code
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.Xrm.Sdk; // Added
using Microsoft.Xrm.Sdk.Client; // Added
using Microsoft.Xrm.Sdk.Query; // Added
using System.ServiceModel.Description; // Added

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
// Webservice
IOrganizationService organizationservice;

// Variables for the CRM webservice credentials
// You could also declare them in the PreExecute
// if you don't use it anywhere else
string CrmUrl = "";
string CrmDomainName = "";
string CrmUserName = "";
string CrmPassWord = "";

// This method is called once, before rows begin to be processed in the data flow.
public override void PreExecute()
{
base.PreExecute();

// Fill variables with values from project parameters
CrmUrl = this.Variables.CrmWebservice.ToString();
CrmDomainName = this.Variables.CrmDomain.ToString();
CrmUserName = this.Variables.CrmUser.ToString();
CrmPassWord = this.Variables.CrmPassword.ToString();

// Connect to webservice with credentials
ClientCredentials credentials = new ClientCredentials();
credentials.UserName.UserName = string.Format("{0}\\{1}", CrmDomainName, CrmUserName);
credentials.UserName.Password = CrmPassWord;
organizationservice = new OrganizationServiceProxy(new Uri(CrmUrl), null, credentials, null);
}

// This method is called once for every row that passes through the component from Input0.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
// Create a Entity object of type 'account'
Entity existingAccount = new Entity("account");

// Most important attribute to fill is the entity id
// This is a GUID column from CRM. Without this
// column you can't update records in CRM.
existingAccount["accountid"] = Row.AccountId;

// Since we joined on the business key, it shouldn't
// be updated. That why this line is a comment.
// existingAccount["cst_caressid"] = Row.CaressId;

// fill crm fields. Note fieldnames are case sensitive!
existingAccount["name"] = Row.AccountName;
existingAccount["emailaddress1"] = Row.Email;
existingAccount["telephone1"] = Row.Phone;

// Address, but check if the columns are filled
if (!Row.Street_IsNull)
{
existingAccount["address1_line1"] = Row.Street;
}

if (!Row.Housenumber_IsNull)
{
existingAccount["address1_line2"] = Row.Housenumber;
}

if (!Row.Zipcode_IsNull)
{
existingAccount["address1_postalcode"] = Row.Zipcode;
}

if (!Row.Residence_IsNull)
{
existingAccount["address1_city"] = Row.Residence;
}

if (!Row.Country_IsNull)
{
existingAccount["address1_country"] = Row.Country;
}

// Filling a OptionSet (dropdownbox) is a little different
// You need to know the codes defined in CRM. You need
// CRM knowledge to find those so ask the CRM consultant.
OptionSetValue accountType = new OptionSetValue();
if (!Row.AccountType_IsNull)
{
switch (Row.AccountType)
{
case "Large":
accountType.Value = 1;
break;
case "Medium":
accountType.Value = 2;
break;
case "Small":
accountType.Value = 3;
break;
default:
accountType.Value = 2;
break;
}
existingAccount.Attributes.Add("accounttype", (OptionSetValue)accountType);
}

// Reference to an other entity
EntityReference Contact = new EntityReference("contact", Row.ClientGuid);
existingAccount["contactid"] = Contact;

// Update account
organizationservice.Update(existingAccount);
}
}


8c) The Script - Delete
Here is an C# example (for VB.Net use this translator) for deleting CRM records with SSIS 2012. It wasn't mentioned in the solution example. You need a full outer join for it. Warning: this is a physical delete which can't be undone. In an other post I will publish an inactivate script example, but it uses other assemblies.
// C# Code
using System;
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.Xrm.Sdk; // Added
using Microsoft.Xrm.Sdk.Client; // Added
using Microsoft.Xrm.Sdk.Query; // Added
using System.ServiceModel.Description; // Added

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
// Webservice
IOrganizationService organizationservice;

// Variables for the CRM webservice credentials
// You could also declare them in the PreExecute
// if you don't use it anywhere else
string CrmUrl = "";
string CrmDomainName = "";
string CrmUserName = "";
string CrmPassWord = "";

// This method is called once, before rows begin to be processed in the data flow.
public override void PreExecute()
{
base.PreExecute();

// Fill variables with values from project parameters
CrmUrl = this.Variables.CrmWebservice.ToString();
CrmDomainName = this.Variables.CrmDomain.ToString();
CrmUserName = this.Variables.CrmUser.ToString();
CrmPassWord = this.Variables.CrmPassword.ToString();

// Connect to webservice with credentials
ClientCredentials credentials = new ClientCredentials();
credentials.UserName.UserName = string.Format("{0}\\{1}", CrmDomainName, CrmUserName);
credentials.UserName.Password = CrmPassWord;
organizationservice = new OrganizationServiceProxy(new Uri(CrmUrl), null, credentials, null);
}

// This method is called once for every row that passes through the component from Input0.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
// Warning: this is a physical delete from CRM which can't be undone

// Delete account. First part in the entityname, second
// is the entity id from the CRM source.
organizationservice.Delete("account", Row.AccountId);
}
}

Note: Downside of this method is that you will get an error when you run the script in 64bit: Fault Detail is equal to Microsoft Xrm.Sdk.OrganizationServiceFault. Solution is to run the package in 32bit. If I find a cause or solution then I will update  this post.

Deploying SSIS 2012 Environments

$
0
0
Case
Parameters and environments are a handy new feature in SSIS 2012, but creating environments with variables and referencing the environment to the project and the variables to the parameters is a lot of effort. No problem if you have to do it once, but if you deploy your packages through the DTAP servers you have to redo it on every single server. Is there a way to deploy environments?

No export option for Environments















Solution
Although it is easy to deploy packages through DTAP servers. Deploying environments is not  possible. That could get a little tiresome especially if you have a lot of parameters in your project.

I have created a script/stored procedure that uses the project identifier as input, loops through the tables of the SSISDB and prints SQL-statements to execute SSISDB Stored Procedures.
How to get the project identifier
















For every environment, variable and reference to this project it will generate creation scripts. You can copy these stored procedure calls, adjust the values where needed and execute them on the next environment.
Execute script then copy and execute output

























Release notes
  1. There are no checks or validations in this version. So it doesn't check whether objects already exist before calling the SSISDB Stored Procedures.
  2. First deploy the SSIS project before executing the stored procedure calls on the next server.
  3. Make sure the folder name is equal on the new server or change it in the stored procedure calls.
  4. Make sure to check sensitive variables values (you can't get the value from the tables).


Here is how you call the stored procedure to generate the scripts. If you don't want to add the new stored procedure then you could just use the script inside the stored procedure.
exec catalog.deploy_environment 11

Add the following stored procedure to the SSISDB or use the TSQL code inside.
USE SSISDB;
GO

-- USE AT OWN RISK! This stored procedure was created on the SSISDB on SQL Server version:
-- Microsoft SQL Server 2012 (SP1) - 11.0.3128.0 (X64)
-- Dec 28 2012 20:23:12
-- Copyright (c) Microsoft Corporation
-- Developer Edition (64-bit) on Windows NT 6.1 <x64> (Build 7601: Service Pack 1)


-- Drop any previous versions of this stored procedure
IF OBJECT_ID ( 'catalog.deploy_environment', 'P' ) IS NOT NULL
DROP PROCEDURE catalog.deploy_environment;
GO

-- project_id is the identifier in the properties of a project
CREATE PROCEDURE catalog.deploy_environment
@project_id bigint
AS

-- Internal variables used within the cursor
Declare @environment_name as nvarchar(128);
Declare @project_name as nvarchar(128);
Declare @folder_name as nvarchar(128);
Declare @environment_folder_name as nvarchar(128);
Declare @reference_type as char(1);
Declare @folder_id as bigint;
Declare @environment_description as nvarchar(1024);
Declare @environment_id as bigint;


DECLARE ref_environment_cursor CURSOR FOR
-- Loop through all in the project referenced Environments
SELECT r.environment_name
, p.name as project_name
, ISNULL(r.environment_folder_name, f.name) as folder_name
, ISNULL(r.environment_folder_name, f.name) as environment_folder_name -- for @reference_type = A
, r.reference_type as reference_type
, f.folder_id
, e.description as environment_description
, e.environment_id
FROM [SSISDB].[internal].environment_references as r
INNER JOIN [SSISDB].[internal].projects as p
on r.project_id = p.project_id
INNER JOIN [SSISDB].[internal].folders as f
on p.folder_id = f.folder_id
INNER JOIN [SSISDB].[internal].environments as e
on e.folder_id = f.folder_id
and e.environment_name = r.environment_name
WHERE r.project_id = @project_id

OPEN ref_environment_cursor

FETCH NEXT FROM ref_environment_cursor
INTO @environment_name, @project_name, @folder_name, @environment_folder_name, @reference_type, @folder_id, @environment_description, @environment_id;

Print '-- Create scripts for deploying enviroments'
Print '-- Project ID: ' + CAST(@project_id as varchar(5)) + ' - Project name: ' + @project_name
Print ''

WHILE @@FETCH_STATUS = 0
BEGIN
-- Create environment
Print '-- Create environment: ' + @environment_name
Print 'EXEC [SSISDB].[catalog].[create_environment]'
Print ' @environment_name=N''' + @environment_name + ''''
Print ', @environment_description=N''' + @environment_description + ''''
Print ', @folder_name=N''' + @folder_name + ''''
Print 'GO'
Print ''

-- Create reference from environment to project. Relative or Absolute
Print '-- Reference environment ' + @environment_name + ' to project ' + @project_name
IF @reference_type = 'R'
BEGIN
-- Reference Relative
Print 'Declare @reference_id bigint'
Print 'EXEC [SSISDB].[catalog].[create_environment_reference]'
Print ' @environment_name=N''' + @environment_name + ''''
Print ', @reference_id=@reference_id OUTPUT'
Print ', @project_name=N''' + @project_name + ''''
Print ', @folder_name=N''' + @folder_name + ''''
Print ', @reference_type=R'
Print 'GO'
Print ''
END
ELSE
BEGIN
-- Reference Absolute
Print 'Declare @reference_id bigint'
Print 'EXEC [SSISDB].[catalog].[create_environment_reference]'
Print ' @environment_name=N''' + @environment_name + ''''
Print ', @environment_folder_name=N''' + @environment_folder_name + ''''
Print ', @reference_id=@reference_id OUTPUT'
Print ', @project_name=N''' + @project_name + ''''
Print ', @folder_name=N''' + @folder_name + ''''
Print ', @reference_type=A'
Print 'GO'
Print ''
END


-- Internal variables used within the cursor
Declare @environment_value as sql_variant--nvarchar(max); -- SQL_VARIANT
Declare @variable_name as nvarchar(128);
Declare @sensitive as bit;
Declare @variable_description as nvarchar(1024);
Declare @variable_type as nvarchar(128);

DECLARE environment_var_cursor CURSOR FOR
-- Loop through all in the variables of the active environment
SELECT CAST(ev.value as varchar(255)) as environment_value
, ev.name as variable_name
, ev.sensitive
, ev.description as variable_description
, ev.type as variable_type
FROM [SSISDB].[catalog].environment_variables as ev
WHERE environment_id = @environment_id

OPEN environment_var_cursor

FETCH NEXT FROM environment_var_cursor
INTO @environment_value, @variable_name, @sensitive, @variable_description, @variable_type;

WHILE @@FETCH_STATUS = 0
BEGIN
-- Environments variables
Print '-- Create variables for environment: ' + @environment_name + ' - ' + @variable_name

-- Variable declaration depending on the type within the environment
IF @variable_type = 'Boolean'
BEGIN
Print 'DECLARE @var bit = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Byte'
BEGIN
Print 'DECLARE @var tinyint = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'DateTime'
BEGIN
Print 'DECLARE @var datetime = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Decimal'
BEGIN
Print 'DECLARE @var decimal(38,18) = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Double'
BEGIN
Print 'DECLARE @var float = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Int16'
BEGIN
Print 'DECLARE @var smallint = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Int32'
BEGIN
Print 'DECLARE @var int = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Int64'
BEGIN
Print 'DECLARE @var bigint = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'SByte'
BEGIN
Print 'DECLARE @var smallint = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'Single'
BEGIN
Print 'DECLARE @var float = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'String'
BEGIN
Print 'DECLARE @var sql_variant = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'UInt32'
BEGIN
Print 'DECLARE @var bigint = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END
ELSE IF @variable_type = 'UInt64'
BEGIN
Print 'DECLARE @var bigint = N''' + ISNULL(CAST(@environment_value as nvarchar(max)),'*S*E*N*S*I*T*I*V*E*') + ''''
END

Print 'EXEC [SSISDB].[catalog].[create_environment_variable]'
Print ' @variable_name=N''' + @variable_name + ''''
IF @sensitive = 0
BEGIN
Print ', @sensitive=False'
END
ELSE
BEGIN
Print ', @sensitive=True'
END
Print ', @description=N''' + @variable_description + ''''
Print ', @environment_name=N''' + @environment_name + ''''
Print ', @folder_name=N''' + @folder_name + ''''
Print ', @value=@var'
Print ', @data_type=N''' + @variable_type + ''''
Print 'GO'
Print ''

FETCH NEXT FROM environment_var_cursor
INTO @environment_value, @variable_name, @sensitive, @variable_description, @variable_type;
END
CLOSE environment_var_cursor;
DEALLOCATE environment_var_cursor;
-- End Environments variables

-- Parameter - Variable mapping
Declare @object_type as smallint
Declare @parameter_name as nvarchar(128);
Declare @object_name as nvarchar(260);
Declare @folder_name2 as nvarchar(128);
Declare @project_name2 as nvarchar(128);
Declare @value_type as char(1)
Declare @parameter_value as nvarchar(128);

DECLARE parameter_var_cursor CURSOR FOR
-- Loop through variables referenced to a parameter
SELECT op.object_type
, parameter_name
, [object_name]
, f.name as folder_name
, p.name as project_name
, value_type
, referenced_variable_name as parameter_value
FROM [SSISDB].[internal].object_parameters as op
INNER JOIN [SSISDB].[internal].projects as p
on p.project_id = op.project_id
INNER JOIN [SSISDB].[internal].folders as f
on p.folder_id = f.folder_id
WHERE op.project_id = @project_id
AND referenced_variable_name is not null

OPEN parameter_var_cursor

FETCH NEXT FROM parameter_var_cursor
INTO @object_type, @parameter_name, @object_name, @folder_name2, @project_name2, @value_type, @parameter_value;

WHILE @@FETCH_STATUS = 0
BEGIN
-- Reference variables
Print '-- Reference variable ' + @parameter_value + ' to parameter ' + @parameter_name
Print 'EXEC [SSISDB].[catalog].[set_object_parameter_value]'
Print ' @object_type=' + CAST(@object_type as varchar(5))
Print ', @parameter_name=N''' + @parameter_name + ''''
Print ', @object_name=N''' + @object_name + ''''
Print ', @folder_name=N''' + @folder_name2 + '''' ----
Print ', @project_name=N''' + @project_name2 + '''' ---
Print ', @value_type=' + @value_type
Print ', @parameter_value=N''' + @parameter_value + ''''
Print 'GO'
Print ''

FETCH NEXT FROM parameter_var_cursor
INTO @object_type, @parameter_name, @object_name, @folder_name2, @project_name2, @value_type, @parameter_value;
END
CLOSE parameter_var_cursor;
DEALLOCATE parameter_var_cursor;
-- End Parameter - Variable mapping

FETCH NEXT FROM ref_environment_cursor
INTO @environment_name, @project_name, @folder_name, @environment_folder_name, @reference_type, @folder_id, @environment_description, @environment_id;
END
CLOSE ref_environment_cursor;
DEALLOCATE ref_environment_cursor;
GO
Download as SQL file

NOTE: Please use at own risk and let me know it things could be improved!

Connecting to Excel (XLSX) in SSIS

$
0
0
Case
I cannot access my Excel sheets in SSIS. The Excel version in the Connection Manager is Microsoft Excel 2007 (xlsx).
Could not retrieve the table information for the
connection manager 'Excel Connection Manager'.
Failed to connect to the source using the
connection manager 'Excel Connection Manager'




























Solution
XLSX files don't use the out-of-the-box Microsoft.Jet.OLEDB provider, but they need the Microsoft.ACE.OLEDB provider. You either did not install it or you installed the 64bit version.

Download and install the 32bit version of the Microsoft Access Database Engine 2010 Redistributable. Because Visual Studio (SSDT/BIDS) is 32bit you can't use the 64bit provider for developing SSIS packages. If you already installed the 64bit version then you first need to remove it. You can't install 32bit and 64bit parts of office on the same machine. You will get an error when you run the installer (and you will get the same error if you have a 64bit version of Microsoft Office installed on your development machine):
You cannot install the 32-bit version of Microsoft
Access Database Engine 2010 because you currently
have 64-bit Office products installed. If you want to
install 32-bit Microsoft Access Database Engine 2010,
your first need to remove the 64-bit installation of
office products.

















However this means that you can't run packages with Excel Connection Mangers in 64bit on your development machine. You need to switch to 32bit, otherwise you will get an error like:
Information: 0x4004300A at DFT - xlsx source, SSIS.Pipeline: Validation phase is beginning.
Error: 0xC0209303 at xlsxSource, Connection manager "Excel Connection Manager": The requested OLE DB provider Microsoft.ACE.OLEDB.12.0 is not registered.
If the 64-bit driver is not installed, run the package in 32-bit mode. Error code: 0x00000000.
An OLE DB record is available. Source: "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered".
Error: 0xC001002B at xlsxSource, Connection manager "Excel Connection Manager": The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine. For more information, see http://go.microsoft.com/fwlink/?LinkId=219816
Error: 0xC020801C at DFT - xlsx source, EX_SRC - My XLSX Source [8]: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0209303. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
Error: 0xC0047017 at DFT - xlsx source, SSIS.Pipeline: EX_SRC - My XLSX Source failed validation and returned error code 0xC020801C.
Error: 0xC004700C at DFT - xlsx source, SSIS.Pipeline: One or more component failed validation.
Error: 0xC0024107 at DFT - xlsx source: There were errors during task validation.

But since there is a 64bit driver... you could install it on your test/acceptance/production server and run the packages in 64bit (as long as you don't use Visual Studio on those servers).

SSIS 2012 with Team Foundation Server - Part II

$
0
0
Case
I have installed Team Explorer and setup Visual Studio to use it. What's next?

Solution
In Part I you read:
A) Install Team Explorer for Visual Studio 2010
B) Install Team Explorer for Visual Studio 2012
C) Setup Visual Studio to use TFS

This second part covers:
D) Adjustingdevelopment process



D) Adjusting development process
Because you can now work with multiple developers on the same project, you have to make some arrangements with your fellow developers, like:

1) Get latest version project
Get the latest version of the project on a regular basis. Otherwise you will miss new packages, project connection managers and project parameters. Do this for example each morning or before you start developing. There is also an option in Visual Studio to automatically get the latest version of the solution when opening it.
Get everything when a solution or project is opened.

















2) Get latest version package
Get the latest version of a package before editing it. There is also an option in Visual Studio to automatically get the latest version of a package when checking it out.
Get latest version of item on check out.

















3) Adding new package to project
When you add a new package to the project, the project self will be checked out. First first rename the new package, save it and then check in the project and the new (empty/clean) package. Otherwise your fellow developers cannot change project properties or add new packages.
Adding new package will check out the project























4) Disable multiple check out
Working together on the same file at the same time is nearly impossible, because it's hard to merge the XML of two versions of a package. Therefore you should disable multiple check out in TFS or check out your package exclusively (not the default in TFS).
In Team-menu click Team Project Settings, Source Control

Uncheck the multiple checkout box






































5) Don't check in faulty packages
Try not to check in package that doesn't work. Especially when you work with the project deployment model, with which you can only deploy the complete project.
Don't check in faulty packages



















6) No large/complex packages
Don’t make packages to large/complex. Divide the functionality over multiple smaller packages, because you can’t work with multiple developers on the same large package at the same time.

7) Sensitive data
The default Package Protection Level is EncryptSensitiveWithUserKey. This will encrypt passwords and other sensitive data in the package with the username of the developer. Because your colleagues will probably have different usernames they can't edit or execute packages that you made without re-entering all sensitive package data.
The easiest way to overcome this, is to use DontSaveSensitive as Package Protection Level in combination with Package Configurations. Then all the sensitive data will be stored in the configuration table or file and when you open the package all this data will be retrieved from the configuration table or file.
If you're using the Project Deployment Model in combination with sensitive parameters instead of Package Configuration, then the easiest workaround is to use EncryptAllWithPassword or EncryptSensitiveWithPassword with a password that is known within the developmentteam.

8) Development standards
When you're developing with multiple people (or someone else is going to maintain your work) then it's good to have some Development Best Practices like using prefixes for tasks and transformations or using templates. This makes it easier to transfer work and to collaborate as a team.

9) Comments
When you check in a package, it's very useful to add a meaningful description of the change. This makes it easier to track history.
Check in comments

















10) Branching, Labeling and building
Beside versioning and checking in/out packages there are more interesting functions in TFS that are probably more common in C# and VB.Net programming, but worth checking out. Here are some interesting links about TFS and SSIS:

 

 



SSIS 2012 with Team Foundation Server - Part I

$
0
0
Case
Team Foundation Server (TFS) is a handy tool when you work with multiple people on the same visual studio project. You can check out the files you work on and you still have the previous versions if you mess up. How do you get TFS working for SSIS 2012?

Solution
SSIS 2012 uses SQL Server Data Tools for Visual Studio 2010 for development, but you can also use Visual Studio 2012. They are called Visual Studio 2010/2012 Shell. Both have a different version of Team Explorer:

Visual Studio 2010: Microsoft Visual Studio Team Explorer 2010
Visual Studio 2012: Team Explorer for Microsoft Visual Studio 2012

This blog post shows how to install Team Explorer and shows how to setup Visual Studio to use TFS for SSIS. I have tested these for TFS 2010 and 2012. The SQL Server version that I used was Developer Edition 11.0.2100.60 RTM and VS2010 Shell 10.0.40219 SP1 and VS2012 Shell 11.0.50727.1 RTM. For SSIS 2008 (R2) and VS2008 Shell see this earlier blogpost.

In Part I covers:
A) Install Team Explorer for Visual Studio 2010
B) Install Team Explorer for Visual Studio 2012
C) Setup Visual Studio to use TFS

In Part II you will read:
D) Adjusting developmentprocess




A) Install Team Explorer for Visual Studio 2010
Use the 2010 download link above. It's an ISO file. Mount or extract the iso file and execute the setup file. Next follow the setup screens. There are no real options to customize the installation: Next, Accept, Next, Install, Finish.
























Reinstall Service Pack 1
If you have Service Pack 1 installed for Visual Studio 2010, then you have to reinstall SP1 when you finish installing Team Explorer. Else you could get an error like this when starting SSDT or SSMS:
Only some of the Microsoft Visual Studio 2010 products
on this computer have been upgraded to Service Pack 1.
None will work correctly until all have been upgraded.
















You can download 'Microsoft Visual Studio 2010 Service Pack 1 (Installer)' here: http://www.microsoft.com/en-us/download/details.aspx?id=23691


B) Install Team Explorer for Visual Studio 2012
Use the 2012 download link above. It's an ISO file or an exe. Execute the setup file. Next follow the setup screens. There are no options to customize the installation: Accept, Install and launch VS2012.






























C) Setup Visual Studio to use TFS
For both versions of Visual Studio setting up TFS is the same.
In the Team-menu choose Connect to TFS





















If you have updated Visual Studio 2012, then this first step will open the Team Explorer pane on the right side. There you can click on the Connect link to open the window in the next screenshot.

Click on Servers-button to add a TFS server

Click on Add-button to add a TFS server

Add the tfs URL and choose between http and https

Click the OK-button and wait

Enter your credentials

Click the Close-button

Select the TFS project


Now you have the Team Explorer pane available



















































































































































In the second part you read what's next.

Package Configurations with BIML

$
0
0
Case
I want to use SSIS Package configurations in my BIML script. How do I do that?

Solution
Here are a couple of examples of the most used package configurations. Screens are from SSIS 2012 package deployment, but it works the same in SSIS 2008.

Environment Variable Config
I have one Connection Manager named Meta and I added package configuration to get its connectionstring from a Windows Environment Variable. That variable already exists and contains a connectionstring. The screens are what the BIML script below will produce.
Environment Variable Config


















<Biml xmlns="http://schemas.varigence.com/biml.xsd">

<Connections>
<!-- My Connection Manager to the Meta database containing a config table and other tables-->
<OleDbConnection
Name="Meta"
ConnectionString="Data Source=.;Initial Catalog=Meta;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;">
</OleDbConnection>
</Connections>

<Packages>
<Package Name="Child01" ConstraintMode="Linear">

<PackageConfigurations>
<!-- Environment Variable Configuration -->
<!-- The Environment Variable should already contain a value -->

<!-- The name of the configuration shown in the Package Configurations Organizer window -->
<PackageConfiguration Name="SSISMeta">
<!-- The name of the environment variable -->
<EnvironmentVariableInput EnvironmentVariable="SSISMeta" />
<ConfigurationValues>
<!-- PropertyPath contains the name of the connection manager -->
<!-- You can leave the value property empty -->
<ConfigurationValue
DataType="String"
Name="ConnectrionStringMeta"
PropertyPath="\Package.Connections[Meta].Properties[ConnectionString]"
Value="" />
</ConfigurationValues>
</PackageConfiguration>

</PackageConfigurations>

<Tasks>
<!-- Dummy Task with connection to make sure the connection manager is added to the package -->
<ExecuteSQL
Name="SQL - Dummy"
ConnectionName="Meta"
ResultSet="None">
<DirectInput>
SELECT @@VERSION AS 'SQL Server Version'
</DirectInput>
</ExecuteSQL>

</Tasks>
</Package>
</Packages>
</Biml>
Download

SQL Server Configuration
I have a second Connection Manager named Source and I added package configuration to get its value from a SQL Server configuration table. This configuration table is stored in the Meta database. Note: the configurations should already exist in that table
SQL Server Config


















<Biml xmlns="http://schemas.varigence.com/biml.xsd">

<Connections>
<!-- My Connection Manager to the Meta database containing a config table and other tables-->
<OleDbConnection
Name="Meta"
ConnectionString="Data Source=.;Initial Catalog=Meta;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;">
</OleDbConnection>
<!-- My Connection Manager to a source database -->
<OleDbConnection
Name="Source"
ConnectionString="Data Source=.;Initial Catalog=AdventureWorks2012;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;">
</OleDbConnection>
</Connections>

<Packages>
<Package Name="Child01" ConstraintMode="Linear">

<PackageConfigurations>
<!-- Environment Variable Configuration -->
<!-- The Environment Variable should already contain a value -->

<!-- The name of the configuration shown in the Package Configurations Organizer window -->
<PackageConfiguration Name="SSISMeta">
<!-- The name of the environment variable -->
<EnvironmentVariableInput EnvironmentVariable="SSISMeta" />
<ConfigurationValues>
<!-- PropertyPath contains the name of the connection manager -->
<!-- You can leave the value property empty -->
<ConfigurationValue
DataType="String"
Name="ConnectrionStringMeta"
PropertyPath="\Package.Connections[Meta].Properties[ConnectionString]"
Value="" />
</ConfigurationValues>
</PackageConfiguration>

<!-- SQL Server Configuration -->
<!-- The configuration table should already contain values -->

<!-- ConnectionName is the name of the connection manager containing the configuration table -->
<!-- Name is for both the Configuration Filter in the database table and the name in the Package Configurations Organizer window -->
<PackageConfiguration
ConnectionName="Meta"
Name="SourceConfiguration">
<!-- Table contains the name of the configuration table -->
<ExternalTableInput Table="[dbo].[SSIS Configurations]" />
</PackageConfiguration>

</PackageConfigurations>

<Tasks>
<!-- Dummy Tasks with connection to make sure the connection manager is added to the package -->
<ExecuteSQL
Name="SQL - Dummy 1"
ConnectionName="Meta"
ResultSet="None">
<DirectInput>
SELECT @@VERSION AS 'SQL Server Version'
</DirectInput>
</ExecuteSQL>
<ExecuteSQL
Name="SQL - Dummy 2"
ConnectionName="Source"
ResultSet="None">
<DirectInput>
SELECT @@VERSION AS 'SQL Server Version'
</DirectInput>
</ExecuteSQL>
</Tasks>
</Package>
</Packages>
</Biml>
Download
The combination of these two configuration types is often used in a DTAP street.

Parent Package Variable Configuration
I have a variable that is filled by a variable from the parent package. This is done with Parent Package Variable Configuration. In BIML script you will find this in the variable tag and not in the configurations tag!
Parent Package Variable Config



















<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="Child01" ConstraintMode="Linear">

<!-- Parent Package Variable Configuration -->
<!-- Note: this is not in the Configurations tag, but in the variable tag -->

<!-- InheritFromPackageParentConfigurationString is for both the name of the parent package variable -->
<!-- and the name in the Package Configurations Organizer window-->
<Variables>
<Variable
DataType="String"
Name="MyChildPackageVariable"
InheritFromPackageParentConfigurationString="MyParentPackageVariable"
Namespace="User">SSISJoost</Variable>
</Variables>

</Package>
</Packages>
</Biml>
Download

XML Configuration File
I have a Connection Manager and I have an XML configuration file to configure its connectionstring. The xml/dtsConfig file already exists with the correct values otherwise the package won't work.
XML Configuration File


















<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<OleDbConnection
Name="Destination"
ConnectionString="Data Source=.;Initial Catalog=Staging;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;">
</OleDbConnection>
</Connections>
<Packages>
<Package Name="Child01" ConstraintMode="Linear">

<PackageConfigurations>
<!-- XML Configuration File -->

<!-- The name is for the name in the Package Configurations Organizer window-->
<PackageConfiguration Name="Destination Configuration">
<!-- ExternalFilePath is the path of the config file -->
<ExternalFileInput
ExternalFilePath="D:\DestinationConfigurations.dtsConfig">
</ExternalFileInput>
<ConfigurationValues>
<!-- You can leave the value property empty -->
<!-- The value of the PropertyPath should also be in the DtsConfig file -->
<ConfigurationValue
DataType="String"
Name="ConnectionStringDestination"
PropertyPath="\Package.Connections[Destination].Properties[ConnectionString]"
Value=""
>
</ConfigurationValue>
</ConfigurationValues>
</PackageConfiguration>

</PackageConfigurations>

<Tasks>
<!-- Dummy Tasks with connection to make sure the connection manager is added to the package -->
<ExecuteSQL
Name="SQL - Dummy"
ConnectionName="Destination"
ResultSet="None">
<DirectInput>
SELECT @@VERSION AS 'SQL Server Version'
</DirectInput>
</ExecuteSQL>
</Tasks>
</Package>
</Packages>
</Biml>
Download
This XML configuration type can also be used in a DTAP street.

Insert, update and delete records in CRM 2013 with SSIS - Part III: Add files as Annotation

$
0
0
Case
I want to upload files to CRM Annotations with SSIS. I already uploaded contacts, but now I also want to upload their documents. How do I do that?

Solution
You can use the same webservice in a Script Component that you used to insert/update records in CRM. For this example I have a source with a businessKey from my contact, a filename and a filepath. If your source doesn't contain the actual file in a blob, but only a filepath then you can use the Import Column Transformation.
I used a Lookup Transformation to get the identity id from a CRM database view because I need it to reference the document to my contacts.
Data Flow Example with Import Columns























1) Download CRM SDK
For this example I used CRM 2013 and I downloaded the free Microsoft Dynamics CRM 2013 Software Development Kit (SDK). Execute the downloaded file to extract all the files. We only need Microsoft.Xrm.Sdk.dll assembly which can be found in the SDK\Bin folder.
SDK download















2) DLL to SSIS machine
To use the assembly (DLL) from step 1 in SSIS, you need to add the DLL to the Global Assembly Cache (GAC) on your SSIS machine. Here is an example for adding to the gac on Win Server 2008 R2. You also need to copy it to the Binn folder of SSIS: D:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\

3) Parameters
To avoid hardcoded usernames, domainnames, passwords and webservices, I created four project parameters (SSIS 2012). These parameters will be used in the Script Components.
4 parameters, password is marked sensitive.







4) Script Component - Parameters
Add a Script Component (type destination) for adding the files. When you edit the Script Component make sure to add the four parameters from the previous step as read only variables.

Add parameters as read only variables


























5) Script Component - Input Columns
Add the columns that you need as Input Columns. For inserting the files as annotations you need the blob column containing the actual file, a filename, an entity ID to reference the file to a CRM contact and optionally a subject and/or description.
Input Columns for insert



























6) The Script - Add assembly
Hit the Edit Script button to start the VSTA editor. In the solution explorer you need to add three references:
  • microsoft.xrm.sdk.dll (from the SSIS bin folder mentioned in step 3, use browse)
  • System.Runtime.Serialization.dll (from .Net tab)
  • System.ServiceModel.dll (from .Net tab)
Right click references and choose add reference

















Now very important: press the Save All button to save the entire internal vsta project (including references)
Save All













7) The Script - Insert
Here is an C# example (for VB.Net use this translator) for inserting files as annotations in CRM with SSIS 2012.
// C# Code
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.Xrm.Sdk; // Added
using Microsoft.Xrm.Sdk.Client; // Added
using Microsoft.Xrm.Sdk.Query; // Added
using System.ServiceModel.Description; // Added

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
// Webservice
IOrganizationService organizationservice;

// Variables for the CRM webservice credentials
// You could also declare them in the PreExecute
// if you don't use it anywhere else
string CrmUrl = "";
string CrmDomainName = "";
string CrmUserName = "";
string CrmPassWord = "";

// This method is called once, before rows begin to be processed in the data flow.
public override void PreExecute()
{
base.PreExecute();

// Fill variables with values from project parameters
CrmUrl = this.Variables.CrmWebservice.ToString();
CrmDomainName = this.Variables.CrmDomain.ToString();
CrmUserName = this.Variables.CrmUser.ToString();
CrmPassWord = this.Variables.CrmPassword.ToString();

ClientCredentials credentials = new ClientCredentials();
credentials.UserName.UserName = string.Format("{0}\\{1}", CrmDomainName, CrmUserName);
credentials.UserName.Password = CrmPassWord;
organizationservice = new OrganizationServiceProxy(new Uri(CrmUrl), null, credentials, null);
}

// This method is called once for every row that passes through the component from Input0.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
// Create new annotation object to store the properties
Entity newAnnotation = new Entity("annotation");

// Create an optionset to indicate to which entity this annotation will be linked
OptionSetValue objectidtypecode = new OptionSetValue();
objectidtypecode.Value = 2; // 2 is the enitity Contact in this case. Look it up in CRM.
newAnnotation.Attributes.Add("objectidtypecode", (OptionSetValue)objectidtypecode);

// Create an entity reference to contact and store it in the column ObjectId
EntityReference Contact = new EntityReference("contact", Row.ContactId);
newAnnotation["objectid"] = Contact;

// Filename of the attachment (without path)
newAnnotation["filename"] = Row.Filename;

// The actual file is retrieved from the blob column as byte[]
// And that byte[] is converted to a Base64String:
newAnnotation["documentbody"] = Convert.ToBase64String(Row.Document.GetBlobData(0, Convert.ToInt32(Row.Document.Length)));

// A subject with some title
newAnnotation["subject"] = Row.Subject;

// Add the annotation
organizationservice.Create(newAnnotation);
}
}
Note: this code example doesn't contain logging or exception handling!

Insert, update and delete records in CRM 2013 with SSIS - Part II: Inactivate records

$
0
0
Case
A while a go I did a post on how to insert, update or delete records in CRM (2013), but what if you want to (de)activate records instead of deleting them to preserve some history in CRM?

Solution
This is a continuation of a previous blog post. Please read it to get more inside details.

1) Start
To deactivate or activate records in CRM, you need the identity id (GUID) of that record. In SSIS you could full outer join your source with a CRM identity view and if the records exists in CRM, but not in your source then you can deactivate the record.


2) Download CRM SDK
For this example I used CRM 2013 and I downloaded the free Microsoft Dynamics CRM 2013 Software Development Kit (SDK). Execute the downloaded file to extract all the files. We only need the Microsoft.Xrm.Sdk.dll and Microsoft.Crm.Sdk.Proxy.dll assemblies which can be found in the SDK\Bin folder.
SDK download















3) DLL to SSIS machine
To use the assemblies (DLL) from step 2 in SSIS, you need to add the DLL files to the Global Assembly Cache (GAC) on your SSIS machine. Here is an example for adding to the GAC on Win Server 2008 R2. You also need to copy them to the Binn folder of SSIS: D:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\

4) Parameters
To avoid hardcoded usernames, domainnames, passwords and webservices, I created four project parameters (SSIS 2012). These parameters will be used in the Script Components.
4 parameters, password is marked sensitive.







5) Script Component - Parameters
Add a Script Component (type destination) for the deactivate. When you edit the Script Component make sure to add the four parameters from the previous step as read only variables.

Add parameters as read only variables


























6) Script Component - Input Columns
Add the columns that you need as Input Columns. For the deactivation you only need the entity ID from CRM. This is the Technical Id (a guid) from the CRM entity that you want to update.

Input Columns for deativate























7) The Script - Add assembly
Hit the Edit Script button to start the VSTA editor. In the solution explorer you need to add four references (microsoft.crm.sdk.proxy.dll is for inactivation):
  • microsoft.xrm.sdk.dll (from the SSIS bin folder mentioned in step 3, use browse)
  • microsoft.crm.sdk.proxy.dll (from the SSIS bin folder mentioned in step 3, use browse)
  • System.Runtime.Serialization.dll (from .Net tab)
  • System.ServiceModel.dll (from .Net tab)
Right click references and choose add reference












    Now very important: press the Save All button to save the entire internal vsta project (including references)
    Save All













    8a) The Script - Deactivate
    Here is an C# example (for VB.Net use this translator) for deactivating an existing CRM account with SSIS 2012.
    // C# Code
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
    using Microsoft.SqlServer.Dts.Runtime.Wrapper;
    using Microsoft.Xrm.Sdk; // Added
    using Microsoft.Xrm.Sdk.Client; // Added
    using Microsoft.Crm.Sdk.Messages; // Added
    using System.ServiceModel.Description; // Added

    [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
    public class ScriptMain : UserComponent
    {
    // Webservice
    IOrganizationService organizationservice;

    // Variables for the CRM webservice credentials
    // You could also declare them in the PreExecute
    // if you don't use it anywhere else
    string CrmUrl = "";
    string CrmDomainName = "";
    string CrmUserName = "";
    string CrmPassWord = "";

    // This method is called once, before rows begin to be processed in the data flow.
    public override void PreExecute()
    {
    base.PreExecute();
    // Fill variables with values from project parameters
    CrmUrl = this.Variables.CrmWebservice.ToString();
    CrmDomainName = this.Variables.CrmDomain.ToString();
    CrmUserName = this.Variables.CrmUser.ToString();
    CrmPassWord = this.Variables.CrmPassword.ToString();

    // Connect to webservice with credentials
    ClientCredentials credentials = new ClientCredentials();
    credentials.UserName.UserName = string.Format("{0}\\{1}", CrmDomainName, CrmUserName);
    credentials.UserName.Password = CrmPassWord;
    organizationservice = new OrganizationServiceProxy(new Uri(CrmUrl), null, credentials, null);
    }

    // This method is called once for every row that passes through the component from Input0.
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
    // Create CRM request to (de)activate record
    SetStateRequest setStateRequest = new SetStateRequest();

    // Which entity/record should be (de)activate?
    // First part in the entityname, second
    // is the entity id from the CRM source.
    setStateRequest.EntityMoniker = new EntityReference("account", Row.myGuid);

    // Setting 'State' (0 – Active ; 1 – InActive)
    setStateRequest.State = new OptionSetValue(1);

    // Setting 'Status' (1 – Active ; 2 – InActive)
    setStateRequest.Status = new OptionSetValue(2);

    // Execute the request
    SetStateResponse response = (SetStateResponse)organizationservice.Execute(setStateRequest);
    }
    }


    8b) The Script - Adding an inactive account
    And you can also combine adding and deactivating. Here is an C# example (for VB.Net use this translator) for inserting an inactive account with SSIS 2012.
    // C# Code
    // This method is called once for every row that passes through the component from Input0.
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
    // Create a Entity object of type 'account'
    Entity newAccount = new Entity("account");

    // fill crm fields. Note fieldnames are case sensitive!
    newAccount["name"] = Row.AccountName;
    newAccount["emailaddress1"] = Row.Email;
    newAccount["telephone1"] = Row.Phone;

    // Create account and store its Entity ID to deactivate the account
    Guid AccountGuid = organizationservice.Create(newAccount);

    // Create CRM request to (de)activate record
    SetStateRequest setStateRequest = new SetStateRequest();

    // Which entity/record should be (de)activate?
    // First part in the entityname, second
    // is the entity id you got from adding
    // the new account.
    setStateRequest.EntityMoniker = new EntityReference("account", AccountGuid);

    // Setting 'State' (0 – Active ; 1 – InActive)
    setStateRequest.State = new OptionSetValue(1);

    // Setting 'Status' (1 – Active ; 2 – InActive)
    setStateRequest.Status = new OptionSetValue(2);

    // Execute the request
    SetStateResponse response = (SetStateResponse)organizationservice.Execute(setStateRequest);
    }
    Note: these code examples don't contain logging or exception handling!

    BIML doesn't recognize system variable ServerExecutionID

    $
    0
    0
    Case
    I want to use the SSIS System Variable ServerExecutionID as a parameter for an Execute SQL Task in a BIML Script, but it doesn't recognize it and gives an error:
    
    Could not resolve reference to 'System.ServerExecutionID' of type 'VariableBase'. 'VariableName="System.ServerExecutionID"' is invalid.
























    Solution
    The current version of BIDS/BIML doesn't recognize all system variables (for example
    LocaleId and ServerExecutionID). Other system variables like VersionMajor or VersionBuild will work. You can overcome this by manually adding these variables in your BIML Script.
    
    
    <VariableName="ServerExecutionID"DataType="Int64"Namespace="System">0</Variable>


























    And if you now run the package (in the catalog) the table gets filled with the System variable ServerExecutionID:

    Number added, it works!
















    Continue Loop after error - Part I

    $
    0
    0
    Case
    I want to continue my loop when in one of the iterations a task fails. It should continue with the next iteration/file. I tried changing the task properties FailParentOnFailure and MaximumErrorCount, but nothing seems to work.
    Task fails, so loop fails

















    Solution
    There are a couple of solutions. You could set the MaximumErrorCount to a higher number on the parent container (not on the task that fails). With FailParentOnFailure property on the failing task you can override that setting and fail the parent on the first error.

    The easiest/stable solution is to use an empty OnError event handler on the failing task with the system variable Propagate set to false.

    1) Empty event handler
    Go to the event handlers tab. Select (1) the failing task as Executable and (2) OnError as the event handler. Next (3) click on the link in the middle of the page to create the event handler: Click here to create an 'OnError' event handler for executable 'DFT - Stage files'.
    Create empty event handler
















    2) System variable propagate
    Open the variable pane in the event handler. Click on (1) the Variable Grid Options button. Check (2) the radio button "Show system variables". Search for the Propagate variable and set (3) it to false.
    System variable Propagate

























    Show system variables for SSIS 2008








    3) Annotation
    An empty event handler could confuse other developers. Adding a simple annotation could solve that.
    Add an annotation























    4) The result
    Now an error in the Data Flow Task won't fail the Foreach Loop Container. This solution works with all containers (Sequence, For Loop, Foreach Loop and package).
























    Note: this solution wont work with parent child packages. Propagate can't be disabled from a child package to a parent package. This is by design according Microsoft. Here is a workaround for that.

    Continue Loop after error - Part II

    $
    0
    0
    Case
    I used the propagate variable trick to continue a loop on error. That works within the package, but the parent package calling the package with the loop still fails.
    Child succeeds, parent fails

















    Solution
    This is by design according Microsoft. The workaround they suggest is setting the DisableEventHandlers property of the Execute Package task to True. This should ignore all errors in the child package. Below here is an alternative solution to only ignore expected errors. Don't hesitate to post your own solution in the comments.

    A parent package variable will be filled by the child package in case of an unexpected error. In the parent package it will be used to throw an error when it's filled.

    1) Variable - Parent Package
    Add a string variable in the parent package called childError.
    Add string variable


























    2) OnError - Child Package
    Go to the child package where you used the propagate variable 'trick' and add an OnError event handler on package level. Go to Event Handlers tab. Make sure the package is selected as Executable and OnError as Event handler. Then click on the link in the middle of the page: Click here to create an 'OnError' event handler for the executable 'Child1'.
    OnError event handler for package


















    3a) Script Task - Child Package
    Add a Script Task to the event handler and give it a suitable name. Then edit it and add the System variables ErrorCode and ErrorDescription as read only variables and type the name of the parent package in the read write box. You can't select it because it's only known at runtime.
    Add the variables




















    3b) The Script
    Hit the edit button and copy the code of my main method to your main method. The example code is in C#. For a VB.Net version you can use this conversion tool.
    // C# Code
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Runtime;
    using System.Windows.Forms;

    namespace ST_ace2311e5a4c4bbb98101cd54888c7c9
    {
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
    public void Main()
    {
    // Fill parent package variables with error message from this child package.
    Dts.Variables["User::ChildError"].Value = Dts.Variables["System::ErrorCode"].Value.ToString() + " - " + Dts.Variables["System::ErrorDescription"].Value.ToString();
    Dts.TaskResult = (int)ScriptResults.Success;
    }

    #region ScriptResults declaration
    ///
    /// This enum provides a convenient shorthand within the scope of this class for setting the
    /// result of the script.
    ///
    /// This code was generated automatically.
    ///

    enum ScriptResults
    {
    Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
    Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    };
    #endregion

    }
    }

    4) Propagate - Parent Package
    Go back to the parent package and add an empty event handler on the Execute Package Task so that it won't fail if the child package has an error. Go to the event handlers. Select the Execute Package Task as executable and select OnError as eventhandler and then create the event handler by clicking on the link in the middle of the screen: "Click here to create an 'OnError' event handler for executable 'EPT - Child1'.

    Next go to the variables pane and hit the Variable Grid Option button to also show system variables (SSIS 2008 has a different button). Then find the system variable Propagate and set it to false.

    Last step is to add an annotation in the empty event handler to explain why it's empty.
    OnError Event Handler
















    
    Show system variables for SSIS 2008








    5a) Script Task - Parent Package
    Go back to the Control Flow of your parent package and add a Script Task to fire an error. Connect it to the Execute Package Task with an expression only on the precedence constraint:
    @[User::ChildError] != ""
    Fire Child Error























    5b) The Script
    Edit the Script Task and add the string variable ChildError as read-only variable. After that hit the edit button and copy the content of my main method to your main method. The example code is in C#. For a VB.Net version you can use this conversion tool.
    Read-only variable























    // C# Code
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Runtime;
    using System.Windows.Forms;

    namespace ST_7b522ca79c9f4428a233a100bfc66e6e
    {
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
    public void Main()
    {
    // Fire error with error message from child package and fail script task
    Dts.Events.FireError(0, "Child package", "Child package error: " + Dts.Variables["User::ChildError"].Value.ToString(), string.Empty, 0);
    Dts.TaskResult = (int)ScriptResults.Failure;
    }

    #region ScriptResults declaration
    ///
    /// This enum provides a convenient shorthand within the scope of this class for setting the
    /// result of the script.
    ///
    /// This code was generated automatically.
    ///

    enum ScriptResults
    {
    Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
    Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    };
    #endregion

    }
    }


    6) Precedence Contraint - Parent Package
    If there is no unexpected error in the child package then we continue with the next task(s). Each task that is connected to the Execute Package Task with a Success Precedence Constraint should be changed to an expression: @[User::ChildError] == ""
    Continue if there are no unexpected errors in the child package


























    7) The result
    I added a Script Task in the child package that always fails to simulate an unexpected error. On the second run there where only expected errors in the loop, so the child package was successful. In the parent package there is still a red cross, but it continues without (unexpected) errors.
    The result

    Create and fill Age dimension

    $
    0
    0
    Case
    Is there an easy way to create and populate an age dimension with age groups?

    Solution
    Creating an age dimension is usually done once and probably not in SSIS, but with a TSQL script.
    For each new assignment I use a script similar to this and adjust it to the requirements for that particular assignment.

    -- Drop dimension table if exists
    IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[dim_age]') AND TYPE IN (N'U'))
    BEGIN
    DROP TABLE [dbo].[dim_age]
    END

    -- Create table dim_age
    SET ANSI_NULLS ON
    GO

    SET QUOTED_IDENTIFIER ON
    GO

    CREATE TABLE [dbo].[dim_age](
    [dim_age_id] [int] IDENTITY(-1,1) NOT NULL,
    [Age] [smallint] NULL,
    [AgeGroup1] [nvarchar](50) NULL,
    [AgeGroup2] [nvarchar](50) NULL,
    CONSTRAINT [PK_dim_age] PRIMARY KEY CLUSTERED
    (
    [dim_age_id] ASC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]

    GO

    -- Enter unknown dimension value (in case a person's date of birth is unknown)
    INSERT INTO [dbo].[dim_age]
    ([Age]
    ,[AgeGroup1]
    ,[AgeGroup2])
    VALUES
    (-1
    ,'Unknown'
    ,'Unknown')
    GO

    -- Enter all ages
    declare @age smallint;
    set @age = 0;

    -- Loop through ages 0 to 130
    WHILE @age < 131
    BEGIN
    INSERT INTO [dbo].[dim_age]
    ([Age]
    ,[AgeGroup1]
    ,[AgeGroup2])
    VALUES
    (@age
    -- Use the common age groups/categories of your region/branch/industry
    -- This is just an example
    ,CASE
    WHEN @age < 15 THEN '0 till 15 year'
    WHEN @age < 25 THEN '15 till 25 year'
    WHEN @age < 35 THEN '25 till 35 year'
    WHEN @age < 45 THEN '35 till 45 year'
    WHEN @age < 55 THEN '45 till 55 year'
    WHEN @age < 65 THEN '55 till 65 year'
    ELSE '65 year and older'
    END
    ,CASE
    WHEN @age < 19 THEN 'Juvenile'
    ELSE 'Mature'
    END
    )

    -- Goto next age
    set @age = @age + 1
    END

    The result: filled age dimension






















    How could you use this dimension?
    A while ago I also posted an example to create and populate a date dimension. So now you can combine those in a datamart. I have an employee table and an absence table with a start- and enddate.
    Employee table

    Absence table



















    I will use the date dimension to split the absence time periods in separate days and then calculate the employee's age of each day of absence. This will go in to a fact table and then I can use the age dimension to see absence per age group.
    -- Split absence time periode in separate days, but go back 2 years max and 1 year forward if end date is unknown
    SELECT Absence.AbsenceId
    , Absence.EmployeeNumber
    -- Date of absence
    , dim_date.Date as AbsenceDate
    , Absence.ReasonCode
    -- Calculation of age at time of absence
    , DATEDIFF(YEAR, Employee.DateOfBirth, dim_date.Date)
    -
    (CASE
    WHEN DATEADD(YY, DATEDIFF(YEAR, Employee.DateOfBirth, dim_date.Date), Employee.DateOfBirth)
    > dim_date.Date THEN 1
    ELSE 0
    END) as Age
    FROM EmployeeApplication.dbo.Absence
    INNER JOIN EmployeeApplication.dbo.Employee
    on Absence.EmployeeNumber = Employee.EmployeeNumber
    INNER JOIN DM_Staff.dbo.dim_date
    on dim_date.Date
    -- change start date to lower bound if it's below it
    BETWEEN CASE WHEN YEAR(Absence.AbsenceStartDate) >= YEAR(GETDATE()) - 2 THEN Absence.AbsenceStartDate
    ELSE DATEADD(yy, DATEDIFF(yy, 0, getdate()) - 2, 0) END
    -- change end date to upper bound if it's null
    AND ISNULL(Absence.AbsenceEndDate, DATEADD(yy, DATEDIFF(yy, 0, getdate()) + 2, -1))
    -- Filter absence record with an enddate below the lower bound (perhaps a bit superfluous with the inner join)
    WHERE YEAR(ISNULL(Absence.AbsenceEndDate, GETDATE())) >= YEAR(GETDATE()) - 2


    Result of query that can be used in a fact package





















    fact absence


























    Note: this is a simplified situation to keep things easy to explain.

    Retry a Task on failure

    $
    0
    0
    Case
    I have a webservice task that sometimes fails (due an external cause) and I want to retry it a couple of times before failing the entire package. How do I create a retry construction for a task?

    Solution
    A solution could be to add a FOR LOOP around the task you want to retry. This works for all tasks, not just the webservice task.
    A For Loop Container for retrying a task















    1) Variables
    We need a couple of variables for the For Loop Container.
    - RetryMax: an integer indicating the maximum number of attempts
    - RetryCounter: an integer to keep track of the number of attempts
    - QuitForLoop: a boolean for quiting loop before reaching the maximum number of attempts
    - RetryPause: an integer for storing the number of pause seconds before retry
    
    Variables for the FOR LOOP























    2) For Loop
    Add a For Loop Container and move the task you want to retry inside the container. Edit the For Loop Container and set the following properties:

    InitExpression: @[User::RetryCounter] = 1
    This will give the RetryCounter an initial value.

    EvalExpression: @[User::RetryCounter] <= @[User::RetryMax] &&
                                 @[User::QuitForLoop] == false
    This will indicate when the For Loop Container stops looping. In this case reaching the MaxRetry or when the Boolean variable is filled with True.

    AssignExpression: @[User::RetryCounter] = @[User::RetryCounter] + 1
    This will increase the RetryCounter.
    The FOR LOOP expressions
















    3a) Pause Task
    For this example I will use a Script Task for waiting a couple of seconds/minutes before a retry . If you don't like scripting there are alternatives for a pause. Add a Script Task, give it a useful name and connect it to the task you want to retry. Make sure the Script Task only executes on error by setting the Constraint Option to Failure.
    Pause after failure
















    3b) The Script
    Edit the script Task and add the 3 integer variables as read-only variables. Then hit the edit button and copy the contents of my main method to your main method. The example code is in C#. For a VB.Net version you can use this conversion tool.
    Read-only variables

















    // C# Code
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Runtime;
    using System.Windows.Forms;

    namespace ST_1c11fe9f84ce4662bdc37ece5316e04d
    {
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {

    public void Main()
    {
    if (Dts.Variables["RetryCounter"].Value.ToString() != Dts.Variables["RetryMax"].Value.ToString())
    {
    // Fire warning message that the previous task failed
    Dts.Events.FireWarning(0, "Wait", "Attempt " + Dts.Variables["RetryCounter"].Value.ToString() + " of " + Dts.Variables["RetryMax"].Value.ToString() + " failed. Retry in " + Dts.Variables["RetryPause"].Value.ToString() + " seconds.", string.Empty, 0);

    // Wait x seconds
    System.Threading.Thread.Sleep(Convert.ToInt32(Dts.Variables["RetryPause"].Value) * 1000);

    // Succeed Script Task and continue loop
    Dts.TaskResult = (int)ScriptResults.Success;
    }
    else
    {
    // Max retry has been reached. Log, fail and quit
    Dts.Events.FireError(0, "Wait", "Attempt " + Dts.Variables["RetryCounter"].Value.ToString() + " of " + Dts.Variables["RetryMax"].Value.ToString() + " failed. No more retries.", string.Empty, 0);

    // Fail Script Task and quit loop/package
    Dts.TaskResult = (int)ScriptResults.Failure;
    }
    }

    #region ScriptResults declaration
    ///
    /// This enum provides a convenient shorthand within the scope of this class for setting the
    /// result of the script.
    ///
    /// This code was generated automatically.
    ///

    enum ScriptResults
    {
    Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
    Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    };
    #endregion
    }
    }


    4a) Quit loop on success
    To quit the loop when your task executes successfully we are setting the Boolean variable QuitForLoop to true. I'm using a Script Task for it, but you could also use an Expression Task if you're using 2012 and above or a custom Expression Task for 2008.

    Add a Script Task, give it a useful name and connect it to the task you want to retry. Edit the Script Task and add the integer variables RetryCounter and RetryMax as read-only variables and the Boolean variable QuitForLoop as read-write variable. After this hit the Edit button and go to the next step.
    read-only and read-write variables


















    4b) The Script
    Copy the contents of my main method to your main method. This code will set the Boolean variable to true causing the loop to stop. The example code is in C#. For a VB.Net version you can use this conversion tool.
    // C# Code
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Runtime;
    using System.Windows.Forms;

    namespace ST_a72ebc0827b64c0f8a1083951014129c
    {
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
    public void Main()
    {
    // Fire information message that the previous task succeeded
    bool FireAgain = true;
    Dts.Events.FireInformation(0, "Succeeded", "Attempt " + Dts.Variables["User::RetryCounter"].Value.ToString() + " of " + Dts.Variables["User::RetryMax"].Value.ToString() + " succeeded. Quiting loop", string.Empty, 0, ref FireAgain);

    // Fill boolean variable with true so that the FOR LOOP EvalExpression will evaluate false and quit
    Dts.Variables["User::QuitForLoop"].Value = true;

    Dts.TaskResult = (int)ScriptResults.Success;
    }

    #region ScriptResults declaration
    ///
    /// This enum provides a convenient shorthand within the scope of this class for setting the
    /// result of the script.
    ///
    /// This code was generated automatically.
    ///

    enum ScriptResults
    {
    Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
    Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    };
    #endregion
    }
    }

    5) Event handler propagate
    Now the most important step! A failure on one of the tasks within a parent container (package, sequence, for loop or foreach loop) will also cause the parent container to fail, but we want to continue after an error. The trick is to add an empty OnError event handler on the task that could fail and change the value of the propagate variable .

    In this example we want to ignore errors on the webservice task. Go to the event handlers. Select the webservice task as executable and select OnError as eventhandler and then create the event handler by clicking on the link in the middle of the screen: "Click here to create an 'OnError' eventhandler for executable 'WST - Call webservice'.

    Next go to the variables pane and hit the Variable Grid Option button to also show system variables (SSIS 2008 has a different button). Then find the system variable Propagate and set it to false.

    Last step is to add an annotation in the empty event handler to explain why it's empty.
    
    Empty OnError event handler with propagate set to false


















    Show system variables for SSIS 2008








    Note: propagate will only work within a package not in a parent-child package construction, but there is a workaround available.

    6) testing
    In the first run the third attempt was successful and the package succeeded. In the second run all five attempts failed and so did the package.
    Third attempt was successful

    All five attempts failed

    Is the Row Sampling Transformation fully blocking?

    $
    0
    0
    Case
    Is the Row Sampling Transformation a (fully) blocking component or a non-blocking component? Various blogs and forums disagree on each other. Is it possible to see it in SSIS?

    Solution
    By adding some logging to the package you can see the Pipeline Execution Trees of your Data Flow. This will tell you whether a transformation creates a new buffer. When it creates a new buffer then the component is a-synchronously / blocking.


    1) Add logging
    Open your package and go to the SSIS menu and choose Logging...
    SSIS Menu - Logging...
















    2) SQL logging
    For this example I selected SQL Server logging, pressed the add button and then selected a Connection Manager.
    SQL Server logging






















    Next check the Data Flow Task (1) and the enable the SQL Server Log (2) for it. After that go to the Details tab (3).
    Enable logging for the Data Flow Task





















    In the details tab scroll down and select PipelineExecutionTrees. Then click on the advanced button.
    Logging Details





















    In the advanced section make sure that the MessageText is checked. In this column you will find the execution trees text. After that click OK to close the logging window.
    Check MessageText
















    3) Run and check log
    Run the package and then check the log. In the column Message you can find the Execution Trees.
    SQL Server Logging. Check message column.











    Now check these examples and see which is blocking. Multiple paths means that there is a transformation that creates new buffers. The last example is the Row Sampling:
    Plain Data Flow: 1 path - No blocking

    Data Flow with Sort: 2 paths - Sort is blocking

    Data Flow with Percentage Sampling: 1 path - No blocking

    Data Flow with Row Sampling: 3 paths - Row Sampling is blocking














































    4) Partial Blocking or Fully Blocking?
    Unfortunately you can't see in the log whether the task is partially blocking (like union or merge join) or fully blocking (like sort and aggregate), but you can see it in the Data Flow Task when running your package in Visual Studio (BIDS/SSDT). The Row Sampling Transformation is fully blocking because it apparently need all data before it sends data to its output. So try not to use it unnecessarily.
    
    Row Sampling is Fully Blocking































    The Row Sampling isn't just doing a TOP X, but it spreads the sampled rows over all buffers. Because you don't know the number of records or buffers that is coming you have to have all rows before you can pick randomly X records from the whole set.

    An alternative (with less random rows) could be to use a Script Component that adds a row number and then use a Conditional Split to select the first X rows (and perhaps combine it with a modulo expression like: Rownumber % 3 == 0 && Rownumber <= 3000). The Conditional Split is a non-blocking component. Note: this isn't necessarily faster. Check it first for your case!

    So why is the Percentage Sampling not blocking? It just takes the same percentage of each buffer and can therefore be synchronous.


    Confirmed by Mister SSIS himself!

    Highlights

    $
    0
    0
    My third son was born yesterday!
    Jasper Arthur



    And less important, but still cool: millionth pageview today :-)

    Regards,

    Joost



    SSIS 2012 Execute Package Task: External Reference does not work with SSISDB

    $
    0
    0
    Case
    I have two SSIS projects (both project deployment) and I within project 'A' I want to execute a package from project 'B' with the Execute Package Task. Project reference won't work because there are two different projects, but External reference won't work either because it doesn't support packages from the SSISDB (only file or MSDB).
    Pointing to MSDB instead of SSISDB























    Solution
    See (/vote) this MsConnect item. They are still considering to address this issue. Here is a workaround, but it requires a little coding. I while a go I did a post on executing an SSIS 2012 package from a .Net application and I thought that it would also be possible within an SSIS Script Task.

    1) Connection Manager
    Create an ADO.Net connection manager that points to the SSISDB on your server.
    ADO.Net connection (don't use OLEDB)























    2) String variable
    Create a string variable and add the path from the package that you want to execute. Format is /SSISDB/folder/project/package.dtsx
    String variable filled with package path
















    3) Script Task
    Add a Script Task to the Control Flow and give it a suitable name. Then edit the Script Task and add the string variable from step 2 as readonly variable.
    Script Task - ReadOnlyVariables


























    4) The Script - References
    Choose the Scripting Language and hit the Edit Script button. We need to reference 4 assemblies, but they are not in the SQL Server folder. They are only available in the GAC. The path varies a little per computer. These are mine:
    C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.ConnectionInfo\11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.ConnectionInfo.dll
    C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.Management.Sdk.Sfc\11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.Management.Sdk.Sfc.dll
    C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.Smo\11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.Smo.dll
    C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.Management.IntegrationServices\11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.Management.IntegrationServices.dll


    Right click references in the Solution Explorer and choose Add Reference... Then browse these four dll files and add them one by one.
    Add references

























    IMPORTANT: After adding the references you should press the Save All button to save the reference changes!

    5) The Script - Code
    Now copy the usings(/imports) from my code and copy the content of my Main method to your main method. The example is in C#, but you can use this translator to get VB.Net code
    // C# Code
    #region Namespaces
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Runtime;
    using System.Windows.Forms;
    // Added:
    using System.Data.SqlClient;
    using Microsoft.SqlServer.Management.IntegrationServices;
    using System.Collections.ObjectModel;
    #endregion

    namespace ST_e71fdb73f68c4a3f9595ea5d37464a62
    {
    /// <summary>
    /// ScriptMain is the entry point class of the script. Do not change the name, attributes,
    /// or parent of this class.
    /// </summary>
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
    public void Main()
    {
    // Boolean variable for firing event messages
    bool fireAgain = true;

    // Execution of child package starting
    Dts.Events.FireInformation(0, "Child Package - " + Dts.Variables["User::SSISDBPackagePath"].Value.ToString(), "Starting", string.Empty, 0, ref fireAgain);

    try
    {
    // Connection to the database server where the packages are located
    // SqlConnection ssisConnection = new SqlConnection(@"Data Source=.\SQL2012;Initial Catalog=SSISDB;Integrated Security=SSPI;");
    SqlConnection ssisConnection = new SqlConnection(Dts.Connections["mySqlServerAdoNet"].ConnectionString);

    // SSIS server object with connection
    IntegrationServices ssisServer = new IntegrationServices(ssisConnection);

    // Split the variable containing the package path in smaller
    // parts: /SSISDB/Folder/Project/Package.Dtsx
    string[] SSISDBPackagePath = Dts.Variables["User::SSISDBPackagePath"].Value.ToString().Split('/');

    // The reference to the package which you want to execute
    // Microsoft.SqlServer.Management.IntegrationServices.PackageInfo ssisPackage = ssisServer.Catalogs["SSISDB"].Folders["folder"].Projects["project"].Packages["package.dtsx"];
    Microsoft.SqlServer.Management.IntegrationServices.PackageInfo ssisPackage = ssisServer.Catalogs[SSISDBPackagePath[1]].Folders[SSISDBPackagePath[2]].Projects[SSISDBPackagePath[3]].Packages[SSISDBPackagePath[4]];

    // Add execution parameter to override the default asynchronized execution. If you leave this out the package is executed asynchronized
    Collection<Microsoft.SqlServer.Management.IntegrationServices.PackageInfo.ExecutionValueParameterSet> executionParameter = new Collection<Microsoft.SqlServer.Management.IntegrationServices.PackageInfo.ExecutionValueParameterSet>();
    executionParameter.Add(new Microsoft.SqlServer.Management.IntegrationServices.PackageInfo.ExecutionValueParameterSet { ObjectType = 50, ParameterName = "SYNCHRONIZED", ParameterValue = 1 });

    // For adding more parameters go to my WIKI post on MSDN:
    // http://social.technet.microsoft.com/wiki/contents/articles/21978.execute-ssis-2012-package-with-parameters-via-net.aspx

    // Get the identifier of the execution to get the log
    long executionIdentifier = ssisPackage.Execute(false, null, executionParameter);

    // If you want to catch the events from the package you are executing then you can add this
    // foreach loop. It reads the events and fires them as events. You can remove this loop if
    // you're not interested in them.

    // Loop through the log and fire events
    foreach (OperationMessage message in ssisServer.Catalogs["SSISDB"].Executions[executionIdentifier].Messages)
    {
    // Translate Message Source Type code and Message Type code to description. See
    // MSDN for the complete list http://msdn.microsoft.com/en-us/library/ff877994.aspx

    string messageSourceType = "";
    switch (message.MessageSourceType)
    {
    case 10:
    messageSourceType = "Entry APIs, such as T-SQL and CLR Stored procedures";
    break;
    case 20:
    messageSourceType = "External process used to run package (ISServerExec.exe)";
    break;
    case 30:
    messageSourceType = "Package-level objects";
    break;
    case 40:
    messageSourceType = "Control Flow tasks";
    break;
    case 50:
    messageSourceType = "Control Flow containers";
    break;
    case 60:
    messageSourceType = "Data Flow task";
    break;
    }

    // Translate Message Type (=event)
    string messageType = "";
    switch (message.MessageType)
    {
    case -1:
    messageType = "Unknown";
    break;
    case 120:
    messageType = "Error";
    break;
    case 110:
    messageType = "Warning";
    break;
    case 70:
    messageType = "Information";
    break;
    case 10:
    messageType = "Pre-validate";
    break;
    case 20:
    messageType = "Post-validate";
    break;
    case 30:
    messageType = "Pre-execute";
    break;
    case 40:
    messageType = "Post-execute";
    break;
    case 60:
    messageType = "Progress";
    break;
    case 50:
    messageType = "StatusChange";
    break;
    case 100:
    messageType = "QueryCancel";
    break;
    case 130:
    messageType = "TaskFailed";
    break;
    case 90:
    messageType = "Diagnostic";
    break;
    case 200:
    messageType = "Custom";
    break;
    case 140:
    messageType = "DiagnosticEx";
    break;
    case 400:
    messageType = "NonDiagnostic";
    break;
    case 80:
    messageType = "VariableValueChanged";
    break;
    }

    // Fire event depending on the message type (event) in the child package. Since there are event types that you
    // can't fire from a Script Task, we need to 'translate' them. For example a TaskFailed event is fired as an
    // error event. More info see: http://microsoft-ssis.blogspot.com/2011/02/script-task-and-component-logging.html
    switch (message.MessageType)
    {
    case -1: // Unknown
    case 120: // Error
    case 130: // TaskFailed
    Dts.Events.FireError(Convert.ToInt32(message.MessageType), "Child Package - " + messageSourceType, messageType + " : " + message.Message, string.Empty, 0);
    break;
    case 110: // Warning
    Dts.Events.FireWarning(Convert.ToInt32(message.MessageType), "Child Package - " + messageSourceType, messageType + " : " + message.Message, string.Empty, 0);
    break;
    default:
    Dts.Events.FireInformation(Convert.ToInt32(message.MessageType), "Child Package - " + messageSourceType, messageType + " : " + message.Message, string.Empty, 0, ref fireAgain);
    break;
    }
    } // END FOREACH LOOP

    if (ssisServer.Catalogs["SSISDB"].Executions[executionIdentifier].Status == Operation.ServerOperationStatus.Success)
    {
    // Execution of child package succeeded
    Dts.Events.FireInformation(0, "Child Package - " + Dts.Variables["User::SSISDBPackagePath"].Value.ToString(), "Succeeded", string.Empty, 0, ref fireAgain);
    Dts.TaskResult = (int)ScriptResults.Success;
    }
    else
    {
    // Execution of child package failed
    Dts.Events.FireError(0, "Child Package - " + Dts.Variables["User::SSISDBPackagePath"].Value.ToString(), "There may be error messages posted before this with more information about the failure.", string.Empty, 0);
    Dts.TaskResult = (int)ScriptResults.Failure;
    }

    }
    catch (Exception ex)
    {
    // Execution of child package failed (server timeout, can't find package, etc.)
    Dts.Events.FireError(0, "Child Package - " + Dts.Variables["User::SSISDBPackagePath"].Value.ToString(), "Failed: " + ex.Message, string.Empty, 0);
    Dts.TaskResult = (int)ScriptResults.Failure;
    }
    }

    #region ScriptResults declaration
    /// <summary>
    /// This enum provides a convenient shorthand within the scope of this class for setting the
    /// result of the script.
    ///
    /// This code was generated automatically.
    /// </summary>
    enum ScriptResults
    {
    Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
    Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    };
    #endregion

    }
    }


    6) The result
    Now you can execute the package and see the result. You could add some filtering in the message loop to reduce the number of messages.
    The result in Visual Studio



















    Note 1: The other project already needs to be deployed to the SSIS Catalog
    Note 2: You get two execution with both their own ServerExecutionId
    Two executions






    If you want to pass parameters to the child package then you could check out my WIKI example on parameters.

    There are alternatives like executing a SQL Server Agent job via an Execute SQL Task or Calling DTExec via an Execute Process Task.

    Nested includes in BIML Script

    $
    0
    0
    Case
    I want to use nested includes in a BIML Script (an include in an include), but the second level isn't working. It seems to skip it without giving an error.
    No second Sequence Container













    Solution
    First be careful with (too many) nested includes! It could make your BIML script obscure. There are two tricks to solve this problem. They came to me via twitter from @cathrinew and @AndreKamman.

    Solution A:
    Use a full path in the include tag instead of only the name:
    Using the fullpath













    Big downside is of course the full path in your BIML Script. In a multi-user environment with for example TFS that could be an issue because everybody needs the same project path.

    Solution B:
    A better option is to use CallBimlScript instead of include:
    Using CallBimlScript












    And you could also pass parameters to the included file and use relative path and then reuse the file in multiple projects.

    Prevent events executing multiple times

    $
    0
    0
    Case
    I have a log task in an OnPreExecute and OnPostExecute event handler, but it executes multiple times. Why is that and how can I prevent it?

    Solution
    This is because all tasks and containers fire events and these events are propagated to their parent container and then to their parent container and so on. This means that if you have a package with a Sequence Container with and with an Execute SQL Task in it, that each of them fires events. Let's test that.

    For testing purposes I added an Execute SQL Task in each event handler with an insert query to show which events are fired. Each task inserts the name of the event and the value of the System Variable SourceName in a log table with an identity column LogId.
    Log all events, executable is package



















    When you run the package you can see that for example the OnPreExecute event has three records:
    1) the Package
    2) the Sequence Container
    3) the Execute SQL Task
    Event handlers executing multiple times



















    The trick
    The trick to execute the Execute SQL Task in the event handler(s) only once, is to check whether the source of the event is the package and not one of it's children (containers/tasks).

    Add a dummy Script Task or an empty Sequence Container in front of the Execute SQL Task and add a Precedence Constrain expression between them: @[System::SourceName] ==  @[System::PackageName]
    Expression to filter events that are not from the package




















    Add Expression Builder to custom task

    $
    0
    0
    Case
    I have created a custom task, but I would like to add the built-in Expression Builder to it. How do you do that?

    Solution
    It is possible, but it's an unsupported feature, which means you have no guarantees that it will still work after the next update of SSIS. For this example I used my Custom Task example and added two references and a couple of code lines. You can download that code and add the code below to the UI project.
    Custom Task with built-in Expression Builder















    1) References
    For the Expression Builder window you need to add a reference to Microsoft.DataTransformationServices.Controls which is located in the GAC:
    C:\Windows\Microsoft.NET\assembly\GAC_MSIL\Microsoft.DataTransformationServices.Controls\v4.0_11.0.0.0__89845dcd8080cc91\Microsoft.DataTransformationServices.Controls.DLL
    And for validating the expression you need a reference to Microsoft.SqlServer.DTSRuntimeWrap which is also located in the GAC (however not in MSIL):
    C:\Windows\Microsoft.NET\assembly\GAC_32\Microsoft.SqlServer.DTSRuntimeWrap\v4.0_11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.DTSRuntimeWrap.dll
     C:\Windows\Microsoft.NET\assembly\GAC_64\Microsoft.SqlServer.DTSRuntimeWrap\v4.0_11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.DTSRuntimeWrap.dll


    2) Usings
    I added two extra usings for the Expression Builder.
    
    Two extra usings


















    3) Controls
    I added a button (for opening expression builder) a readonly textbox (for showing the expression) and a label (for showing the evaluated expression) to my editor.
    Extra controls in the editor








    4) The code
    I added an onclick event on my button and added the following code (simplified version).
    // C# code
    private void btnExpression_Click(object sender, EventArgs e)
    {
    try
    {
    // Create an expression builder popup and make sure the expressions can be evaluated as a string:
    // Or change it if you want boolean to System.Boolean, etc. Last property is the textbox containing
    // the expression that you want to edit.
    using (var expressionBuilder = ExpressionBuilder.Instantiate(_taskHost.Variables,
    _taskHost.VariableDispenser,
    Type.GetType("System.String"),
    txtExpression.Text))
    {
    // Open the window / dialog with expression builder
    if (expressionBuilder.ShowDialog() == DialogResult.OK)
    {
    // If pressed OK then get the created expression
    // and put it in a textbox.
    txtExpression.Text = expressionBuilder.Expression;
    lblExpressionEvaluated.Text = "";

    // Create object to evaluate the expression
    Wrapper.ExpressionEvaluator evalutor = new Wrapper.ExpressionEvaluator();

    // Add the expression
    evalutor.Expression = txtExpression.Text;

    // Object for storing the evaluated expression
    object result = null;

    try
    {
    // Evalute the expression and store it in the result object
    evalutor.Evaluate(DtsConvert.GetExtendedInterface(_taskHost.VariableDispenser), out result, false);
    }
    catch (Exception ex)
    {
    // Store error message in label
    // Perhaps a little useless in this example because the expression builder window
    // already validated the expression. But you could also make the textbox readable
    // and change the expression there (without opening the expression builder window)
    lblExpressionEvaluated.Text = ex.Message;
    }

    // If the Expression contains some error, the "result" will be <null>.
    if (result != null)
    {
    // Add evaluated expression to label
    lblExpressionEvaluated.Text = result.ToString();
    }
    }
    }
    }
    catch (Exception ex)
    {
    MessageBox.Show(ex.Message);
    }
    }

    5) Runtime
    Now you can store that expression in a property and retrieve in on runtime. On runtime you can evaluate the expression with the same code as above.

    XSD location hardcoded in XML source

    $
    0
    0
    Case
    The path of the XSD file in the XML Source component is hardcoded, but the path on my production environment is different than my development environment. The XML source doesn't have expressions. How do I make this XSD path configurable?
    XML Source Connection Manager Page
    without Connection Managers



























    Solution
    Unfortunately the XML source component does lack some very basic functionalities like the use of a Connection Manager (although ironically it's called the Connection Manager page). The source can use variables as input, but there isn't such option for the XSD file.
    XML Task can use Connection Managers


























    1) Find Expressions
    The XML Source Component doesn't support expressions, but the Data Flow Task itself does. Go to the properties of the Data Flow Task and locate the expressions and click on the ... button
    
    Select Data Flow Task and press F4 to get properties


























    2) Select Property
    Locate the XMLSchemaDefinition property of your XML Source in the Property Expression Editor and add an expression on it by clicking on the ... button.
    Property Expression Editor
















    3) Add Expression
    Now you can either replace its value by a variable or a parameter if you use SSIS 2012 and above.
    Expression Builder
























    That's it. Now you have a workaround for the absence of a real Connection Manager. An alternative could be to use a Script Component as XML source or an XML file with an inline XSD schema.
    Viewing all 149 articles
    Browse latest View live