OptionSet Batch Management for Microsoft CRM 2011

So long no post since the previous post. I used to work with Microsoft CRM 4.0 and Microsoft CRM 2011 for a few years back and made several tools for particular usages. I was also trying to blog and share them but something blocked me from finishing it. Here it is one of them I can recall the location.

I recently have a chance to back to work on Microsoft CRM 2011 after quite a long switched to custom application and web application development. I discovered that Microsoft CRM 2011 has now quite a lot of tools to ease the developer and/or administrator to develop and customize the system. There are many more than the last time I developed 2011 version. However, as we all know the clients never ending asking for a special feature which standard features of Microsoft CRM doesn’t provide. It is the reason why those complexity and huge systems like CRM and ERP need to have developers.

My latest client has many teams joining in the project and many people do entity customization. It causes inconsistency to be happened. One of the ruffle thing is that the value of OptionSets are like a jumping number. For instance; 1:Cash, 2:Credit Card, 1000000:Debit. This could easily happen if you have many distributed solutions(for dev maybe) to merge to UAT and PROD. Each solution could set it’s own configurable initial number of OptionSet value.

When I see this kind of mess, I really need to clean it up before hand it to production environment. The problem was the system was huge and custom OptionSets are all over the place. I saw several OptionSet manager tools but all of them aim one by one OptionSet edit and actually none of them touch the OptionSet’s value.  I can understand that it is unsafe to change the OptionSet value if it is referenced by any existing record. Though in some case like just before delivering the system to the new environment with completely blank data; we are safe to do this kind of clean up.

I so develop my own tool to manage this kind of need and I named it OptionSet Batch Manager for Microsoft CRM 2011. I uploaded to CodePlex here. For the first version, it could do reorder by value and name and resetting value of OptionSets either as a batch(upon selection) or individual. Hope someone find it useful!

OptionSet Batch Manager screenshot

Addition note: I used Reactive Extension or so called Rx to manage thread for service calls. For anyone who interested, you can either use TFS to checkout the full source code from CodePlex or directly download source code which I will soon upload.

I’ll share some tricks to compare and merge two inconsistent CRM 2011 solutions in the next post.

Update Service Reference then get error – The maximum nametable character count quota (16384) has been exceeded while reading XML data.

If you have WCF and trying to update service reference via Visual Studio then right away get error message saying that “The maximum nametable character count quota (16384) has been exceeded while reading XML data”, this means your service becomes too big.

This was just because of the service grows too big and exceed the maximum size that default setting examined. So we have to expand the limitation of size of Visual Studio so we can update service through Visual Studio as usual again.

1. Go to C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE (up to the version you are using, mine is Visual Studio 2010).

2. Find devenv.exe.config and open to edit it with Administrator right, thus the editor must be opened as Administrator. Otherwise you won’t be able to save change.

3. Add below config for service model then save it.

<system.serviceModel>

 <client>

 <endpoint binding="netTcpBinding" bindingConfiguration="GenericBinding" contract="IMetadataExchange" name="net.tcp" />

 </client>

 <bindings>

 <netTcpBinding>

 <binding name="GenericBinding"

 maxBufferPoolSize="2147483647" maxBufferSize="2147483647"

 maxReceivedMessageSize="2147483647">

 <readerQuotas maxDepth="2147483647"

 maxStringContentLength="2147483647"

 maxArrayLength="2147483647"

 maxBytesPerRead="2147483647"

 maxNameTableCharCount="2147483647" />

 <security mode="None" />

 </binding>

 </netTcpBinding>

 </bindings>

 </system.serviceModel>

Note that line 3, 6 and 17 are up to the protocol that you are using.

4. Restart Visual Studio if opening.

Done. You should be able to update service via Visual Studio now.

Use Transactions and Rollback for Unit Testing against WCF RIA Services

Apparently Silverlight provide use of WCF RIA Services as a data access tool, it behaves just like an ORM tool but over the wire.

How it works ? Well, it names express quite straightforward. It has WCF as interface to let client request and response the data. At the client side, it does generate sort of proxy as same as regular WCF or even old-fashioned web service does. It uses Entity Framework as an ORM tool to manipulate database.

In this post, I will assume that we are going to write unit test against server operations. So proxy and communication over the wire thing will not be considered.

The approach that I’m going to show you is pretty much the same as my previous post “Use Transactions and Rollback for Unit Testing against Entity Framework“, because as I mentioned that RIA uses Entity Framework as ORM tool.

Note that you will need to keep classes from 1-3 in RIA Services project, 4-5 in Test Project.

1. Create a new class that override DBTransaction class.

• Make the constructor to accept a parameter of DbTransaction object. So this class acts just like an adapter of DbTransaction class.
• By pass the real rollback and commit behavior by overriding Rollback and Commit functions so actual rollback and commit functions will not do anything.
• Create RollbackUnitTest function to be called from another class that we are going to write it up next.

public class TestDbTransaction : System.Data.Common.DbTransaction
{
   public DbTransaction ActualDBTransaction { get; private set; }

   public TestDbTransaction(DbTransaction actualDBTransaction)
   {
      ActualDBTransaction = actualDBTransaction;
   }

   public override void Rollback()
   {
      // do nothing : the test transaction is already running
   }

   public override void Commit()
   {
      // do nothing : the test transaction is already running
   }

   public void RollbackUnitTest()
   {
      ActualDBTransaction.Rollback();
   }

   protected override System.Data.Common.DbConnection DbConnection
   {
      get { return ActualDBTransaction.Connection; }
   }

   public override IsolationLevel IsolationLevel
   {
      get { return ActualDBTransaction.IsolationLevel; }
   }
}

2.  Create extension methods of ObjectContext named BeginTransaction to begin the transaction and return TestDbTransaction object out.

  public static class TestObjectContextExtender
  {
        public static System.Data.Common.DbTransaction BeginTransaction(this System.Data.Objects.ObjectContext context, IsolationLevel isolationLevel)
        {
            var actualTransaction = context.Connection.BeginTransaction(isolationLevel);
            return new TestDbTransaction(actualTransaction);
        }

        public static System.Data.Common.DbTransaction BeginTransaction(this System.Data.Objects.ObjectContext context)
        {
            return new TestDbTransaction(context.Connection.BeginTransaction());
        }
  }

3. Create a partial class of Domain Service.
We need to get access to the ObjectContext which has access level as protected. So we can either override it or create partial class. I go with partial and make extension method to avoid lots of changes.

Don’t forget to replace YOURDomainService and YOUREntities with yours.

    public partial class YOURDomainService : LinqToEntitiesDomainService<YOUREntities>
    {
        public TestDbTransaction BeginTransactionForUnitTest()
        {
            this.ObjectContext.Connection.Open();
            return this.ObjectContext.BeginTransaction() as TestDbTransaction;
        }

        public YOUREntities GetDataContext()
        {
            return this.ObjectContext;
        }
    }

4. Create a base class to be inherited by each unit test class.

TestStart method is being called before every unit test and TestEnd after. This is the result of pasting attribute TestInitialize and TestCleanup on them.

    [TestClass()]
    public class TransactedTestClass
    {
        protected GeneralDomainService _service;
        private TestDbTransaction _transaction;

        [TestInitialize()]
        public void TestStart()
        {
            DomainServiceContext context = new DomainServiceContext(new ServiceProvider(), DomainOperationType.Query);
            _service= new GeneralDomainService();
            _service.Initialize(context);

            _transaction = _service.BeginTransactionForUnitTest();
        }

        [TestCleanup()]
        public void TestEnd()
        {
            _transaction.RollbackUnitTest();
        }
    }

5. Sample of use. Create a test class and inherit from TransactedTestClass.

• _service is an instance declared in TransactedTestClass.
• GetDataContext method is declared in partial class in 3.

[TestClass()]
public class GeneralDomainServiceTest : TransactedTestClass
{
        [TestMethod()]
        public void AddNewUser_Test()
        {
            // Get ObjectContext for validation (directly query to database).
            var dataContext = _service.GetDataContext();
            // Assume that our test database has no record in Users table
            Assert.AreEqual(0, dataContext.Users.Count());
            _service.AddNewUser(new User());
            // Verify that one record is added
            Assert.AreEqual(1, dataContext.Users.Count());
        }
}

Use Transactions and Rollback for Unit Testing against Entity Framework

Refer to my previous post that talked about Unit Test with Rollback Pattern, Entity Framework is one of ORM tool that developer has to learn it to use accurately. So if you use Mock to do such a by pass of querying against Entity Framework, it is pretty unsafe really. Except you are very confident that you (or your developer) are mastery of Entity Framework.

First of all you need to have a separate database for test only and prepare some data for testing purpose. Then change the connection string in config file to point to this database rather than the development database.

Here is how I wrap and override the object context.

1. Create a new class that override DBTransaction class.

• Make the constructor to accept a parameter of DbTransaction object. So this class acts just like an adapter of DbTransaction class.
• By pass the real rollback and commit behavior by overriding Rollback and Commit functions so actual rollback and commit functions will not do anything.
• Create RollbackUnitTest function to be called from another class that we are going to write it up next.

public class TestDbTransaction : System.Data.Common.DbTransaction
{
   public DbTransaction ActualDBTransaction { get; private set; }

   public TestDbTransaction(DbTransaction actualDBTransaction)
   {
      ActualDBTransaction = actualDBTransaction;
   }

   public override void Rollback()
   {
      // do nothing : the test transaction is already running
   }

   public override void Commit()
   {
      // do nothing : the test transaction is already running
   }

   public void RollbackUnitTest()
   {
      ActualDBTransaction.Rollback();
   }

   protected override System.Data.Common.DbConnection DbConnection
   {
      get { return ActualDBTransaction.Connection; }
   }

   public override IsolationLevel IsolationLevel
   {
      get { return ActualDBTransaction.IsolationLevel; }
   }
}

2.  Create extension methods of ObjectContext named BeginTransaction to begin the transaction and return TestDbTransaction object out.

  public static class TestObjectContextExtender
  {
        public static System.Data.Common.DbTransaction BeginTransaction(this System.Data.Objects.ObjectContext context, IsolationLevel isolationLevel)
        {
            var actualTransaction = context.Connection.BeginTransaction(isolationLevel);
            return new TestDbTransaction(actualTransaction);
        }

        public static System.Data.Common.DbTransaction BeginTransaction(this System.Data.Objects.ObjectContext context)
        {
            return new TestDbTransaction(context.Connection.BeginTransaction());
        }
  }

3. Create a base class to be inherited by each unit test class.

TestStart method is being called before every unit test and TestEnd after. This is the result of pasting attribute TestInitialize and TestCleanup on them.

    [TestClass()]
    public class TransactedTestClass
    {
        protected DBEntities _dataRollbackContext;
        private TestDbTransaction _transaction;

        [TestInitialize()]
        public void TestStart()
        {
            _dataRollbackContext = new DBEntities();
            _dataRollbackContext.Connection.Open();
            _transaction = _dataRollbackContext.BeginTransaction() as TestDbTransaction;
        }

        [TestCleanup()]
        public void TestEnd()
        {
            _transaction.RollbackUnitTest();
        }
    }

4. Sample of use. Create a test class and inherit from TransactedTestClass.

[TestClass()]
public class DataRepositoryTest : TransactedTestClass
{
        [TestMethod()]
        public void AddNewUser_Test()
        {
            // In this example I made up the data repository and use dependency injection for the data context.
            DataRepository target = new DataRepository(_dataRollbackContext);
            // Assume that our test database has no record in Users table
            Assert.AreEqual(0, _dataRollbackContext.Users.Count());
            target.AddNewUser(new User());
            // Verify that one record is added
            Assert.AreEqual(1, _dataRollbackContext.Users.Count());
        }
}

There are other approaches that expect the same result.

• One is wrapping up each unit test with usage of TransactionScope without commit statement. This ensure that any change to the data in database will be rollback when the unit test is done.  Even though, this seems to be easy and straightforward but because of Entity Framework basically begins transaction automatically and commits once SaveChanges method is being called. Begin transaction within another transaction requires MSDTC. Thus, MSDTC must be enabled and this is not make any sense.
• Second is doing a database snapshot and restore every time the test completed. This approach is fairly complicated and difficult to manage.

Prevent Browser Back Button for Silverlight with Confirm Dialog

One of common issues of the Silverlight, is that sometime the user accidentally clicks the browser back button. Since the  Silverlight Application hosts on a HTML page as a Frame. Any action on Silverlight does not cause the browser to keep state. So if the user clicks back button of browser, it would just go back to the page just before it reach the page hosting the Silverlight Application.

Silverlight has Navigation Framework that take care of navigate history over the pages. But some developers may decide not to use it. Or the application goes too far to add such this core functionality.

As many forums have discussed about disable browser back button. There is no direct solution and the browser should not provide it as well. It’s not fair to limit the user not to go out of the website if it really needs to. We can alternatively open the webpage that host the Silverlight Application in new window.  This way can guarantee that no previous page thus the browser back button disables itself. But this workaround is not elegant and unacceptable by my customer.

I then made another tricky workaround to confirm to the user when they clicked the browser back button.

Tested with IE8, Firefox and Chrome.

A couple of things needed to add in order to do this.

1. Insert a new page with a Frame Control at the root of application. This page does capture and confirm when the user clicked the back button.

XAML

    <Grid x:Name="LayoutRoot">
        <sdk:Frame Name="frame1" Source="/MainPage" HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
            <sdk:Frame.UriMapper>
                <sdk:UriMapper>
                    <sdk:UriMapping Uri="/MainPage" MappedUri="/ActualMainPage.xaml" />
                </sdk:UriMapper>
            </sdk:Frame.UriMapper>
        </sdk:Frame>
    </Grid>

Code

public partial class MainPage : Page
    {
        private bool _doNotBlockNavigation = false;

        public MainPage()
        {
            InitializeComponent();

            this.frame1.Navigating += new NavigatingCancelEventHandler(frame1_Navigating);
            this.frame1.NavigationStopped += new NavigationStoppedEventHandler(frame1_NavigationStopped);
        }

        void frame1_NavigationStopped(object sender, NavigationEventArgs e)
        {
            ConfirmWindow confirm = new ConfirmWindow();
            confirm.Closed += (s, ea) =>
            {
                if (((ConfirmWindow)s).DialogResult ?? false)
                {
                    // Invoke Javascript function that simply go back the history
                    HtmlPage.Window.Invoke("GoBack");
                }
                else
                {
                    // Get current page from the Frame
                    var mainPage = frame1.Content as Page;
                    // Get the NavigationService of current page
                    var navigationService = mainPage.NavigationService;

                    // If the user refuses to leave the page then
                    // mark flag not to block GoBack() process
                    _doNotBlockNavigation = true;
                    if (navigationService.CanGoBack)
                        navigationService.GoBack();
                }
            };
            confirm.DataContext = "Are you really want to leave this application to go back ?";
            confirm.Show();
        }

        void frame1_Navigating(object sender, NavigatingCancelEventArgs e)
        {
            // if _doNotBlockNavigation = true then just set it back to false so the navigation works properly for the next time.
            if (_doNotBlockNavigation)
            {
                _doNotBlockNavigation = false;
            }
            else if (e.NavigationMode == NavigationMode.New && e.Uri.OriginalString == "")
            {
                // If the user presses browser's back button then temporary blocks from navigating out.
                // This would course the event NavigationStopped to be fired.
                e.Cancel = true;
            }
        }
    }

Note that the ConfirmWindow on line 15 is my custom Child Window. It behaves just like a regular confirm dialog.
Also if you did not create the project as Navigation Silverlight Project, you would need to add reference to Navigation library.

2. Add this function to the javascript block of aspx/html host page. This function will be called by line 21 of above code-snippet.

function GoBack() {
     history.go(-1);
}

3. Make sure that ActualMainPage.xaml overrides from Page not UserControl. Also do not forget to put the appropriate title for ActualMainPage.xaml, it will be shown as the HTML title. If the Title attribute is not set, it will take the current URI from UriMapper.

4. Modify App.xaml.cs to set root as MainPage instead of ActualMainPage.

this.RootVisual = new MainPage();

5. That is it! It should work properly now.

If you think this post is useful, please leave your comments.

Unit test with rollback pattern

For the person who is not familiar with Unit Testing yet. We basically have two options to do unit testing against functions that do CRUD operations, probably with some easy and/or complex logic. They are Mock and Rollback patterns. To keep this post short, I wouldn’t go into details of what the Mock is but you can search it yourself, there is a bunch of contents out there. This post will go with the Rollback pattern.  So, what is the Rollback pattern?

Well, we need to rollback what we did change to the data in database by the function that the unit test invokes. Including operation(s) in unit test that we may need to execute and it does change data in database.

Pretty simple example. If we run this below test twice, Assert will fail in the second run because the user Id 1 was already deleted by the first run.

 [TestMethod]
 public void DeleteUser_Test()
 {
    int userId = 1;

    bool isDeleted = service.DeleteUser(userId);

    Assert.IsTrue(isDeleted);
 }

The easiest way is to use TransactionScope to wrap it up.

 [TestMethod]
 public void DeleteUser_Test()
 {
    int userId = 1;
    using(TransactionScope ts = new TransactionScope())
    {
       bool isDeleted = service.DeleteUser(userId);

       Assert.IsTrue(isDeleted);
    }
 }

Without calling ts.Complete(), the transaction ts would rollback what DeleteUser function did.

If you are using ORM, you must ensure that it does support TransactionScope. I’m sure that LLBLGen, LinqToSQL, LinqToEntities (Entity Framework) do support.

Also beware of executing the store procedure that creates another transaction inside. If you wrap this calling up in TransactionScope, the TransactionScope does not take over the transaction created inside the store procedure. This means that the TransactionScope couldn’t rollback the transaction inside the store procedure.

I would recommend not to use Rollback Unit test if the testing service executes the store procedure.

Filter Sub Entity in Query of LinqToEntities DomainService

If you are one of the developers, who are working on the project that use Domain Service with LinqToEntities, this common issue should annoy you sometimes.

Normally we use Include(“RelatedEntity”) to tell LinqToEntitiesDomainService to also retrieve ALL records of RelatedEntity that have relation to the main querying entity.

For example with this data:
Customer Table

CustomerId	Name	Sex
1	Mr. Wise	Male
2	Mrs. Fool	Female

Order Table

OrderId	CustomerId	Total	PaymentTypeId
1	1	$5000	1
2	1	$200	1

PaymentType Table

PaymentTypeId	PaymentTypeName
1	Cash
2	Credit Card

With this sample data, if we need all Male Customers along with their Orders. It would look like this.

var customers = from c in this.ObjectContext.Customers.Include("Orders")
                where c.CustomerId == 1
                select c;

But what if we need all Male Customers along with their Orders that Total over $1000 ONLY ?

Well, there is no regular operation to embed condition into the query for filtering out the sub entities. The Where statement only filters the main entity, the Customer in this case.

There is a workaround by modify query with this tricky query.

var customers = (from c in this.ObjectContext.Customers
                 where c.CustomerId == 1
                 select new {
                   MaleCustomer = c,
                   OrderOverThousand = c.Orders.Where(o => o.Total > 1000)
                 }).ToList().Select(c=>c.MaleCustomer);

Note that you do not need to have Include operation for this query. It does populate all entities you specified in the enclosed anonymous class after the Select statement.

You may find this workaround in several blogs but I couldn’t find two level depth of related entity.

With the same query condition but also expect PaymentType of Orders, it would look like this:

var customers = (from c in this.ObjectContext.Customers
                 where c.CustomerId == 1
                 let filteredOrders = c.Orders.Where(o => o.Total > 1000)
                 select new {
                   MaleCustomer = c,
                   OrderOverThousand = filteredOrders,
                   OrderPaymentType = filteredOrders.Select(o => o.PaymentType)
                 }).ToList().Select(c=>c.MaleCustomer);

As you can see, all you have to do is just put another property of anonymous class as many entities as you want. You may doubt, “What does the let statement do?” and “Why do we need it?”. It is optional but it can save you from unnecessary addition Select operation when the context transforms it into the SQL statement.

As you can see at line number 6 and 7, without let statement, it would be c.Orders.Where(o => o.Total > 1000) instead of the filteredOrders variable.  As a result, both will do Select operation twice for the same result set.

Hope this post can elucidate some who struggle in such problems 😉 .

Add Include Attribute for Domain Service’s Metadata by Visual Studio AddIn – Part 2

As promised at the end of previous post that I would share the source code and describe more details.

Metadata and Domain Service are both generated automatically after created new Domain Service via wizard. It should be the T4 that do this job behind the scene. Before I began implementing this Add-In, I had taken some thoughts of extending the Text Template file of metadata (.tt). This great post, MetaGen: A project metadata generator for Visual Studio using T4 written by Daniel Vaughan, was done quite the same as my thought but for another purpose.

However I want to keep away from overriding the standard that the Visual Studio provides us especially with the Beta and RC version. If  the Visual Studio changes the template of metadata next month, I don’t want to find the name solid template from somewhere that up-to-date and modify it from scratch. Here is only my personal opinion and I still think that extending template for T4 is more effective for some complex requirements.

Also modifying very complex template is a bit painful. You can easily mess up the generated code by a small mistake.

Add Include Attribute for Domain Service’s Metadata by Visual Studio AddIn – Part 1

I have been working on the project using Silverlight with WCF RIA Service which included Entity Framework and Domain Service using Visual Studio 2010 Beta2  (upgraded to RC). Everyone who experiences creating Domain Service,  must knows metadata file. The well-known problem is that domain service class and metadata (entity definition classes) are automatically generated by wizard.  Everything goes well until you need to update database schema. Since metadata is definition of every class representing tables, it also needs to be updated according to the changes made to Entity Framework.  So far, the Visual Studio doesn’t yet provide ability to use wizard/tool to update metadata. At this point you have two options:

1. Manually update metadata to have up-to-date version that compatible with schema of Entity Framework (EDM file).
2. Delete the existing metadata and domain service files then re-create domain service so metadata would then be up-to-date.

I choose the second option so this post will describe the progress accordingly. If you choose the first option I bet you have got a very stable database schema. Unfortunately, I just don’t have it 😦 as the traditional sad story, when the customers change their minds very often.

You may doubt whether or it, we lose all custom functions that we have implemented to domain service. Well, if you decided to go this way, a little tricky is needed. All custom functions are better in partial class(es) and different file(s), so you wouldn’t get the issue anytime you delete domain service for metadata generating. For me, I categorized functions by its purpose and named them such as – GeneralDomainService.authenticate.cs, GeneralDomainService.profile.cs and so on.

Deleting domain service and metadata then re-adding domain service, this is still acceptable approach since we lose nothing. Again, everything we deleted is only what template generator generated for us.

Until I realized that in order to use domain service over RIA, one additional action must be done.  For any relational entity, which refers to other entity, the Include attribute must be placed over the property.