Simple post.  Simple list for installing TFS 2010.

Here are the downloads required for the installation of TFS 2010, a few simple steps for installation, and guidance for verification. 

Download

Update: Grant Holliday has recently posted a far more comprehensive list at: http://blogs.msdn.com/b/granth/archive/2012/01/03/tfs-2010-what-service-packs-and-hotfixes-should-i-install.aspx

Download the required software and guidance.

Team Foundation Server 2010:

• TFS 2010 installation guide: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=24337
• TFS 2010: Download from your MSDN account, or Trial ISO: http://www.microsoft.com/download/en/details.aspx?id=15070
• TFS 2010 SP1: http://www.microsoft.com/download/en/details.aspx?id=20506
• TFS 2010 SP1 CU1: http://www.microsoft.com/download/en/details.aspx?id=26211

Visual Studio 2010:

• Visual Studio 2010: Download from your MSDN account, or Trial ISO: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=12187
  • Alternatively, you can download the web installer Trial: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=12752 (web installer)
• VS 2010 SP1: http://www.microsoft.com/download/en/details.aspx?id=23691 (web installer)
• TFS 2010 Power Tools (December 2011): http://visualstudiogallery.msdn.microsoft.com/c255a1e4-04ba-4f68-8f4e-cd473d6b971f

SQL Server 2008 R2:

You can use SQL Server 2008 Standard or Enterprise, or SQL Server 2008 R2 Standard or Enterprise for you TFS 2010 environment.  Keep in mind, however, that only the Standard versions are included in the TFS 2010 price of free – you must license the Enterprise licenses separately if you wish to use them with TFS.  Don’t forget to download the x64 version if you have a 64 bit OS.

• SQL Server 2008 R2 Standard: Download from your MSDN account, or Trial: http://msdn.microsoft.com/en-us/evalcenter/ff459612.aspx
• SQL Server 2008 R2 Enterprise: Download from your MSDN account, or Trial: http://msdn.microsoft.com/en-us/evalcenter/ff459612.aspx 

SharePoint 2010:

You can point to an existing SharePoint 2007 or 2010 installation, or you can install and configure a new environment.  If you choose to do a new environment, use SharePoint 2010 (if at all possible), as opposed to WSS 3.0.  It’s not only newer, it’s just easier to use, with far more capabilities.

• SharePoint Foundation 2010: http://www.microsoft.com/download/en/details.aspx?id=5970
• SharePoint Server 2010: Download from your MSDN account, or Trial ISO: http://technet.microsoft.com/en-us/evalcenter/ee388573

Miscellaneous Software:

• TFS 2010 Power Tools – December 2011 (or later): http://visualstudiogallery.msdn.microsoft.com/c255a1e4-04ba-4f68-8f4e-cd473d6b971f

Install

Basically, simply follow the install guide.  Be sure to walk through each of the steps carefully.  Of course, that’s easier said than done.  For instance, if you want to install using 2 machines, you still need to go to the “Install a Stand-Alone Server” section to get information on how to install SQL Server or configure the IIS role.  Still, it’s the best set of steps to follow. 

If you’re doing a dual server install install, you could follow the steps below:

Data Tier:

• Install SQL Server 2008 R2 Standard or Enterprise.
  • The bottom line is to simply install using the wizard with defaults.  However, you’ll need to install SQL Server with Full Text, Analysis Services, Client Tools Connectivity, and Management Tools (at least Basic).
  • Reporting Services should generally be placed on the Application Tier, but it can, if required, be placed on this tier.
    • If installer here, ensure you use the “Install the native mode default configuration.” (It’s the default selection.)
    • WARNING: If you do not install SSRS on the Application Tier, you will still need to install the client connectivity tools there.  Otherwise the App Tier won’t have access to the Data Tier!

Application Tier:

• Configure the IIS Role. You need:
  • ASP.NET (including .NET Extensibility, ISAPI Extensions, and ISAPI Filters)
  • IIS 6.0 Management Compatibility (including IIS 6 Management Console, IIS 6 Scripting Tools, IIS 6 WMI Compatibility, and IIS Metabase and IIS 6 Configuration compatibility)
  • Windows Authentication
• Install and Configure SQL Server Reporting Services (same version as corresponding SQL Server you installed on the Data Tier).
  • Ensure you use the “Install the native mode default configuration.” (It’s the default selection.)
• Install and Configure SharePoint 2010
  • You can also install WSS 3.0 or SharePoint 2007.  WSS 3.0 is the easiest to install, since the TFS 2010 installation can handle that for you.  However, I highly recommend SharePoint 2010.
  • For guidance on installing and configuring SharePoint Foundation 2010, please see http://myalmblog.com/2010/12/20/installing-tfs2010-with-sharepoint-foundation-2010/.  This also applies (close enough) to SharePoint Server 2010.

Verify

You need only do two things for basic verification. 

• Create a new Team Project
  • This exercises much of the TFS installation, and is one of the best ways to ensure your base TFS implementation is working.
• Run the TFS 2010 Best Practices Analyzer.
  • A BPA run can identify issues that might not turn up when simply creating a Team Project, including disk space issues, non-standard configurations and the like.

This post describes how to disable reporting to the warehouse and the OLAP cube for an entire Team Project Collection.  It also discusses why you may want to do so.  But first:

Bottom Line Up Front: Pass only the Collection Name and the state value FullyDisabled to the SetWarehouseJobEnabledState web method in the WarehouseControlWebService.

When to Disabling Reporting

One of the main reasons to disable reporting for an entire TPC is for proof of concept work.  For instance, we are currently moving a IBM Rational ClearQuest implementation to TFS 2010 for a client of ours.  Since we’ll want to run one or more test migrations, we’d like to have a separate TPC.  This prevents work item IDs from getting ‘used’ from their core TPC, which has a tendency to throw people off.  They would go from IDs in the thousands to IDs in the tens of thousands over night.  Not a huge problem, but one we can easily avoid.  Thus, the new TPC.  But we also don’t want to have all the test migration data flow into the data warehouse and the cube, thus muddying the data there. 

Thus, when doing test work that doesn’t involve the data warehouse and cube, disabling the data warehouse is an important step to keep things clean.

How to Disable Reporting

It’s pretty easy, and can be done in only a few steps.

Browse to the WarehouseControlWebService

If you’re on a TFS app tier the URL is likely http://localhost:8080/tfs/TeamFoundation/Administration/v3.0/WarehouseControlService.asmx.  (The only caveat being if you’ve put TFS under a non-standard virtual directory, in which case you’d need to replace the /tfs/ above with your virtual directory.)

Invoke the SetWarehouseJobEnabledState method

This will bring up the SOAP UI for setting the enabled states of various parameters.  For a full list of parameters for TFS 2010

For the collectionName value, enter your Team Project Collection name, leave the jobName blank, and enter FullyDisabled in the state field.  Click Invoke.

Validate the services are disabled

Go back to the WarehouseControlWebService page, and click the GetJobProperties.  Enter no data in any of the fields and click Invoke.

You are provided with a list of Job settings.  Scroll down to the collection name and verify that the EnabledState is FullyDisabled.

How to Enable Reporting

Of course, you may need to turn reporting back on.  To do so, follow all the steps above, but in the SetWarehouseJobEnabledState method, pass in Enabled (instead of FullyDisabled), and click Invoke.  That’s it!  You can then verify using the GetJobProperties method, if you like.

More Info

You may also want to enable or disable individual jobs, instead of removing the entire collection from warehouse and cube processing.  To do so, you can specify an individual jobName in the SetWarehouseJobEnabledState method.  To find out all of the job names that are available to you, run the GetJobProperties method.  Note that some jobs are specified at the server level, and others at the collection level.  For those at the server level, you must leave the collectionName parameter blank when using the SetWarehouseJobEnabledState method. 

Here are the names of the jobs in a default install of TFS 2010:

Server / Instance Level:

• Common Structures Warehouse Sync
• Full Analysis Database Sync
• Incremental Analysis Database Sync

Team Project Collection (TPC) Level:

• Build Warehouse Sync
• Common Structures Warehouse Sync
• Test Management Warehouse Sync
• Version Control Warehouse Sync
• Work Item Tracking Warehouse Sync

Example results from a GetJobProperties call, from a default TFS install, is provided in the image below.

Happy reporting!

Applying TFS 2010 SP1 may, in rare instances, cause an error which prevents the data warehouse (and, by extension, the analysis cube) from updating work item information beyond the date and time at which the SP1 was installed.  This means all work item reports will freeze at the date in which TFS 2010 SP1 was applied.  In addition, the Work Item Tracking Warehouse Sync job will hang and never complete.

Symptoms

After a TFS 2010 SP1 install, work items are no longer processed in the data warehouse, however all of the other elements (such as version control, test cases, etc) are processed. In addition, the Work Item Tracking Warehouse Sync job hangs and never finishes. If you view the data warehouse or the analysis cube, you will find no new data has been recorded since the install of TFS 2010 SP1.

In addition, you will receive an error in the event log similar to this: TF221122: An error occurred running job Work Item Tracking Warehouse Sync for team project collection or Team Foundation server DefaultCollection

Bottom Line Up Front

There was an error in TFS 2010 SP1 that set a watermark improperly. Watermarks are used to mark how many work items have been processed into the warehouse, and where to start for the next iteration. Verifying the problem requires running a SQL query against the transactional database. Fixing the problem (also described at the bottom of the post) required executing an update directly against the SQL transactional database.

WARNING: No direct changes to the database are supported by Microsoft. In order to be supported, you should contact Microsoft PSS, and explain the problem, point them to this blog post, let them do their due diligence first, and have THEM recommend running the script!! I am hesitant to even list the fix, and you run it at your OWN RISK!

Table of Contents

Because this is a relatively large post, I want to provide a quick table of contents to the remainder of the post to help clarify the structure.

• Troubleshooting Steps
  • Show that problem occurs on any work item changed AFTER SP1 was installed
  • Highlight two warnings that occurred during SP1 install (Ended up not being related to problem.)
  • Common, but strange warning in the Event Logs (Ended up not being related to problem.)
  • Event Logs show both Errors and Warnings that are directly related to problem
  • Similar Error found on Bing shows corrupted work item from upgrade caused problems.
  • Other errors caused by permissions or unsolved bugs in TFS, but doesn’t apply since these block ALL processing, not just work items
  • Rebooting doesn’t help.
  • Rebuilding the data warehouse / OLAP cube doesn’t help (and it gets stuck in the exact same place each time)
• Problem Verification
  • SQL Script that identifies that there IS a problem
• Problem Solution (WARNING: Contact MS PSS before running, since it requires making direct changes against the TFS database!)
  • SQL Script that identifies that resolves the problem

Troubleshooting Steps

The troubleshooting steps are included to walk through much of the thought process I ran through to identify the problem, and to help folks who may be running into a SIMILAR problem, but one that is not exactly the same. Feel free to go to the bottom of the post for the verification and fix steps.

First, prove that this problem occurred immediately upon install of SP1.

Here’s the strange thing: I can process all data, including all changes to the Work Item store, up until the time that SP1 was installed. So, I can only get data until that point. This shows two columns. First is the number of work items, which becomes frozen as of Sept 22 (the date of the SP1 install), however, you can see that the DATE dimension in the cube goes beyond that. That’s because everything ELSE is processing, just not the work items. Second column shows the work item revision count and it stops as of Sept 22 (the date of the SP1 install).

Second, I examined the logs for the TFS 2010 SP1 install and CU1 install. Everything was clear with the exception of one [WARNING] in each of the installs:

Warning from SP1 install:

[Warning@04:15:02.495] [2011-09-23 02:06:36Z][Warning] Servicing step data still contains "$$" after token replacement. This likely means there are undefined tokens referenced in the step data: <ArrayOfServiceDefinition><ServiceDefinition serviceType="TestResultsServiceEx" displayName="Test Results Service Ex" identifier="7826fd15-ef51-42a4-83c5-f7c659d5a835" description="$$TEST_MANAGEMENT_TEST_RESULTS_SERVICE_DESCRIPTION$$" toolId="TestManagement" relativeToSetting="0" relativePath="/TestManagement/v1.0/TestResultsEx.asmx" /></ArrayOfServiceDefinition>  

[Info   @04:15:02.495] [2011-09-23 02:06:35Z] Servicing step Upgrade Version Control database passed. (ServicingOperation: Tfs2010SP1KB2580221Install; Step group: Tfs2010SP1VersionControlInstall)
[Info   @04:15:02.495] [2011-09-23 02:06:36Z] Executing servicing step Tfs2010SP1KB2527572Install SQL servicing step. (ServicingOperation: Tfs2010SP1KB2580221Install; Step group: Tfs2010SP1KB2527572Install)
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Performer: Framework
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Type:      ExecuteSql
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Data Text: <SqlStepData category="Framework" resource="KB2527572CollectionInstall.sql" />
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Data: DatabaseCategory => Framework
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Data: SqlResource => KB2527572CollectionInstall.sql
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Data: BatchTimeout => -1
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Data: Parameters =>
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Data: RequiresExclusiveDatabaseLock => False
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Executing KB2527572CollectionInstall.sql with zero sql parameters.
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Data source: vstssql
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Database name: Tfs_DefaultCollection
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] SQL Info: Creating Procedure prc_Shelve
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] SQL Info: Creating Procedure prc_CreateWorkspace
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] SQL Info: Creating Procedure prc_PendAdd
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] SQL Info: Warning! The maximum key length is 900 bytes. The index 'PK__#73EC3F1__00FD296475D4878D' has maximum length of 1040 bytes. For some combination of large values, the insert/update operation will fail.
Warning! The maximum key length is 900 bytes. The index 'PK__#77BCCFF__3ECB18DF79A51871' has maximum length of 1041 bytes. For some combination of large values, the insert/update operation will fail.
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] SQL Info: Creating Procedure prc_iCheckLockStatus
[Info   @04:15:02.495] [2011-09-23 02:06:36Z] Servicing step Tfs2010SP1KB2527572Install SQL servicing step passed. (ServicingOperation: Tfs2010SP1KB2580221Install; Step group: Tfs2010SP1KB2527572Install)
[Warning@04:15:02.495] [2011-09-23 02:06:36Z][Warning] Servicing step data still contains "$$" after token replacement. This likely means there are undefined tokens referenced in the step data: <ArrayOfServiceDefinition><ServiceDefinition serviceType="TestResultsServiceEx" displayName="Test Results Service Ex" identifier="7826fd15-ef51-42a4-83c5-f7c659d5a835" description="$$TEST_MANAGEMENT_TEST_RESULTS_SERVICE_DESCRIPTION$$" toolId="TestManagement" relativeToSetting="0" relativePath="/TestManagement/v1.0/TestResultsEx.asmx" /></ArrayOfServiceDefinition>
[Info   @04:15:02.495] [2011-09-23 02:06:36Z] Executing servicing step Add SP1 Test Management Location Data. (ServicingOperation: Tfs2010SP1KB2580221Install; Step group: Tfs2010SP1TeamTestInstall)
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Performer: Framework
[Info   @04:15:02.495] [2011-09-23 02:06:36Z][Informational] Step Type:      AddLocationData

Figure: Surrounding information in upgrade log (click View Source above)

Warning from the CU1 install (very similar to above):

[Warning@04:22:22.214] [2011-09-23 00:44:15Z][Warning] Servicing step data still contains "$$" after token replacement. This likely means there are undefined tokens referenced in the step data: <ArrayOfServiceDefinition><ServiceDefinition serviceType="TestResultsServiceEx" displayName="Test Results Service Ex" identifier="7826fd15-ef51-42a4-83c5-f7c659d5a835" description="$$TEST_MANAGEMENT_TEST_RESULTS_SERVICE_DESCRIPTION$$" toolId="TestManagement" relativeToSetting="0" relativePath="/TestManagement/v1.0/TestResultsEx.asmx" /></ArrayOfServiceDefinition>

[Info   @04:22:22.214] [2011-09-23 00:44:14Z] Executing servicing step Patch Collection with PS Sync. (ServicingOperation: Tfs2010SP1Install; Step group: Tfs2010SP1ProjectServerInstall)
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Performer: Framework
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Type:      ExecuteSql
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Data Text: <SqlStepData category="Framework" resource="Tfs2010SP1ProjectServerCollectionInstall.sql" requiresExclusiveDatabaseLock="false" />
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Data: DatabaseCategory => Framework
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Data: SqlResource => Tfs2010SP1ProjectServerCollectionInstall.sql
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Data: BatchTimeout => -1
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Data: Parameters =>
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Step Data: RequiresExclusiveDatabaseLock => False
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Executing Tfs2010SP1ProjectServerCollectionInstall.sql with zero sql parameters.
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Data source: vstssql
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] Database name: Tfs_DefaultCollection
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] SQL Info: Creating Procedure prc_sync_getmapping
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] SQL Info: Creating Procedure prc_sync_getmappingchanged
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] SQL Info: Creating Procedure prc_sync_getproperties
[Info   @04:22:22.214] [2011-09-23 00:44:14Z][Informational] SQL Info: Creating Procedure prc_sync_getproperty
[Info   @04:22:22.214] [2011-09-23 00:44:15Z][Informational] SQL Info: Creating Procedure prc_sync_savemapping
[Info   @04:22:22.214] [2011-09-23 00:44:15Z][Informational] SQL Info: Creating Procedure prc_sync_saveproperty
[Info   @04:22:22.214] [2011-09-23 00:44:15Z] Servicing step Patch Collection with PS Sync passed. (ServicingOperation: Tfs2010SP1Install; Step group: Tfs2010SP1ProjectServerInstall)
[Warning@04:22:22.214] [2011-09-23 00:44:15Z][Warning] Servicing step data still contains "$$" after token replacement. This likely means there are undefined tokens referenced in the step data: <ArrayOfServiceDefinition><ServiceDefinition serviceType="TestResultsServiceEx" displayName="Test Results Service Ex" identifier="7826fd15-ef51-42a4-83c5-f7c659d5a835" description="$$TEST_MANAGEMENT_TEST_RESULTS_SERVICE_DESCRIPTION$$" toolId="TestManagement" relativeToSetting="0" relativePath="/TestManagement/v1.0/TestResultsEx.asmx" /></ArrayOfServiceDefinition>
[Info   @04:22:22.214] [2011-09-23 00:44:15Z] Executing servicing step Add SP1 Test Management Location Data. (ServicingOperation: Tfs2010SP1Install; Step group: Tfs2010SP1TeamTestInstall)
[Info   @04:22:22.214] [2011-09-23 00:44:15Z][Informational] Step Performer: Framework

Figure: Surrounding information in upgrade log (click View Source above)

Third, here are some of the Application Event Log entries that may indicate an SP1 install failure. (Ended up NOT to be the case.)

The first is a common error each time TfsAgent throws information, warning, or error messages. You can ignore most of the content of this error log, since it happened when I’d stopped the services anyway. The only critical thing is the highlighted part. This leads me to believe there is something wrong with the SP1 install. (Google / Bing have not been helpful in this scenario, even though there were plenty of hits.)

The description for Event ID 3300 from source TFS Services cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

Log Name:      Application
Source:        TFS Services
Date:          10/8/2011 8:59:42 PM
Event ID:      3300
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      VSTS01.myCo.com
Description:
The description for Event ID 3300 from source TFS Services cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 10/9/2011 3:59:42 AM
Machine: VSTS01
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
  Process Name: TFSJobAgent
  Process Id: 7012
  Thread Id: 1208
  Account name: MyCo\TfsService

Detailed Message: TF221107: Reporting for Team Foundation Server cannot execute job Work Item Tracking Warehouse Sync for team project collection DefaultCollection because the warehouse is offline. Use the Team Foundation Administration Console to start reporting.

the message resource is present but the message is not found in the string/message table

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="TFS Services" />
    <EventID Qualifiers="0">3300</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-10-09T03:59:42.000000000Z" />
    <EventRecordID>399224</EventRecordID>
    <Channel>Application</Channel>
    <Computer>VSTS01.myCo.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 10/9/2011 3:59:42 AM
Machine: VSTS01
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
  Process Name: TFSJobAgent
  Process Id: 7012
  Thread Id: 1208
  Account name: MyCo\TfsService

Detailed Message: TF221107: Reporting for Team Foundation Server cannot execute job Work Item Tracking Warehouse Sync for team project collection DefaultCollection because the warehouse is offline. Use the Team Foundation Administration Console to start reporting.</Data>
  </EventData>
  </Event>

Figure: Full Text of Warning (click View Source above)

Fourth, an Error received when the Work Item Tracking Warehouse Sync runs. This one matters, since it’s the one that represents the failure. And when it fails, it will NOT stop the TFS Job from running, it will simply hang the job forever. Interesting thing, though, since the warehouse IS updated to the day that the SP1 was installed, so the cube can get the data up to that point. Notice that the highlighted areas show that it is a SQL Transport level problem. (Quick reminder to everyone that ALL the other Warehouse jobs process successfully, as do the Cube jobs.) It appears as though a network name is no longer available. (Again, recall that the Work Item job can process up to the date that the TFS 2010 SP1 was installed, so it’s not failing immediately, only after processing all the work item events up to that point – we’ve repeatedly rebuilt the warehouse and cube from scratch, and each time it can process up to the date of the SP1 install.)

Detailed Message: TF221122: An error occurred running job Work Item Tracking Warehouse Sync for team project collection or Team Foundation server DefaultCollection.
Exception Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) (type SqlCannotOpenDatabaseException)

Log Name:      Application
Source:        TFS Services
Date:          10/8/2011 8:27:21 PM
Event ID:      3305
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      VSTS01.myCo.com
Description:
The description for Event ID 3305 from source TFS Services cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 10/9/2011 3:27:21 AM
Machine: VSTS01
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
  Process Name: TFSJobAgent
  Process Id: 1696
  Thread Id: 1908
  Account name: MyCo\TfsService

Detailed Message: TF221122: An error occurred running job Work Item Tracking Warehouse Sync for team project collection or Team Foundation server DefaultCollection.
Exception Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) (type SqlCannotOpenDatabaseException)

Exception Stack Trace:    at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.TranslateException(SqlException sqlException)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(SqlException ex)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchQuery(IRequestContext context, String sqlBatch, List`1 parameterList)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchInternal(IRequestContext context)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayerImpl.GetWarehouseData(String logicalTrackingId, Int32 batchSize, String fieldLogicalTrackingId, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload warehousePayload)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.TryGetNextWorkItemRevisionsBatch(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload& workItemRevisionBatch)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.GetWorkItemRevisionBatchAndPopulate(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, WarehouseMetadata allTpcWarehouseMetadata, String& dimensionFieldQueryList, String& measureFieldQueryList, String& maxRevisionWatermark)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.SynchronizeNewWorkItems(WarehouseMetadata allTpcWarehouseMetadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.MakeDataChanges()
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)

Inner Exception Details:

Exception Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) (type SqlException)
SQL Exception Class: 20
SQL Exception Number: 64
SQL Exception Procedure:
SQL Exception Line Number: 0
SQL Exception Server: vstssql
SQL Exception State: 0
SQL Error(s):

Exception Data Dictionary:
HelpLink.ProdName = Microsoft SQL Server
HelpLink.EvtSrc = MSSQLServer
HelpLink.EvtID = 64
HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink
HelpLink.LinkId = 20476

Exception Stack Trace:    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadByte()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)

the message resource is present but the message is not found in the string/message table

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="TFS Services" />
    <EventID Qualifiers="0">3305</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-10-09T03:27:21.000000000Z" />
    <EventRecordID>399205</EventRecordID>
    <Channel>Application</Channel>
    <Computer>VSTS01.myCo.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 10/9/2011 3:27:21 AM
Machine: VSTS01
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
  Process Name: TFSJobAgent
  Process Id: 1696
  Thread Id: 1908
  Account name: MyCo\TfsService

Detailed Message: TF221122: An error occurred running job Work Item Tracking Warehouse Sync for team project collection or Team Foundation server DefaultCollection.
Exception Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) (type SqlCannotOpenDatabaseException)

Exception Stack Trace:    at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.TranslateException(SqlException sqlException)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(SqlException ex)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean&amp; errorOnBulkUpdate)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchQuery(IRequestContext context, String sqlBatch, List`1 parameterList)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchInternal(IRequestContext context)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayerImpl.GetWarehouseData(String logicalTrackingId, Int32 batchSize, String fieldLogicalTrackingId, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload warehousePayload)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.TryGetNextWorkItemRevisionsBatch(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload&amp; workItemRevisionBatch)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.GetWorkItemRevisionBatchAndPopulate(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, WarehouseMetadata allTpcWarehouseMetadata, String&amp; dimensionFieldQueryList, String&amp; measureFieldQueryList, String&amp; maxRevisionWatermark)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.SynchronizeNewWorkItems(WarehouseMetadata allTpcWarehouseMetadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.MakeDataChanges()
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String&amp; resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String&amp; resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String&amp; resultMessage)

Inner Exception Details:

Exception Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) (type SqlException)
SQL Exception Class: 20
SQL Exception Number: 64
SQL Exception Procedure:
SQL Exception Line Number: 0
SQL Exception Server: vstssql
SQL Exception State: 0
SQL Error(s):

Exception Data Dictionary:
HelpLink.ProdName = Microsoft SQL Server
HelpLink.EvtSrc = MSSQLServer
HelpLink.EvtID = 64
HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink
HelpLink.LinkId = 20476

Exception Stack Trace:    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadByte()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)

</Data>
  </EventData>
</Event>

Figure: Entire Error message pertaining directly to the problem (click View Source)

Next is a related WARNING in the Application Event Log. It occurs just prior to the ERROR noted above. Notice that it states that a severe error has occurred and the results should not be trusted. It also mentions that the action was cancelled by the user. No HUMAN cancelled it, however, the TFS Job agent may have cancelled it.

Detailed Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user.
Exception Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user. (type SqlExistingConnectionForciblyClosedException)

Log Name:      Application
Source:        TFS Services
Date:          10/8/2011 10:51:03 PM
Event ID:      3200
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      VSTS01.myCo.com
Description:
The description for Event ID 3200 from source TFS Services cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 10/9/2011 5:51:03 AM
Machine: VSTS01
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
  Process Name: TFSJobAgent
  Process Id: 4740
  Thread Id: 2492
  Account name: MyCo\TfsService

Detailed Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user.
Exception Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user. (type SqlExistingConnectionForciblyClosedException)

Exception Stack Trace:    at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.TranslateException(SqlException sqlException)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchQuery(IRequestContext context, String sqlBatch, List`1 parameterList)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchInternal(IRequestContext context)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayerImpl.GetWarehouseData(String logicalTrackingId, Int32 batchSize, String fieldLogicalTrackingId, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload warehousePayload)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.TryGetNextWorkItemRevisionsBatch(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload& workItemRevisionBatch)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.GetWorkItemRevisionBatchAndPopulate(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, WarehouseMetadata allTpcWarehouseMetadata, String& dimensionFieldQueryList, String& measureFieldQueryList, String& maxRevisionWatermark)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.SynchronizeNewWorkItems(WarehouseMetadata allTpcWarehouseMetadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.MakeDataChanges()
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)

Inner Exception Details:

Exception Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user. (type SqlException)
SQL Exception Class: 11
SQL Exception Number: 0
SQL Exception Procedure:
SQL Exception Line Number: 0
SQL Exception Server: vstssql
SQL Exception State: 0
SQL Error(s):
SQL Error[1]: System.Data.SqlClient.SqlError: Operation cancelled by user.
    Class: 11
    Number: 0
    Server: vstssql
    Source: .Net SqlClient Data Provider
    State: 0
    Procedure:
    Line Number: 0

Exception Data Dictionary:
HelpLink.ProdName = Microsoft SQL Server
HelpLink.ProdVer = 10.00.5500
HelpLink.EvtSrc = MSSQLServer
HelpLink.EvtID = 0
HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink
HelpLink.LinkId = 20476

Exception Stack Trace:    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.HasMoreRows()
   at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.PayloadTable.Populate(SqlDataReader reader)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.PayloadTableCollection.Populate(SqlDataReader reader)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate)

the message resource is present but the message is not found in the string/message table

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="TFS Services" />
    <EventID Qualifiers="0">3200</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-10-09T05:51:03.000000000Z" />
    <EventRecordID>399296</EventRecordID>
    <Channel>Application</Channel>
    <Computer>VSTS01.myCo.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 10/9/2011 5:51:03 AM
Machine: VSTS01
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
  Process Name: TFSJobAgent
  Process Id: 4740
  Thread Id: 2492
  Account name: MyCo\TfsService

Detailed Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user.
Exception Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user. (type SqlExistingConnectionForciblyClosedException)

Exception Stack Trace:    at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.TranslateException(SqlException sqlException)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean&amp; errorOnBulkUpdate)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchQuery(IRequestContext context, String sqlBatch, List`1 parameterList)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchInternal(IRequestContext context)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayerImpl.GetWarehouseData(String logicalTrackingId, Int32 batchSize, String fieldLogicalTrackingId, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload warehousePayload)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.TryGetNextWorkItemRevisionsBatch(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, String dimensionFieldQueryList, String measureFieldQueryList, Nullable`1 fieldId, Payload&amp; workItemRevisionBatch)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.GetWorkItemRevisionBatchAndPopulate(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, WarehouseMetadata allTpcWarehouseMetadata, String&amp; dimensionFieldQueryList, String&amp; measureFieldQueryList, String&amp; maxRevisionWatermark)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.SynchronizeNewWorkItems(WarehouseMetadata allTpcWarehouseMetadata)
   at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.MakeDataChanges()
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String&amp; resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String&amp; resultMessage)
   at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String&amp; resultMessage)

Inner Exception Details:

Exception Message: A severe error occurred on the current command.  The results, if any, should be discarded.
Operation cancelled by user. (type SqlException)
SQL Exception Class: 11
SQL Exception Number: 0
SQL Exception Procedure:
SQL Exception Line Number: 0
SQL Exception Server: vstssql
SQL Exception State: 0
SQL Error(s):
SQL Error[1]: System.Data.SqlClient.SqlError: Operation cancelled by user.
    Class: 11
    Number: 0
    Server: vstssql
    Source: .Net SqlClient Data Provider
    State: 0
    Procedure:
    Line Number: 0

Exception Data Dictionary:
HelpLink.ProdName = Microsoft SQL Server
HelpLink.ProdVer = 10.00.5500
HelpLink.EvtSrc = MSSQLServer
HelpLink.EvtID = 0
HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink
HelpLink.LinkId = 20476

Exception Stack Trace:    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.HasMoreRows()
   at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.PayloadTable.Populate(SqlDataReader reader)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.PayloadTableCollection.Populate(SqlDataReader reader)
   at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean&amp; errorOnBulkUpdate)

</Data>
  </EventData>
</Event>

Figure: Full Text of Warning (click View Source above)

Fifth, a Google / Bing search reveals only ONE exactly related hit for TF221122: “An error occurred running job Work Item Tracking Warehouse” “A severe error occurred on the current command”

http://www.go4answers.com/Example/tfs-2010-warehouse-update-severe-148743.aspx – Exactly the same problem, and points to a ROGUE work item that was corrupted during the upgrade from TFS 2008 to TFS 2010. We were good till the update to SP1, so maybe we got corrupted work items there…

Using the SQL at http://blog.mohammadjalloul.com/blogs/mo/archive/2011/08/29/how-to-detect-corrupted-tfs-work-items.aspx I searched for any corrupted work items. No luck. Query returned no corrupted work items.

So I queried for every work item that was changed on Sept 22 (the date of the last work item update to the warehouse) and I manually reviewed every one of the work items changed that day. I could not find any that had an error that would not permit me to resave them back to TFS, or any other indication anything was wrong.

At this point in the exploration, I did not believe that work item corruption was the problem, or at least the corruption was very hard to spot.

Sixth, Steven St Jean ran into a similar problem here: http://sstjean.blogspot.com/2010/07/tfs-2010-rtm-throws-tf221122-error-when.html

My situation is different since Analysis Services is already running under a domain account. I could change to NETWORK SERVICE to see if that would help, however, the problem is still different. He had an Analysis Services Network Service account that was failing REPEATABLY, and changing to a domain account worked. And in his case, the ENTIRE processing of the Warehouse was failing. For me, it’s ONLY the work items.

Seventh, reboot multiple times. Declared this weekend to be open maintenance, and rebooted multiple times after trying various things. No change to the behavior.

Eighth, delete and rebuild Tfs_Warehouse and Tfs_Analysis multiple times. In each case it will process the work items up to the date and time the SP1 was installed. (I can show that the last work item change prior to SP1 makes it into the warehouse/cube, but that the very next work item change, done after the SP1 was installed, does NOT make it into the warehouse/cube.)

Ninth, work directly with some of the people at Microsoft DevDiv who built the product. (Ah, the advantages of being an MVP.) This step led to the solution.

Problem Verification

Once we understood the problem more deeply, this script was run to prove that there was an issue between the two particular tables.  It verifies that there is an incorrect change order between the WorkItemsLatest and WorkItemsAre tables.  If this SQL query returns any rows, then you have the problem.  If not rows are returned, look elsewhere, because the fix script won’t help you.

SELECT COUNT(*)
FROM [dbo].[WorkItemsLatest] L
JOIN [dbo].[WorkItemsAre] A
ON L.[ID] = A.[ID]
AND L.[Rev] = A.[Rev]
WHERE L.[Changed Order] <> A.[Changed Order]

Problem Solution

Below is the SQL Script that solves the problem by setting the valies of the Changed Order columns in the WorkItemsLatest table to the values currently in the WorkItemsAre table.

WARNING: RUN THIS AT YOUR OWN RISK!  I HIGHLY recommend you contact Microsoft PSS, and work with them through this issue.  This SQL script is provided “for entertainment purposes only”…

SET WL.[Changed Order] = WA.[Changed Order]
FROM dbo.WorkItemsLatest AS WL
JOIN dbo.WorkItemsAre AS WA
ON WL.ID = WA.ID
WHERE WL.[Changed Order] <> WA.[Changed Order]

Conclusion

I’ve done a large number of TFS 2010 SP1 migrations, and this is the first time I’ve run into this problem.  So, don’t let it freak you out.  However, if you run into this problem, get PSS on the phone right away, and get this problem solved!  (As a side note, I don’t believe you’ll be charged for the call, since it does appear to be a bug in the SP1 upgrade process.)

I love this about kanban – a simply, nearly trivial, change to the board and the kanban is now telling me a new story, and helping me visualize my work in a new, enlightening way. Read on for the details!

Time for a New Change

I took my original personal kanban board design from a pattern described on their website, http://personalkanban.com.  It had four basic states, Backlog, To Do, Today, and Done.  After about two weeks, I updated it to include date-specific columns for the upcoming week. Then I let it percolate.  It’s been unchanged for the last several months…  until last week.

Background:
A month ago one of our customers was struggling mightily with their Kanban implementation, and asked us to conduct an assessment. After asking a rather lengthy chain of ‘whys’, we identified a fairly substantial problem with the way their prioritization was conducted.  Their backlog was visible to the development team, and was ordered in priority order. (GOOD) However, after some observation (and questioning) it became clear that new work was nearly always urgently prioritized at the very top, and the remainder got pushed down. (BAD)  In other words, there backlog behaved more like a stack than a queue.  Now, responding to change is an important agile principle, yet when all new items are consistently placed right at the top, it makes little sense to do any forward planning if the only items that ever get worked on are the latest ones.  (This bears a far deeper discussion, but in a later post.)

Last week, while frustrated about my inability to accomplish my weekly goals, I decided to modify my personal kanban to track a new form of information – urgent tasks that appeared throughout the day vs. planned tasks which I had prioritized and planned earlier. 

In other words, I wanted to track how much of my week was Stack vs Queue.

Visualizing Planned vs Unplanned Work

So I modified my process to use 3 different color ‘stickies’ to track my work.

• RED : Items that go directly into my Today column. 
  • These items most likely pre-empting planned work, are considered Work In Process (WIP) and are given a start date.
• YELLOW : Items that go into my To Do column.
  • These state is my first Work In Process (WIP) state, and once an item enters this state it is assigned a Start date.
• GREEN : Items to go into my Backlog column. 
  • These items haven’t yet entered WIP, and do not have associated Start dates.

First week results

As you can see, my initial gut was correct, and the majority of work I did each week simply “came up” and had to be done urgently. Now, this is not necessarily a bad thing. Much of my day to day work (when I’m not working with a client) is being called into sales calls, creating proposals, working through deep technical problems with others on my team, and other very important work that is nearly impossible to plan for ahead of time. 

(Quick note: Some of the items you see in the backlog are ‘legacy’ yellow stickies.)

Next Steps

Despite the value of the urgent, unplanned work, I still have concerns. There is an awful lot of red in my Done column, and precious little green. And that seems to wrong.  My gut feel is that I’ll need to look into ways to fit more strategic (green) items into my weekly work.

I love how a simple change to my kanban can tell me a new story, one I haven’t yet decided how I’ll respond to. But, I’m using Plan-Do-Check-Act (PDCA) to improve, I’ll be able to have a baseline now for comparison during the Check phase!

Installing TFS 2010 SP1 will solve the problem. (Also, please install TFS 2010 SP1 CU1 to fix the issues introduced by SP1.)

After running windows update on your TFS App Tier and your Team Build machines, automated builds will begin to fail or hang.  Looking into the event log will show the error below.  (Also, see notes below for other ways to test if you have the problem.)

The exception text reads something like:

Exception Message: Could not load file or assembly ‘System.Runtime.DurableInstancing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35′ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0×80131040) (type BuildServerException)

Exception Stack Trace:    at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ProcessHttpResponse(HttpWebResponse response, Stream responseStream, WebException webException, XmlReader& xmlResponseReader)
   at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ExecWebServiceRequest(HttpWebRequest request, XmlWriter requestXml, String methodName, HttpWebResponse& response)
   at Microsoft.TeamFoundation.Build.Client.AdministrationWebService.TestBuildControllerConnection(Uri controllerUri)
   at Microsoft.TeamFoundation.Build.Client.BuildServer.TestConnectionForBuildController(IBuildController controller)
   at Microsoft.TeamFoundation.Build.Machine.ControllerService.TestService()

Unfortunately, short of rolling back the Windows Update, you’re going to have to either install TFS 2010 SP1 immediately, or work some version re-direction magic in the GAC.  This means you’re likely installing SP1 under duress, but there are very limited options.

Don’t forget to install VS 2010 SP1 as well if you have Visual Studio 2010 or Team Explorer 2010 on any of your TFS 2010 or Team Build servers.

You can find downloads here:

• TFS 2010 SP1 (KB2182621) : http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=20506
• VS 2010 SP1 Web Installer (KB983509) : http://www.microsoft.com/download/en/details.aspx?id=23691
• TFS 2010 SP1 Cumulative Update 1 (KB2580221) : http://www.microsoft.com/download/en/details.aspx?id=26211

Your Team Builds will begin running immediately upon installing the TFS 2010 SP1 (and possibly a reboot), however, I really recommend getting everything up to the same versions while your in forced maintenance anyway.

1) Open up the TFS Administration Console, Select Build Configuration and click Properties on any of the Agents.

2) In the Build Agent Properties window, click the Test Connection button.  If you get the System.Runtime.DurableInstancing error, you have the problem and will need to install TFS 2010 SP1 to correct it.

Good luck!

I’m struggling with a concept from Toyota Kata, by Mike Rother. The way I’m reading it, teams of cross-functional individuals hide dysfunctions and can slow process improvement efforts. Or, to be precise, allowing cross-functional team members to assist others hides a potential improvement opportunity by limiting the pain of not having work balanced appropriately (heijunka).

Let me explain.

The Arguments from Toyota Kata

1×1 Flow

In Chapter 5, Mike talks about 1×1 flow (AKA single piece flow), whose purpose is to reveal dysfunctions or problems in the current process. He contrasts two manufacturing teams to illustrate.

In Example 1, Figure 5-6 in his book, the people are all working at their assembly stations (and there is some slack in the system), and when one person gets overwhelmed with work, a downstream employee who is starved for work will move to assist, thus increasing overall throughput. Now, this process is flexible, the employees can work around problems, and they can absorb any variation in the processes.

In Example 2, Figure 5-7 in his book, the people have no slack, and are matched to each. Now, if an operator experiences a problem, throughput immediately suffers, since there is no slack, and any problems, including issues such as variation in the process length, are apparent, and can be handled by process improvement efforts.

Analysis

The first process has a lot of flexibility, and allows the team to make production. However, “…at Toyota this sort of flexibility is considered negative, since problems go unresolved and the process gets into a non-improving, firefighting cycle.”

He then specifically addresses cross-functional teamwork as an issue. “To compensate for this fluctuation, the downstream operators naturally walk from one value stream to assist in another, rather than idly waiting… Of course, this workaround is not a process improvement, and although it is done with good intention, it introduces even more fluctuation into the value streams.” And, even more strongly worded: “At Toyota, such self-compensating flexibility in processes would strike fear in the hearts of managers… Such a mode of operation would not be allowed, and would be viewed as a failure to manage the process.”

Pull Systems (Kanban)

Chapter 5 also addresses pull systems such as Kanban. In Figure 5-13, he shows a Supplying Department with numerous machines, each of which can produce items for a machine in the Customer Department (the next step in the process). The important thing to note is that every machine on the left can produce an item for every machine on the right. In other words, these machines are purely “cross-functional” and can “fill in” for any other machine in the same department.

Next, the author recommends putting a kanban system between them. He states we need two pieces of information: 1) Where the parts associated with a kanban card are located, and 2) On what machines the parts associated with the kanban card should be produced. Number 1 is pretty clear – we need to be able to go find the pieces. But number 2 is challenging. Mike states we should specify which machine actually produces the part we need, since this “helps us to see what kanban in really about at Toyota.”

Analysis

He continues, and I’ll quote at length, “How do you think the supervisor will respond if we ask him to define what parts run on what machine? The supervisor is likely to object to someone taking away his flexibility to run the parts on whatever machine is available. Perhaps he will say something like, ‘If we are going to define what parts will run on what machine, and thereby reduce my ability to run items on any machine, then we better start improving the reliability of those machines.’ And so kanban has already started working for us.” This is illustrated in Figure 5-14, where the second pattern makes it “Easier-to-understand causes of problems”

Finally (last set of quotes, I promise): “If your purpose is ‘make production’ then the flexible system looks good because despite the existence of problems you can work around them and still make the numbers.” and “Flexible systems that autonomously bypass problems are by their nature nonimproving.”

My thoughts

This flies in the face of so much I’ve thought in the past that the past few days I haven’t been able to get it out of my mind. I’ve been going back and forth with Hakan Forss and Alan Shalloway for a while, but the 140 character limits are too constraining for actually fleshing out a thought.

Then, while doing some test process consulting with a local client, I ran across an example.

The Scrum team has a total of 7 people on the team, and are a cross functional (in the sense they have everyone necessary to build product) and the developers on the team are cross functional (in the sense they can work on tester tasks). The team has been successfully delivering value at the end of every Sprint. By all external Scrum measures they look good. However, in their retrospectives, they’ve been frustrated by the amount of testing-related work that the developer-focused team members are doing. Not surprisingly, they’ve been told that good Scrum teams work with each other to ensure delivery of working software at the end of the Sprint.

The team believed it was “under committing” to what could be delivered in a Sprint due to severe constraints in the capacity of team members focused on testing. Now, if that’s all the tested features they could deliver in a Sprint, then they were committing correctly. But we have a problem. Since the coding folks were helping out on the testing tasks, they couldn’t code as productively as they would have liked.

Question: “What if the developers couldn’t work on the tester’s tasks any longer” (ala Toyota Kata).

Breakthrough! We restructured the development process to allow each team to focus on their core competencies, while they will still be able to deliver tested, high quality code. We used a target condition, from Toyota Kata, to stimulate the creativity and come up with solutions.

So, in my first use of the “cross functional individuals can hide dysfunctions” thought pattern, I believe we made serious headway in a difficult problem. It provided a very useful lens.

Do we give up on Cross Functional Individuals?

No.  I don’t believe so. I believe it to be a very useful thought pattern, but software development is radically different from manufacturing.

• Variation in Software is High – Extreme variation exists among seemingly equivalently sized tasks. Task time variation of hundreds or thousands of percent is common in software development. This means that if we have a “balanced line” (heijunka) on AVERAGE, we will be out of balance the vast majority of the time. To put it in Theory of Constraints terms, the constraint would whipsaw around the team unpredictably. Without at least some “load balancing” we might identify a whole host of problems, but nearly all of them would be variation-related. Trying to solve for variation in software development is a fool’s errand, and would block discovery of more substantial issues.
• Strong Teams Perform Better – High performing teams are generally made up of people who assist one another and form strong personal bonds. Working together on a difficult task can bring two people together in a way that working separately (even if cooperatively) can. This one is a bit fuzzy for me, since strong teams are surely created in Toyota; but I still have a gut feel that teams in which people are willing to bail each other out creates more positive improvement than can otherwise be gained with a strict load balancing effort.

My bottom line – I will continue to encourage teams to work cross-functionally and assist one another, but will periodically use the “cross functional teams and individuals can hide dysfunctions” thought pattern for process improvement.

What am I missing?

Addendum:

In doing some research for this post, I re-read the new Scrum Guide (from Scrum.org), and noticed that there is no mention of individuals on a team acting cross-functionally. In fact, it only states “Cross-functional teams have all competencies needed to accomplish the work without depending on others not part of the team.” Thus, it appears the team could , in formal Scrum, be made up of two testers, two developers, one SQL DBA and one build engineer, all of whom only focus on their own core competencies and do act as “cross-functional individuals” to assist one another. (I’m NOT saying this is a good idea, just saying it doesn’t appear to violate Scrum, which surprised me greatly.)

Did Ken Schwaber see this as a possible dysfunction already?

I’m doing some research for a newsletter article and blog post and am trying to understand cross-functional teams in Scrum.  From my prior understanding, one of the defining factors of cross-functional teams is that they are made up of “cross-functional” individuals, who can work on different types of problems.  For instance, in Scrum everyone is called a Developer if they are on the Team.  This is true whether or not they have deep skills in actual coding.  (reference: latest Scrum Guide on Scrum.org).  The benefit being that if the testing effort is threatening the sprint, then the people with software development skills can work with the team members with stronger test skills to complete the sprint on time.  This benefit is very clear and has been noted by multiple people, such as Don Reinertsen (Principles of Product Development Flow) and many, many others.  I even seem to remember learning about that benefit from my very first Scrum Master course. 

However, it appears I’ve made an assumption that is not necessarily true of Scrum, and I can’t find good references in the canonical Scrum literature (new versions) telling me that cross-functional individuals are encouraged (demanded?) as part of Scrum.  The Scrum Guide defines a cross functional team as “Cross-functional teams have all competencies needed to accomplish the work without depending on others not part of the team.”  I can find nowhere in the guide that states that Scrum teams should be able to work in areas that are strengths of other team members.

So, the question again: Can a Scrum team be made up of two testers, two developers, one SQL DBA and a build engineer, each of whom ONLY do their own tasks and work as separate silos within the Team?

Note that I’m not asking if it’s a good idea, but whether it is OK in formal Scrum.  It has, in the past, seemed to me that it not a good idea.  However, I will argue in the upcoming newsletter article that cross-functional individuals on cross-functional teams may be hiding dysfunctions and slowing process improvement efforts.  (Based on some thoughts in  Toyota Kata, by Mike Rother.)

Thoughts?

Scrum, the grand-daddy of Agile practices vs Kanban the challenger.  Come see Martin Hinshelwood (ALM MVP and Scrum proponent) and Steven Borg (ALM MVP and Kanban fan) will go head to head, mano-a-mano, wit against wit, to persuade you which is the correct approach for your agile adoption!

Come see the benefits and weaknesses of both as Martin and I fight it out!  Plus, you’ll get a chance to cheer for your side and vote for the winner! 

More information is available at out Events page where you can find the follow on sessions diving deeper into both Scrum and Kanban.  Or you can register for the Smackdown directly by clicking this link: Smackdown! 

Photo Credit: THQInsider, Flickr.com

Tom DeMarco is a smart man, and I love the book “Slack”, so I hesitate to disagree.  However, I believe DeMarco is looking at the organization through a very particular lens, once which misses some critical aspects.  When I look through a lean product development lens, I see several times where internal competition can act as “Slack”.

Let’s use a thought experiment.  Recall that since DeMarco is claiming his statement is universal, only one counterexample is necessary. 

Software manufacturer MyCo has two very different, but both high risk, approaches to solving a customer problem.  Unfortunately for MyCo, it’s not at all clear up front which of the approaches will be successful technically and/or successful at delivering the most value to the customer.  However, they have determined that being early to market will be very profitable (in other words, their “cost of delay” is very high).  One way to solve the problem, while not relying on internal competition, is for the company to allocate their resources to one of the approaches, and wait for the other one.  If the first one works out, great (although it may not be as optimal as the other approach).

Instead, if MyCo decides to examine both approaches, they are setting up an internal competition.  They have also provided a time buffer (i.e., slack) by exploring both options simultaneously.  If only one approach is successful, then they will have the successful approach.  If both approaches are successful, they can select the best one.  And if neither one is successful, they have quickly eliminated this possibility and can move onto other things.

Whether or not it makes sense to set up this internal competition depends on the cost of exploring the options and the cost of delay.  If the cost of delay is high (alternatively stated, if the benefits of being early to market are high) relative to the cost of exploring the options, then both should be explored simultaneously.

Thanks to @aJimHolmes and @alshalloway on Twitter for bringing up this topic, as well as Donald Reinertsen for providing the particular “lens” to look through.

UPDATE:  This is a common issue with technocrats – people who believe that having a smart person in charge of things can improve results over a marketplace (or goods, ideas, or whatever).  Just like DeMarco believes that smart managers should eliminate internal competition as wasteful, smart politicians often believe they should eliminate competition in the external marketplace as wasteful.  It’s so intellectually appealing, but is only a vanity.  F.A. Hayek expresses these concepts far better than I when he discusses the limitations of knowledge in society in “The Fatal Conceit”.  Although he talks in terms of entire economies, I believe these thoughts apply (although to a lesser extent) in smaller societal units like organizations.

I get to work with a wonderful team!  I know a blog isn’t really a place to gush about how wonderful your coworkers are, but I’m going to.  So, to the Northwest Cadence team – Thank you!  You’re such a wonderful group of friends, and each and every one of you makes this team great.  I am so happy to work with you.  This is, by far, the best team I’ve ever worked with in my life. 

I can’t wait to see what’s in store for tomorrow, when it’s all fun and games!  (Please, no karaoke…)