Team Foundation Server Field Guide
(Building and Testing Software)
Team Foundation Server is a product that specializes in the building and testing of Windows software. TFS is a product that’s been around for quite some time and is considered an extremely important tool for software shops to utilize.
Build Agent Deployment
This is step 1. You need to install the agent on servers in order to tie them into TFS for deployment purposes. The build agent in TFS 2015+ is cross platform and also allows you download/install from the TFS web GUI. Once a build agent is configured it detects the build servers capabilities and installed packages. You are able to add capabilities and select agents from a pool via TFS.
Setup Build Agent
Go to Settings at the top right -> Default Collection -> Agent Queues
This allows you to download the agent in order to install on a build server. This downloads a .zip file of the agent. Extract the contents, if you wish to use multiple agents on the same server breakdown the file tree.
ie: C:\Build\Agent1, C:\Build\Agent2 etc.
Two scripts should be included. You need to run the ConfigureAgent script in order to configure the build agent. Preferred method is to open cmd as admin cd to the agent directory and execute the Configure Agent script (ConfigureAgent.cmd) this will ask you for values: Name, URL of TFS server, Configure Agent against agent pool (An Agent Pool is just a grouping of agents)
**To create an agent pool:
Go to Settings at the top right -> Default Collection -> Agent Queues -> Manage Pools
This allows you to create your own pools. Different software products is where this is applicable. Different products, different build servers. This function also grants you more control over who can deploy agents and where those agents can be deployed. Each pool allows you to add Administrators as well as Users via AD groups.**
The ConfigureAgent script then prompts for you to specify a working folder directory. This directory is used to store the build you’re testing from TFS temporarily. This ensures the build server can store and test the build simultaneously.
You can choose to install the agent as a server or from the cmd prompt. If you run from cmd prompt the server will show you diagnostic information when a build is pushed to this particular build server. Choosing to install as a service is a good option. You should also create a domain account in order for the build agent service to run under.
Once this step is completed you should see the build agent registered via the TFS web interface. You need to refresh the Agent Queue page in order to see this.
Agent Pool -> Capabilities:
During the install the agent populates the machine’s capabilities these include:
OS Version/Software/.Net Packages/Hardware Specs
From this tab you can also select your build server and add .dll files you can even add frameworks within this function.
You can create these via the browser. There is no need for VS anymore with TFS 2015+. You can setup multiple configurations in order to use multiple flavors. You can create a debug and release build within the same definition. The most valuable part of the new Build Definition system is history. This resolves errors with XAML in previous versions in TFS. You can now see who changes what within definitions. To access build definitions within your TFS instance just go to the BUILD tab within TFS. Click the Green + in order to create a new Build Definition here you have the following options:
In this example we’ll be utilizing the Visual Studio build definition. The next screen will ask you for a Repository. If you’ve created an Agent Pool as explained above you’ll see it populated within this list. Branches will also show here. Examples of these branches are: DEV/INT/QA/UAT/PROD. Agent queue should populate with the pool you’ve created as well. If you don’t wish to configure an agent pool at this time you can just use the default one for now. Upon clicking create you’ll see the definition was successfully created. You’re now presented with steps to complete once the build definition has been defined. You can just save and run in order to see what presents with a test. Click Save in order to save your definition.
Running a Build
Now that we have a build definition defined within TFS we can test the build process. The new build system in TFS 2015+ includes:
Real-time log view - This allows you to see what’s happening with the build as it’s ran and is incredibly useful for troubleshoot issues. Project-by-project breakdown allows you to dig into each specific build and check each step the build takes. The Build outputs allow you to see the same deployment process that you’d see locally on a server/workstation.
To start running the build go to Build Definitions and click the “Queue build…” button. This will allow you to select a queue to put the build in as well as select a different branch if applicable. Click OK and the build should kick off via the WEB GUI. The console will then present you with real-time functions/tasks as the build deploys. You’re presented with an overall completion % as well as each step on the left hand side of the GUI. If your build runs cleanly you’ll see a Succeeded status. If your console runs into any errors during this process the Build will state a Failed status.
If you click the Build field on the left side it’ll show you the following details: start, finish, definition, and branch.
Timeline Tab: Shows you how long each step took in order to ran and lists the worker name (build agent) that performed the task.
Artifacts Tab: Shows you build outputs. ie: new directories/files deployed. From here you can choose to browse the location on the server or download the directories/files in .zip format.
Tests Tab: Shows test cases if you’ve configured tests to run via the Visual Studio Test step in your build definition. If configured, this will show what tests Passed/Failed a total Pass percentage and the total run time of the tests. History is also a useful stat. This shows you if failures and duration are growing over time. Often this can be referred to in order to remove old legacy tests within the build.
Customizing a Build
Tasks – Pull more functionality into Build Definitions
Multi-platform – Allows you to build debug and release version of the application without creating additional build definitions
Triggers – Automated deployments or builds based on specific actions through a scheduled task or via pool request
Demands – Agent requirements in order for build definition to run
Editing Build Definitions
After creating a build is possible to go back into the definition and fine tune your solution. In order to do this proceed by going to the Build Tab in TFS Right-Click the build solution you want to edit and select the edit option. You can specify exactly what solution you want to run against by clicking NuGet Installer -> Path to Solution (…) This ensures the build definition only runs against the solution you specify. You should also specify the same solution in the Visual Studio Build -> Solutions (…) field. It’s also a good idea to check the clean build option under this step. This will ensure previous builds/scripts don’t complicate future builds. If you leave the Clean option unchecked it’ll keep source code on the server and continue to run against that code. You can also specify Visual Studio version should be used for the build(s).
It’s important that you also configure the Control Options within the build definition to suit your needs. You almost always want the Enabled option checked. Continuing on error is also useful. If something doesn’t run your build will simply continue processing your build until all steps are completed and the build either succeeds or fails. Always run means that even if previous steps fail in the build definition that the step you’ve selected always continues to run.
Make sure after setting these option you save on the build definition you’re editing. Otherwise no changes are made via TFS. Upon clicking Save you’ll be asked to enter a comment. This is extremely useful for auditing and tracking purposes.
The Visual Studio Test step is where you’ll specify test cases and configure automated tests to run in succession with a build. Be aware that if any cases specified here fail, your build will fail as well. In this step you almost always want the code coverage option enabled. This allows you to see what code is being run within you’re automated tests.
Customizing with Build Steps
You’re able to rearrange build steps within a definition to help your build process. To do this go to Build Definition -> Select Build -> Build Tab -> Drag and Drop to rearrange the order you wish steps to run in. You can also add steps using the “Add build step…” button. You can remove steps by clicking the “X” after hovering over the step you wish to remove. You’re able to add nearly any step within this pane.
ie: Scripts, Command Line, PowerShell, Azure Processes (Cloud Deployment)
Editing Variables/Multiple Build Configuration
Each step has variables that you can set values for. To find out which variable each step within a Build Definition is using just click that step and you’ll see variables represented by $Value. In order to edit these variables click the Variables tab under the definition you want to set for.
Say you want to build two different flavors of the same build and utilize the same definition for ease of use. (New with 2015+) In order to do this you need to perform the following: Under Build Definition select the Options tab. Here you’ll see a box for Multi-configuration. Check this box and specify BuildPlatform,BuildConfiguration also ensure you check Continue on Error. Go back to your Variables tab. Under BuildConfiguration specify: “release,debug”. Also, be sure to save and add a comment about what you’ve changed. Now you could QUEUE BUILD and you should see “release,debug” specified under the BuildConfiguration field. Press OK if this is the case and you’ll see two separate builds running under the Build panel on the left hand side. If you wish to browse these builds directories after the build completes you can click the Build tree at the top left panel -> Artifacts (This shows two separate drop folders. One for release and one for debug.) You can speed up the process of deploying release and debug by running builds in Parallel. To do this you will need more than one build agent. To set this option -> Build Definition -> Options -> Under Multi Configuration check Parallel and Save. This is a big time saver and it’s highly recommended that you utilize it.
Automate Builds when Code is Changed / Time Schedule / Pull Request
You can configure your build definition to run when code is changed by selecting the definition you wish to do this for and proceeding to the Triggers tab. From here check Continuous Integration (CI), choose the branches you want to push when code is updated and make sure to Save. This can be tested by opening the project via VS making a change and committing. You should see TFS build out after commit. You can see this take place by clicking the Build Definition -> Builds -> Queued. You can also do scheduled builds by time of day and also configure branches to push to at these specified times. The scheduled option is also configurable by time zone which is good if you have multiple clients testing/using the build branches you’re pushing.
The last way you’re able to configure a build is by utilizing the pool request process. Within TFS click the gear icon at the top right of the browser. Select the Version Control tab, click the repository you want to configure and select Branch Policies. Enable the “Automatically build pull requests” option by checking the box. You should check “always require a new build” option. Again test this by opening the repository in VS editing code and doing a pull request on the corresponding branch. The build must complete successfully in order for the pull request to complete. You can always check the current build status by clicking the “details” button next to the Build in progress step.
Build Agent Capabilities & Demands
The purpose of this section is to go over build agents in more detail. If you open up the “Agent queues” option within TFS you’ll see your build agents listed. Click the capabilities tab and you’ll see what framework each build agent is configured for. This is often used to define which agent runs the builds you kick off.
Select your Build Definition and click the “General” tab. Here you’ll see a list of Demands these demands must be met by the build server/agent in order for the build to run against that agent. Here you can add a demand including framework and/or version. When adding a framework to the list of demands also ensure you specify type. You can make this demand if the item only exists or specify a version of the framework. If you select equals within type also make sure you specify the value (in this case version of framework) and save. These options ensure that the build only runs on an agent that meets all demands in the list. You can test demands you’ve made by clicking “Queue build…” Demands can also be seen from the popup window displayed by clicking the “Demands” tab. After starting the build you’ll see the agent represented this agent will also build out you’re also presented with a position in the TFS build queue. Click the specified builds under the build definition on the left hand side and you’ll see the agent that ran your build as well as build status.
Build Definition History
Build definition history is good for tracking down changes in code after they’ve occurred. The comments you’ve hopefully been added are added within the history and this allows for quick tracking. You can see history for a build definition by selecting the history tab under the corresponding definition. Here you’ll see a complete history along with any notes you added for your builds. You’re also able to select two different builds and click the “Diff” button. This displays the changes in code side by side. Which makes change tracking that much easier! Highlighted lines represent the changes in code between the two builds you’re comparing.
Build Definition Templates
Once you have your build definition configured the way you want and things are kosher you can use the template function. This is extremely helpful when adding new build definitions to TFS. This way, you don’t have to go through the entire TFS suite to mock up another definition for a new software product. If you just right-click your build definition you’re given a “SAVE AS A TEMPLATE” option. Click this, name the template and also provide a short description of what the template is for followed by “OK”. Now, if you go into TFS and create a new build definition you’ll notice a “Custom” tab at the top. Click this and you’ll see your template build definition referenced. If you select this you’ll notice all values are setup just the way you left them in the parent build definition. ☺
Test hub is accessible via the browser within the TFS GUI. This Test Hub allows you to plan, coordinate and track testing functionality of your software projects. Test hub was previous known as an independent software referred to as Microsoft’s Test Manager (MTM). Prior to TFS 2015 this was always a completely separate product. Within Test Hub you’re also able to run each step and view results within the portal. You’re able to add notes and track bugs with ease. To open this “Test Hub” you’ll need to log into your TFS 2015+ instance. Select a project you’d like to test and click the TEST tab near the top left of the page. Upon doing so you’ll see that the Test Hub is segregated into 4 different categories:
Test plan- This is a prerequisite for Test Hub and is required for test cases to be created.
Machines- These are the with test agents. These test agents are the slaves of the TFS server and run the tests you configure on their host OS.
Test Plan Creation
To create a Test Plan select the project you want to test within TFS. Go to the “TEST” tab and create a test plan using the “+”. You’ll be asked for a Name, Area Path (Project you want to test). Iteration (Allows you to associate test plan to specified iteration). Iteration can always be root if you wish. If you specify root the test plan can be applied to your entire project which is a good starting point for any new users. Once created, you’re able to add more test plans if you wish via the left hand “+”. This allows you to add the following to the Area Path (Software Product):
This test plan is a work item within TFS don’t overthink or be intimated by its presence. If you right click the test plan name you’ll see these functions presented:
Creating Test Suites
Now that you’ve got a test plan created it’s time to create some suites to utilize it. Test suites are like folders that allow you to organize test cases you’ll create shortly. Right-click your test plan within TFS and select one of the following options-
New static suite - Can contain both test cases and other suites. The root suite of the test plan is a static suite.
New requirement based suite – These are derived from product backlog items, user stories or other requirements. The suite contains all the test cases that are linked to its requirement. This type helps you track how well each requirement has been tested.
New query-based suite – Shows the results of the query you define. For example, you could select all the test cases that have a Priority = 1.
For the first suite let’s create a requirement-based suite. Click the Add new clause and specify Iteration Path as the field. Operation should be “=” and the value should be an iteration. If user stories are specified within the iteration you selected you’ll see User Story’s populate upon clicking “Run query”. You can simply select all of these using CTRL+Shift or CTRL+Click and click the “Create suites” button.
You’ll see a test suite created for each User Story within the iteration populate on the left hand side. Now we’ll add a Static Suite and name it Sprint. Next right click the Sprint test suite and select “New requirement-based suite” your same query from earlier should still be present. Just click the “Run query” button. Select all user stories listed once again and click “Create suites”. This helps PMs with sprint tracking and associates the user stories to a sprint. This also helps automated testers and lets them know these are the user stories that need to be resolved in the current sprint. Now just delete the suites that aren’t listed under the Sprint suite. To do this just hover over each and click the red “X”.
Lastly we’ll create a query-based suite. Right-click your root test suite and select the “New query based suite” option. This will allow you to bring in test cases that match a specific query. This way, test cases won’t need to be added manually. Click “Add new clause” specify “Tags” as the Field, Operation as “Contains” and Value as “Regression” add a Name for the Suite ie: Regression Tests and click “Create suite”.
You’ve successfully setup your first Test Suite environment. From now on, as test cases are created they’ll be added to the appropriate Test Suite depending on how they’re created.
Creating Test Cases
Now we’ll go over creating test cases to utilize the Suite(s) we created. To start, we’ll create these cases within the Static Suite. Select Static Suite from the list of Test Suites. Click the “New” dropdown button and select “New test case”. Name the test case according to what function you’re testing within your application(s). Under STEPS click to add a step. Your first step should always to be log into the application. Second should be to navigate to the function your testing. Third should be to view (add expected result). Last should always be to close out of the application. Click Save and close to add your test case to the suite.
Earlier we created a Query-based test suite for regression tests. This suite was created to pull in any test cases that contained a tag of regression. To add this new test case or any others to this Query-based suite we simply double click the test case -> Tags -> Add… Enter in “Regression” Save and close. Now the Regression Tests suite should have this specific test case listed under it.
Creating Test Cases from the Grid View
This feature is solid for importing data (Test Cases) in from another Test Case Management system into TFS. Or if you have a list of test cases you know you’re going to be entering you can do so in the grid. This is far better than entering them over one at a time. Select the Test Suite you want to add the Test Cases to. Click New -> New test case using grid. You can test this by entering dummy data.
ie: Title – Test Case From Grid
Step Action – Step 1, Step 2, Step 3
Once entered click the “Save test cases” button.
You’ll want to switch back to the list view instead of grid after doing so. This will show your user story created earlier and visually show you the integration test cases if you click into the Test Cases created via grid.
Shared Steps between Test Cases
You may find yourself entering the same steps over and over for Test Cases you create within TFS. Shared Steps allows you to save time by populating steps automatically depending on Test Suite associated. This is useful when specifying Open/Close within a countless number of Test Cases. The other advantage to using Shared Steps is that if your application ever changes, you’re able to go into a single view and change all shared step attributes accordingly. In order to create a Shared Step perform the following:
Open Test Case and proceed to the STEPS tab
Highlight any/all steps you want to be shared
Right-click and select “Create shared steps” from the dropdown menu
Name your step or step(s) accordingly (Best to name after step function)
If you selected multiple steps you’ll see these steps aggregated into a single step within your Test Case and they’ll use the name you specified. Make sure to save here in order to keep changes and your new Shared Step. Now, if you do this within just one test case it’s not reflected automatically across all others using the same steps. You’ll need to go back into each Test Case delete the steps you aggregated and use the “Insert shared steps” option to add your new Shared Step in. Shared steps are searched for by query. This is all done from the STEPS menu within the Test Case.
Parameters are used to specify specifics for test(s). You’re able to add additional data to each step within a test case if needed. Say you want to test long terms or character data within a field. Parameters allows you to set and tie in multiple values to a test case and runs through all specified within a given step or Test Case. Say you have a step about entering information let’s consider:
ie: Enter info for user: First: John Last: Doe Address: Area 51
The above example is a step that we’ll change to enter not one one value of names but multiple. Reflect this by altering the step:
ie: Enter info for user: First: @First Last: @Last Address: @Address
If you add values of “@field” they show as parameters and populate at the bottom of the test case window. Here is where you can define parameter values to use within the step. Each specified will be tested.
This uses the same concept as shared steps. You can essentially “share” parameters across test cases in order to streamline your Test Case processes. This saves you tons of time by not having to go into each individual Test Case in order to edit the same step over and over. In order to do this you just need to click “Convert to shared parameters” when adding parameters to a step in any given Test Case. To add Shared Parameters to any other Test Case double click it in TFS. You’ll see Add a shared parameter set near the bottom of the Test Case window.
Test Cases must be set to a State of Ready in order to run. To change this open the Test Case and select Ready from the State drop down menu. If you find yourself stumbling to find all Test Cases you want to set there is a way to view all under the root directory. If you select the top level directory and click the “Show tests from child suites” (near top right pane) option you’ll see all Test Cases listed. From here you can check the status of all test cases by clicking “Column options” and adding the State field to your view. You can also assign Test Cases from here to different team members. To do this just select the ones you want assigned -> Right-click -> Assign tester -> Select from list. Select the Test Cases you want to run in the view. Click Run within this window and you’ll see two options:
Run test- brings up the web based test runner
Run test using client- brings up Microsoft’s Test Manager (MTM)
Select “Run test” and you’ll see a pop-up populate with all steps specified in the Test Case you selected.
Pass the steps as you work through the Test Case by clicking the checkmark button for pass or the x button for failed. If you do have failed steps you should log a bug on that step and add helpful notes to assist your development team. When you create a bug from the Test Case it’ll link to the Test Case that found the bug within test. If a step fails the Test Case is marked as failed within the Test Suites.
Select the same Test Case you just ran proceed with Run -> Run test using client. This will launch an external protocol request to open the Microsoft Test Manager on your local machine. Your test case will pull into the client. From here click Start Test run through the steps as needed. When finished click “End iteration and move to next”. This should open the next test case in your list if you have another marked as Active. Make sure to Save and Close when running through test so TFS is notified of changes. You do need to refresh the WEB GUI within TFS to reflect changes made within MTM.
One more thing to note is that you’re able to mark tests as Passed/Failed/Blocked/NA straight from the Test Suite menu using the buttons provided at the top of the view pane.
Test Run Analysis
After you populate Test Case Outcome you’re able to view data about Test Cases within a Chart. Click the Charts Tab from the Test Suite you want data for. This will present Failed/Passed/NA/Block Test Case Status in a chart for you. From here, you’re also able to create your own charts. Click New -> New test case chart. Select variables you want reflected on the chart. This instantly populates a demo of what your chart will look like should you save it. This is very helpful for providing transparency to other teams within your organization or even clients. Group by is the most important variable here and presents these options by default:
Every chart you create populates under the Charts view for the Test Suite selected.
If you want to see a history of all test cases ran under TFS no matter what method used you can use the Runs tab. This shows a complete list of all tests ran and shows current state/completed date. If you double click a test run from this list you’ll get a detailed summary of the run. You’ll also see a Test results and Filter tab listed up top. Test results will show you additional details about the run. You can also add notes and explain why a run failed as well as change Failure type and add a Resolution. This changes the run state from Needs investigation to Completed upon adding these values. The Test results tab also allows you to create a bug if needed. These will integrate with the Test Case specified if/when added.
Exporting Test Plans
This is a function designed to utilize Test Case data and present high level reports to share with stakeholders or even fellow team members. From the Test Suite screen just click “Email or print test artifacts” in the left hand pane. There are a ton of options in order to help generate the data you want in this report. You can select just data for the selected Test Suite or selected suite and all child suites. When using the email function TFS pulls AD objects within your domain. This is the only way to email out reported data but you can always forward on after receiving the email from TFS. The print function opens up your default browser and its print functions. You can export and save as .pdf which is another way to send out data beyond your domain infrastructure.
TFS Browser Extension
TFS 2015 Update 2 and up include a Visual Studio Team Marketplace. This offers extensions to enhance overall function and productivity within your TFS instance. As part of the marketplace Microsoft released an extension for Chrome that improves function. This stores data within the API. The extension allows you to capture exploratory testing sessions including screen shots and videos of your tests. It also allows you to enter bugs much more fluidly. You can find additional information about this extension here: https://marketplace.visualstudio.com/items?itemName=ms.vss-exploratorytesting-web