testing

1. Testing, testing
A first-look at the new testing
capabilities in Visual Studio 2010
Mathias Olausson
Callista Knowledgebase | QWise
QWise software engineering – refactored!
www.qwise.se

2. About Mathias...
QWise software engineering – refactored!
www.qwise.se

3. About the Presentation...
• What’s on the menu?
– Test planning and execution
– Issue tracking with rich bugs
– Eliminating the "works on my machine / cannot
reproduce" scenario
– Automated UI Testing
– Virtual Lab Management
QWise software engineering – refactored!
www.qwise.se

4. TEST PLANNING AND EXECUTION
QWise software engineering – refactored!
www.qwise.se

5. Agile Testing
QWise software engineering – refactored!
www.qwise.se

6. Core Practices
QWise software engineering – refactored!
www.qwise.se

7. Agile Testing Quadrants
Business-facing
Manual
Functional Tests
Exploratory Tests
Unit Tests
Component Tests
Performance Tests
Load Tests
Security Tests
”-ilitets” Tests
Critisizing the product
Supporting the team
Automated &
Manual
Automated
Automated
Technology-facing
QWise software engineering – refactored!
www.qwise.se

8. Test Types
Manual Tests
UI Tests
Acceptance Tests
Unit Tests / Component Tests
QWise software engineering – refactored!
www.qwise.se

9. Testing in an Iteration
sprint
sprint
plan
plan
build 11
build
sprint
sprint
plan
plan
implement
implement
user story ‘1’
user story ‘1’
(us1)
(us1)
build 22
build
implement
Implement
Implement
us2
US2
US2
fix bugs
fix bugs
build 33
build
write tests
write tests
for us1
for us1
fix bugs
fix bugs
build 44
build
test us1 &
test us1 &
file bugs
file bugs
write tests
write tests
for us2
for us2
QWise software engineering – refactored!
build 55
build
build 66
build
verify fixes
verify fixes
test us2 &
test us2 &
file bugs
file bugs
build 77
build
regress
regress
impacted
impacted
tests
tests
verify fixes
verify fixes
www.qwise.se

10. Microsoft Test Manager and Visual Studio
Testing (Teams)
Development (Teams)
Engineering Support
Unit Testing
Specialist
Generalist
Coded UI Test
Microsoft Test Runner
Web Test
Fast Forward For Manual Testing
Load Test
Test Case Management
Lab Management
Data Collectors (IntelliTrace, Test Impact, Metrics)
Team Foundation Server
QWise software engineering – refactored!
Reporting
www.qwise.se

11. The Fundamentals
QWise software engineering – refactored!
www.qwise.se

12. Defining and Running Tests
QWise software engineering – refactored!
www.qwise.se

13. ISSUE TRACKING WITH RICH BUGS
QWise software engineering – refactored!
www.qwise.se

14. Should We Do Bug Tracking?
Pros
– Tracability
– Priority of work
– Visibility
Cons
– Overhead
Should we log bugs within an iteration?
QWise software engineering – refactored!
Source: Pixar ”A bugs’ life”
www.qwise.se

15. Defect Tracking in MTM and MTR
QWise software engineering – refactored!
www.qwise.se

16. No-more-no-repro
• Automatic data
collection
• Test settings
– Configuration of
diagnostic data
collectors
– Assigned to test plan
• Or per test run
QWise software engineering – refactored!
www.qwise.se

17. Diagnostic Data Adapters
QWise software engineering – refactored!
www.qwise.se

18. No-more-no-repro
QWise software engineering – refactored!
www.qwise.se

19. AUTOMATED UI TESTING
QWise software engineering – refactored!
www.qwise.se

20. Why Automated Testing?
Källa: Tools for Agility, Kent Beck
QWise software engineering – refactored!
www.qwise.se

21. Automation Strategy
iteration 11
iteration
feature aa
feature
feature bb
feature
iteration 22
iteration
feature c c
feature
release iteration
release iteration
feature dd
feature
unit
testing
manual
testing
regression
testing
performance
testing
QWise software engineering – refactored!
www.qwise.se

22. Coded UI Tests
• Unit test for UI
• High-level framework
• Create test from
– Recording
– Test result (action
log)
– Using API
QWise software engineering – refactored!
www.qwise.se

23. Automation Platform Support
QWise software engineering – refactored!
www.qwise.se

24. Coded UI Tests
QWise software engineering – refactored!
www.qwise.se

25. LAB MANAGEMENT
QWise software engineering – refactored!
www.qwise.se

26. Motivation
QWise software engineering – refactored!
www.qwise.se

27. Lab Management Aims To...
QWise software engineering – refactored!
www.qwise.se

28. Lab Management Infrastructure
Team
Foundation
Server (TFS)
Lab Management
System Center Virtual
Machine Manager (SCVMM)
Test Case management
Build management
Work Item Tracking
Source Control
QWise software engineering – refactored!
Hyper-V
Hosts
Library
Shares
Virtual
Machine1
Library
Share1
Virtual
Machine2
Library
Share2
Virtual
Machinen
Library
Sharen
www.qwise.se

29. Environments
• An environment is a set of machines with
different roles
• An environment can
– Be deployed, saved, removed
– Started, stopped
– Snapsshot
VM1
Lab Agent
QWise software engineering – refactored!
VM2
Lab Agent
www.qwise.se

30. Manual Tests in Lab
• Testing from the outside
– Easy for web applications
– Limitations in data collection
• Testing from the inside
– Run application in lab via environment viewer
QWise software engineering – refactored!
www.qwise.se

31. Automatic Tests in Lab
• Testing in VM
– Test case connected to test environment
– Test component comes from selected build
• Testing in build
– Test case connected to build definition
– Runs test list or test component
QWise software engineering – refactored!
www.qwise.se

32. Lab Management Build Workflow
QWise software engineering – refactored!
www.qwise.se

33. Build and Deploy
QWise software engineering – refactored!
www.qwise.se

34. Summary
•
•
•
•
•
Structured test planning with MTM
Integrated test execution
Effective defect tracking with rich bugs
Seamless integration with automated tests
Powerful lab management
QWise software engineering – refactored!
www.qwise.se

35. Questions?
mathias.olausson@callista.se
http://msmvps.com/blogs/molausson
QWise software engineering – refactored!
www.qwise.se