Wow, how would the release testing of SQL Server 2012 feel like?

After 20 months of release of SQL Server 2008 R2, now we are on the journey of shipping the next big milestone-SQL Server 2012! You might ask a question: how would the release testing of SQL Server look like? You know it is such a complicated mission critical server product…

We have been luckily working with SQL Server worldwide teams together including development team, test team and PM team on the product and also on the release testing during the past years, here are how we think and feel from different perspectives. Please note we just participated in the testing on SQL Server management related features, for example SQL Server Management Studio, so below are just about SQL Server management related testing:

• Overall? The release testing is very complicated but well planned according to priority and dependencies, it took hundreds of people and a thousand of lab machines for months, but it is efficient and effective;

• Automated? Most of the testing are automated and distributed to execute on the thousand of lab machines, for example the function testing; for branding and copyright testing, they are manual testing as visual check;

• How many test cases were executed? Take the full function test for example, it is more than 1M;

• UI and API based testing? Both. In other words, we have both white box testing like unit testing and black box testing like integration testing;

• What kind of tests did we run: in short, a lot --functional, performance, compatibility, security, stress, localizability, globalization, localization, fuzzer test, upgradability, patch-ability, supportability, accessibility and scenario testing and so on, also bug bash by all of SQL Server team members;

• Any other Microsoft teams needed to work together? Yes, that is really a lot of communication and collaboration-a big big cross team work! For example: Visual Studio teams and their v-next, Windows 8 team, .net team, office team…

• A little bit more detail about some above examples? For example security, we had both code review, code static analysis by powerful Microsoft internal tools and tools published like PREfix, PREfast, and FxCop; and also testing by security professionals.

You also might ask, “Rome is not built in one day”, how could we ensure the release schedule and high quality of SQL Server?

• Firstly, we are using the mainline and branch engineering model, the mainline code is always kept in a good status and the new features are developed in the branch, we will ensure the new code is well tested before it comes back to the mainline;

• Secondly, we have Community Technical Preview (CTP) versions periodically released to the community and some customers for preview and trial, there are many valuable feedback and comments from them which ensure we are on the right direction that meet the customer requirement and market;

• Thirdly, we also dog food it  Before release it worldwide, we also use it internally on many mission critical infrastructure and application servers. For example, we also deployed AlwaysOn features on some important test scheduling and tracking servers which requires high availability, high performance and execute complicated workflow.

That is about our story - it has been such an amazing experience! And we are looking forward to your questions, experiences or feedback using the SQL Server 2012!

作者：刘春雨

职位：SDET II

关注领域:：SQL Server, SQL Azure, Cloud computing and services, BI and DataMining