Software Testing: A SteP-IN Dataquest Survey shows it's poised for growth

Software testing industry has undergone a big change over the last decade and is now being perceived as an engineering discipline, firmly integrated into the development or product implementation process. In addition, this has attracted top talent and demands attractive compensation. But the testing industry has come with its own challenges and complexities which the industry needs to look into, and address them for further growth. STeP-IN forum and Dataquest have conducted a survey on The State of Practice in the Indian Testing Industry to identify the challenges. The survey dwells on areas relating to business objectives, forecasting, and approach to test services delivery including tools and automation.

Testing Business, Demand and Growth

Testing is one of the key revenue generators for IT service organizations:

• A majority of the organizations (54%) reported that 20% or more of their staff are dedicated to full-time software testing. This included 14% respondents indicating 30% or more staff are dedicated to software testing
• More than 80% respondents reported full-time (dedicated) testers for 70% or more of test activities. This reaffirms that organizations view testing not as a stop-gap work, but as a dedicated activity. The use of part-time testers (10%) and consultant staff (10%) is also suggested
• Fifty-four percent respondents indicated revenue from software testing in the range of $10-50 mn and 23% indicated this revenue to be in the range of $50-100 mn
• While the testing business is in plenty, contracts seem to be short-term/project based. Only 17% respondents indicated more than 50% of their customers on long-term contracts

Testing Demand and Business Growth

• A majority of the organizations (54%) reported that 20% or more of their staff are dedicated to full-time software testing. This included 14% respondents indicating 30% or more staff are dedicated to software testing
• More than 80% respondents reported full-time (dedicated) testers for 70% or more of test activities. This reaffirms that organizations view testing not as a stop-gap work, but as a dedicated activity. The use of part-time testers (10%) and consultant staff (10%) is also suggested
• Fifty-four percent respondents indicated revenue from software testing in the range of $10-50 mn and 23% indicated this revenue to be in the range of $50-100 mn
• While the testing business is in plenty, contracts seem to be short-term/project based. Only 17% respondents indicated more than 50% of their customers on long-term contracts

Testing Demand and Business Growth

• Respondents acknowledge growth in the software testing business. Twenty-seven percent have been highly bullish and reports an upward growth of 30%. This includes 10% respondents who report a growth of 50% or more. Our analysis suggests software testing business average growth rate at 20%
• The key business domains served by organizations catering to software testing business includesBFSI (31%), telecom (16%), and healthcare (11%). This is in line with the rigor, expected in verification/validation in these industries given stringent statutory regulation and compliance requirements
• Key technologies serviced by test organizations include web technologies (28%), enterprise apps (27%), and client-server (19%). Respondents recognize technologies (SOA, business intelligence, real-time/embedded) in their business served in smaller percentages



• Key geographies included Europe (31% respondents), North America (30%), APAC (19%)
• Twenty-two percent respondents indicated seasonality in testing demand pattern, primarily around quarterly releases and SLA-bound Hotfix releases
• Offering Proof-of-Concept (POCs) is common, as 80% confirmed the use of POCs to materialize new clients with 41% indicating 60-80% conversion following successful POCs

State of Software Testing Delivery

Eighty-six percent respondents reported the separation of testing teams from development teams to offer dedicated testing services. Seventy percent of the respondents have indicated the co-location of test teams with development teams. Separation of testing has happened over the last decade as the need for a formal testing discipline was recognized. Coming to whether testers would continue to co-locate with developers or not and according to the survey, the answer would be yes as long as the testers continue to need knowledge on applications from developers and analysts.

In the survey, test automation framework development and test script creation emerged as intensive activities. One-third of the respondents cited 10-20% and another third cited 20-30% efforts spent on the automation framework creation. Forty-three percent respondents cited 20-30% of test efforts in automation script creation. While there is enough focus on process improvement in testing activities, 43% respondents cited less than 10% of overall testing efforts being dedicated towards the testing process improvement.

The survey also analyzed the types of testing being performed by test teams. On an average 22% of the testing effort was spent on regression testing. This is an area that benefits primarily from automation and has potential for saving efforts. Product organizations indicated as high as 35% efforts being spent on localization testing.

In the survey, changing development plans (46%) and non-availability of test data (19%) were cited as the top 2 reasons towards non-achievement of test schedules. Lack of test staff, lack of tool availability, or lack of domain knowledge rarely prevented the achievement of schedules and appeared low in the list.

Trends in Testing Skill Development and Hiring

Recommendations from reporting managers appeared as the most popular mode of skill upgradation requirement capture, cited by 41% respondents, followed by other methods such as periodic assessments, business needs, etc.

• Internal training through SMEs, followed by hiring of skilled practitioners from the market, appeared as the most preferred mode of skill upgradation. External faculty for inhouse training appeared low among options. Respondents cited challenges in the acceptability of fresh campus graduates due to curriculums not linked to industry requirements and the lack of practical exposure students have in practical areas
• Thirty-one percent organizations indicated more than 50% testing staff certified with recognized certifications and an additional 23% cited 30-50% certified staff. ISTQB (from Indian Testing Board) and CSTE (from QAI) certifications were listed as the top 2 choices by 80% respondents

Testing Best Practices and Innovation

Respondents cite maximizing customer satisfaction (90% respondents), maximizing profits and increasing RoI as the key business objectives.

• Sixty-eight percent respondents cited test planning/strategy, and 54% cited test process design as important testing tasks in short-term. Test process design, test planning/strategy also appeared as important long-term tasks among 83% and 63% respondents, respectively. Test tool selection, and building a test COE, though important, appeared lower in the list
• CMMI (84% indicated compliance) and TMM (57%), followed by Lean/Six Sigma appeared as key best practice initiatives being pursued for 1 or more years by test organizations
• Test ODC and test COE models emerged as the popular models with each averaging 26% of test business among the respondents. Innovative models such as test factory are gaining ground and 15% respondents indicate offering the same

Test Outsourcing

Respondents reported test execution and results analysis, regression testing, test automation as the top 3 areas of test outsourcing. Key drivers for outsourcing included lower costs, process effectiveness, and strategic reasons

• Customers evaluate outsourcing partner (supplier) performance through a combination of parameters. It is interesting to note that while cost is the top driver for outsourcing, when assessing a supplier performance, parameters like quality, on-time delivery, defect detection efficiency, and flexibility assume importance
• Seventy-two percent respondents cite ideal lead-time to on-board a new customer as 1 week to 1 month. However only 52% cite an actual lead-time in that range. Only 34% respondents confirmed the ability to match the actual lead time to ideal lead time. This has implications on process improvements required to match the expectations
• Respondents cite the use of metrics for test services. More popularly used in the list include test cases executed, defect find rate, defect density or removal rate, defects introduced and re-opened, and test effectiveness
• The use of third-party providers by test service providers is an accepted practice as well. Forty-five percent providers used third-party providers. The top 2 areas for the use of third-party providers included test automation and support for project delivery (resourcing). Use of long-term contracts for third-party providers is yet to become a practice (only 26% confirmed long-term contracts). Forty-five percent respondents cite short-term contracts and 29% cite informal partnerships/alliances
• On the use of third-party providers, 45% respondents plan to maintain current levels, 34% plan to decrease the use, and 21% plan to introduce additional scope. In a nutshell, the use of third-party providers is here to stay and grow as an extended arm of test service vendors

Use of Test Tools

Respondents cite functionality and customer recommendation as the key influencers in the choice of test automation tools.

• Respondents cite concerns in acquiring/implementing new test solutions. Forty-one percent indicate the lack of requisite expertise, and an equal percent indicate inability to find right solution. Thirty-nine percent indicate the need to educate and train staff. Thirty-seven percent indicate the lack of compatibility with the existing systems. An equal percent have difficulty justifying the benefits compared to costs involved

Tools for Testing

Among the types of tools implemented to support testing, respondents were polled on the usage of tools in 3 broad categories:

Test Resource Management: Respondents cite the use of configuration management (88%), project management (75%), and test planning (73%)

Test Requirement & Design Support: Sixty-one percent respondents cite the use of requirement tracers, 49% cite the use of requirements and specs analyzers

Test Implementation and Maintenance Support: Eighty-five percent cite the use of test evaluators, 76% cite the use of requirement based test case generators, 59% cite the use of test execution tools and 56% cite the use of test preparation tools

On the source of tools, 49% indicated the use of commercial tools, 31% indicated the use of open-source tools and 20% indicated the use of home-grown tools.

Changing Dynamics

The survey points out at few significant areas that are bound to transform the software testing market:

Test Outsourcing: Test outsourcing would see an upward growth rate of 20%. The number of players is seeing a consolidation and the industry would see more players with testing revenues in the range of $50-100 mn and >$100 mn. Europe is more open to outsourcing and would become a larger contributor to growing testing outsourcing business, to closely equal or exceed business outsourced by North America. Consumer-facing and regulatory compliance focused areas (financial services & insurance, telecom, and healthcare) and technology areas supporting these would remain the key revenue generating areas for test service outsourcing

Customers Continue to Pick and Choose: While long-term contracts provide cost-advantage, customers shy away from locking into long-term contracts. Vendors have to be ready to adapt to evolving business needs and provide services on-demand, and be willing to charge on consumption basis within long-term contracts

Value-proposition of Independent Testing is Well Understood: First-time customers however like to see end benefits along with proof-of-concept exercises. Idea of such PoC is to establish interlock with business, development organization, vendors and establish that testing can be carried out with better quality, acceptable cost and reduced time to production with outsourcing

Separation and Co-location: Separation and co-location are 2 contradictory themes, but seen to co-exist in testing business. For testers to know the business domain of applications/products, the independent test teams are co-located with developers. This is going to see a change with the factory model getting popular. As the test teams gain ground in domain knowledge, it would be possible to run testing as a centralized factory and the need for co-location with development teams would become minimal or need based. The industry would see more reinforcement of dedicated testing careers and testing only through full-time dedicated testers

Test Schedules: Test schedules are primarily dictated by customer needs. There is a gap between expected lead time to onboard new business vis--vis actual lead time. Changing development plans contribute as the top reason for non-achievement of schedules. Overtimes in range of 20% are common. Customers use quality and on-time delivery as the key parameters to assess vendor service. All these facts lead to the conclusion that providers have a long way to go to establish predictability of schedules in test delivery. Standardization of processes, use of tools, and collaboration with the customer (or development teams) has a key role to play towards accomplishing on-time delivery

Test Automation: Frameworks and automation script generation would be some of the areas that would gain attention as customers look for cost savings in testing in 22% or more efforts spent in regression testing. Here, some of the area vendors would need to focus on the use of cost-effective tools, offering RoI and maintainability of scripts. Test automation vendors would need to look for interoperability of tools

(Extracts from the STeP-IN and Dataquest survey on Software Testing)