-
Search
-
Recently Written
-
Categories
-
Archives
-
Blogroll
-
Admin
Performance Metrics of WebBased Systems
The performance of web-based system will be measured in the following areas:
1 Load Size
The maximum number of users supported under normal and peak load conditions. Load size revolves around simultaneous and concurrent users. Simultaneous users have active connections to the same web site, whereas concurrent users are accessing the site at the exact same time. A workload profile, thus, consists of a likely user composition where the users perform various system operations. Simulating simultaneous users can be done by making sure the test scripts incorporate so-called think time. The purpose of think time is to ensure that not all user requests being simulated will occur at the same time. Removing think time from the test script makes sense if the goal is to stress test the web application by simulating concurrent users (Hagen, 2000).
2 Response Time
Response time, also referred to as latency, is the time elapsed until a request has been processed. Response times can be measured on both server and client, where the latter includes the request queue, network latency, as well as the time required by the server to complete request execution. Response time increase as the number of users increases, owing to higher levels of resource utilization on the system’s server and network. Response time can also be affected by factors not related to user load, such as database size and poor software implementation (Ricca, 2001). Web system end users typically perceive response time to be the amount of time taken from the moment they click the mouse to the moment that a new web page has been fully display on the screen.
3 Throughput
Throughput is defined as how much work the system can support. Throughput denotes the number of operations that can be completed in a given period of time, such as bits transmitted per second, bytes transmitted per second, HTTP operations per day, or million of instructions per second (MIPS). Byte transmitted per second is the most common measure of unit as far as throughput is concerned.
4 Resource Utilization
Utilization is simply the part of the capacity of a component that is actually being used. These components are CPU, RAM, Disk I/O and Network I/O. Resource utilization with respect to these components represents a cost in system operation. If resource utilization will be too high, then response time will be higher as well as a result of longer queuing delays. If resource utilization will be too low, then it could be assumed that a large amount of money has been invested for excess capacity (Lyer, 2005).
Appropriate Approach For Testing
Determining the appropriate approach for testing applications is very important. As it has been mentioned above, there are three different ways to go about performance testing of web-based systems, some of them more difficult than others. Usually the approach of performance testing we will use depends on client requirements or what type of results client wants. In this case, the client wants to measure and test a web-based system to estimate how many users the system can handle simultaneously, when the response time is acceptable. The goal is to simulate real-world usage to find out whether the web-based system can maintain the requested number of users with acceptable response times. As load testing is used to measure how much capacity web system can handle, as maintaining adequate response time. A load test of the entire baseline of software is one of the more involved tests and will require a lot of different scripts to be run at the same time (Kumar, 2001). The load generators will be sending and receiving requests at very high rates. They will also be reporting on the most important performance measurement in Web systems, end user response times. On the other hand, stress testing, however, seeks to determine the behaviour of the system once it reaches load limits, when the server can no longer cope with the load. Load and capacity testing are relatively similar as far as execution is concerned, and hence they are frequently mixed up. It is important to stress, that contrary to a stress test, a load test does not push the system beyond its breaking point. Because load testing is comprehensive, I think it is most appropriate technique to precisely measure and test performance of a web-based system before going online.
Capacity Testing
Capacity testing complements load testing by determining the particular server’s failure point. Load testing, on the other hand, monitors results at different levels of load and traffic patterns. Capacity testing enables identification of a scaling strategy to assess whether to upgrade the current server(s) or employ additional servers, also known as scaling out. The purpose of capacity testing is to determine how many maximum concurrent users a web-based system can handle. This type of testing is referred to as performance testing, where the amount of traffic exposed to the system is regulated and the resulting performance is measured at different traffic loads (Macintosh, 2000).
The table below presents an overview of testing techniques and their main objectives.
| Load testing | Stress testing | Capacity testing |
| Verify application behaviour under normal and peak load conditions.
Enable measurements of performance variables. |
Verify application behaviour when pushed beyond operational limits.
Uncover weak “links”. |
Determine server failure point.
Identify scaling strategy. |
Table 3.1: Goals of load, stress and capacity testing
Stress Testing
Stress testing is performed to evaluate the behaviour of a web application when it is subjected to loads beyond its operational limits. In other words, the purpose of stress testing is to see how the system handles an amount of traffic beyond what it is designed for. For such kind of test, the web-based system is simulated with a heavy load than expectations. Its very helpful to evaluate the resutls of an unexpected heavy internet traffic. The purpose of stress testing is to identify breakpoints and bottlenecks in the system, so that the system can be tuned, upgraded and durable (Zwanzinger, 2004). Stress testing provides a measurement of throughput, application response time, and resource utilization at levels further than the initial loads. Stress testing also makes it possible for Performance and Capacity Management teams to identify if there is any need of additional resources or corrective action which must be taken in order to meet future business growth demands.
Load Testing
For web-based systems, Load testing is useful for figuring out how much capacity a web-based system can handle, but it is more important for identifying bottlenecks that occur only under heavy load. The goal of load testing is to simulate real-world usage to find out whether the web-based system can maintain the requested number of users with acceptable response times. We can define this load in terms of concurrent users or HTTP connections trying to accecss the web system. It is considered to be very useful to test the web-based system to its limits by saturating it with a large amount of transactions, users, and data. It’s very helpful to know that how the web application will respond in the real life. According to Kumar, “Load Testing is creation of a simulated load on a real computer system by using virtual users who submit work as real users would do at real client workstations and thus testing the systems ability to support such workload” (Kumar, 2001). To simulate real users, scripts are created that combine together many common user actions into virtual sessions. Bottlenecks in the system will usually become apparent during this type of test. Load testing is also performed with single operations, or use cases, in order to locate performance issues with specific components under load.
Load testing is performed incrementally, with a fixed number of users added per increment. As users are added, the response time and resource utilization values will increase (Mercury, 2006). Obtaining these performance measures helps facilitate planning for future needs. Also, analysis may indicate that the system needs to be scaled in order to accommodate defined performance requirements.
The problem with load testing is to define the load requirements. It is possible to predict a workload in intranets, because they have usually finite and known users. However, with internets, it is very hard to predict how many users can access and load a web system. Mostly calculations are based on word of mouth recommendations, success of marketing campaigns, and in many cases, luck (Hower, 2006).
Web-based Systems Architecture
Where the web-based systems have increased the number of users, there number of risks has also been increased. So, application performance needs greater attention. In online survey in USA, it has been found that an average waiting time of a web user is eight seconds for downloading a complete page before leaving the site. In France, there may be some variation in this limit but to provide high performance is a main issue in the success of a web system (Cassone, 2001).
A web-based system is different from client-server application i.e. users and customers don’t have much knowledge, its architecture (Proxies, DNS, Servers, firewalls etc.) is more complex, and there are more risks. A web-based system can be consisted of a variety of components, like web servers, databases, and application servers, as well as network devices such as firewalls, routers, switches and load balancers (Dustin, 2002). The range of complexity in web-based systems varies from simple websites to large web systems like Yahoo, Google, or Amazon which includes complex search engines and order completion. We can see the architecture of web-based system in this way, if we take the model of a traditional business transaction system and to change the customer front end by the website. A customer purchases the company products or any kind of services, in exchange for money. There are methods that make it possible that transaction among customer and company. As a substitute of a sales representative, a cashier, a clerk, or such person, the browser will make it possible to point that website. This will result that company will never be closed. Customers can purchase their selves.
For example think about a vending machine. It has a basic user interface. The key function of the machine is to fill orders on the basis of input from users and it also authenticates transfer of funds. Now let’s add some complexity into the machine. We make User Interface a browser-based solution in place of touch pad, which is required to run in different browsers on different operating systems. And machine will have capability to fulfil orders directly from a warehouse, whilst re-stocking and tracking inventory. As a result, people don’t need to put coins in the machine, but they will enter their credit card information. When the people will make transactions, the machine will be required a real time access to credit companies in order to approve these transactions. Furthermore, we would also be expecting that all credit card information that we enter should be really secure. (Macintosh, 2000)
Performance has been a driving force in system architecture for quite a while. Since application performance greatly depends on the architecture, and it is considered to be vital, “as the price/performance ratio of hardware plummets and the cost of developing software rises, other qualities have emerged as important competitors to performance” (Bass, 2003). To test and measure performance of web-based systems efficiently, and to identify what possible enhancements could be there, it’s relatively important to describe its architecture.
which is connected to the web server through internet service provider. Relational database is the main element of web-based systems where we can store dynamic contents. A web server is used to control the communication between database and application server
Cost Benefit Model and Financial Analysis in Software Testing
There is no hard and fast rule for calculating cost and benefit analysis of quality assurance. There are some basic rules that every company has to follow but rest of criteria has been defined according to every company’s own circumstances and conditions. The development strategies in every company are influenced by some factors. These factors might be company specific, customer specific or some times the mix of both. These specific factors have strength to draw new measures or decisions even though the economic feasibility has been proven in mathematical form.
Some of these factors can be the good employees that take 100% responsibility of their work and encouraged an environment of testing product from staff. The other factor might be the involvement of client in testing, such as UAT (User Acceptance Testing). Cost benefit analysis also said to give reasonable proof in favour or implementation and benefit of quality assurance. On the other hand these analyses also take cares that the cost that are being spend on quality assurance is properly justified. Testing is one of important measure of quality management systems. Cost benefit analysis also estimates that cost spend on testing can also be justified or not (Golner, 2003).
The cost spend can be determine by three things, the number of errors found, users working on these errors and average effort during each error. The theory of error in cost benefit analysis is little bit different as compared normal concept of error. In cost benefit analysis error is only fault that need effort to fix and effort need money.
The unit of cost spend in testing is PD [Person Day]. One PD is equal to eight hours of work (Normal one day office time worldwide).
Golner (2003), Wieczorek (2001) and Rothermel (2006) all are agreed that If the cost is represented by “C”, the Users working on error is represented by “U”, number of Errors are represented by “ER” and average effort during work per effort is represented by “BM” then following equation can be true;
C = BM * U * ER [PD]
In the above equation we are unknown of the average effort by user on each error (BM). The average effort per user per error can be determined by following equation.
BM = e * p [PD per error and user]
Where;
e = Average effort per error in Person day [PD]
p = Proportion of Users working on error.
The effort caused by data loss during error does not calculated with BM. Because no one would sure about the loss before its happening that’s why this loss is almost impossible to calculate without empirical studies. Effort by user on each error can be calculated through empirical studies. Wiczorek (2006) reported in his studies that it is the time spend on one error by user is between half an hour to half a day. Therefore
0.0625< e< 0.5 [PD per error and user]
In this case the proportion p is only 50% although the users affected with error is 100% but it is possibility that 30% of user might be not present, 10% also not directly involved, they might hear error from someone else and further 10% might got warning from other users and do not able to use error oriented function. Then the result would be
0.03125< BM <0.25
The minimum cost could be (If the error effort is half an hour per error and user)
CMIN = 0.03125 * U * ER [PD]
The maximum cost could be (effort time is half a day per error and user)
CMAX = 0.25 * U * ER [PD]
The average cost can be
CM = 0.14 * U * ER [PD]
Importance of Cost Analysis in Software Testing
Software quality assurance is actually measurement of processes in Software development life cycle and it also measures software through testing during development and after development of software. No doubt it is very recognized and treated as one of pillars for software development. The soul of software development based on four dimensions that are functionality, quality control, deadlines and cost analysis. The scope of any software project is restricted in these dimensions. If the cost of project and deadlines are specified in advance then functionality and quality assurance has to be limited within these boundaries (Goldner, 2001). So both cost and time have great impact on the quality and working of quality assurance.
There are a lot of miss understanding on the cost of quality in software industry. There are two types of opinions. First says that quality of product should achieve its maximum level and in their opinion it is very economical. This ideal is known as “quality is free” in the industry and first time presented by Juran and Gryna (1998). Their point of view at that time was that the cost of defect prevention is increasing and the ratio of rework is decreasing with high speed as compared with increase in defect prevention so the net cost of project is low hence, quality is free. The other group has different opinion. They think that there is no need for high quality levels and quality can be scarified to gain other objectives of project such as in time delivery and project cost. For example a study of implementing Capability Maturity Module from SEI reports a quote of a development manager “I’d rather have it wrong than have it late we can always fix it later” (Pulk et. al. 1994). The experience shows that there are always return of quality expenses (Slaughter et. al. 1998). The quality improvements always results in saving money that over comes the impression of spending money on quality assurance. But quality assurance only able to save cost if the costs spend on quality assurance is properly planned and used and it is big managerial issue. The cost of quality assurance is very important because every hour and every penny used for rework (caused by poor quality) can be used to improve whole process. Slaughter (1998) presents two major divisions of quality assurance cost. First is conformance and other is non-conformance.
Conformance is actually the cost used to achieve proper quality standard. Broadly speaking there are two basic elements to invest in quality improvement. These are prevention and appraisal.
Prevention is actually the cost used to prevent the defects before their occurrence this includes staff training or design methods etc. and appraisal is actually measuring of quality.
Appraisal ensures that software product is achieving to its desired quality by evolution, audit and software testing etc.
The non conformance costs are actually the costs spend for rework. This includes failures, retesting and software failure on customer site.
It is responsibility of management that costs spend on quality assurance must be justified. Many companies are adopting the system for financial evaluation and also considering on software quality investment (Banker, 1997). It is recommended that software quality should be analyzed as investment because it is possible that company spend a huge investment on quality and not able to justify it (Juran et. al. 1998). If conformance and non conformance costs are properly monitored then there is chance that conformance policies will help company to reduce cost spend on quality.
Economics of Quality Assurance
There is always a statement used that is “Quality Assurance is Expensive” - but it is still confusing that comparing the cost of quality assurance with what?
If we compare the cost of Quality Assurance with the cost of the basic development effort then quality assurance may appear expensive. However, this would be a fake picture because the total lifecycle quality is dependent on quality assurance. The more faults are there in the software or processes the longer time will take to correct them and will take time and money to spend on reporting faults and re-testing them.
Asking the cost of quality assurance is actually the wrong idea to do. It is much more interesting to ask the cost of not to implement quality assurance. Software quality assurance is very helpful for indicating and removing or even predicting faults even before their occurrence.
The cost of faults escalates as project progresses from one stage of the development life cycle to the next. A requirement fault found during a review of the requirement specification will cost very little to correct since the only thing that needs changing is the requirement specification document. If a requirement fault is not found until system testing then the cost of fixing it is much higher. The requirement specification will need to be changed together with the functional and design specifications and the source code. After these changes some component and integration testing will need to be repeated and finally some of the system testing. If the requirement fault is not found until the system has been put into real use then the cost is even higher since after being fixed and re-tested the new version of the system will have to be shipped to all the end users affected by it.
Furthermore, faults that are found in the field (i.e. by end-users during real use of the system) will cost the end-users time and effort. It is possibility that the fault makes the users’ work more difficult or perhaps impossible to do. The fault could cause a failure that corrupts the users’ data and takes time and effort to repair.
The longer a specification fault remains undetected and more likely that it will cause other faults because it may encourage false assumptions. In this way faults can be multiplied so the cost of one particular fault can be considerably more than the cost of fixing it.
Normally it is observed that cost of testing is lower than the cost linked with faults because of poor quality or retesting or re fixing faults. A very few companies take record of these figures.
Some of the more spectacular software failures are well known. For example the Ariane 5 rocket of ESA (European Space Agency) exploded 37 seconds into its launch because of a software fault. This fault cost $7billion. Although it had been tested a series of errors meant that out-of-date test data was used. Another example is the Mariner Space probe. This was meant to be going to Venus but lost its way as a result of a software fault. The FORTRAN program had a full stop instead of a comma in a FOR (looping) statement that the compiler accepted as an assignment statement. (Thetechmag, 2003)
Of course not every software fault causes such huge failure costs. The consequences of faults in safety critical systems can be much more serious. The follow examples are all real cases.
Therac-25 (Canada) - this machine was used in the treatment of cancer patients. It had the capability of delivering both x-rays and gamma rays. The normal dose of x-rays is relatively low while gamma ray dosage is much higher. Switching from one type of ray to the other was a slow process but the operators discovered that if they typed fast it could be made to switch more quickly. However, it only appeared to have switched. Six people died as a result of having too high an x-ray dose. This was a design fault. (Vliet, 2003)
Risk Assessment in Software Testing
Even the best developed software product is useless if it can not be used (Horch, 1998). It seems a very basic common sense concept but security and reliability of product is often main concern of organizations. Risks are the thing like hazards or hurdles which come across during the achievement of required level of software reliability and project goals. Specifically risk is the unwanted event of some potential and that if this event occur, it will result in some damage to either functionality, correctness or any thing else (McGregor et. al. 2001). Testing always find problems in software product but the main job of tester is to prioritize these problems. Experienced testers always have idea where the problem is likely to occur so they arrange whole testing process according to that. This fashion of testing is also called risk based testing but clearly saying it is mostly on personal judgement (Amland, 2000). Metrics is very good idea for measuring of software quality quantitatively and is very suitable for risk based testing. Amland (2000) presents a risk model in his recent research and it is very helpful for risk exposure. There are actually two basic concepts of this model. First is volume of faults present in the software. For example if one software component has some fault and it is detected with testing then it is more possibility that again the fault find on this document. It may not be that same fault but any other one. So components with detected faults must be more carefully examine in re-testing and regression testing. As the amount of known faults increases it is more likely that there are other faults exists which is not detected yet (Myers, 1997). Second basic element in risk model is cost of fault. It is common practices that companies deploy software and it contains faults. Although companies choose to build project with the knowledge they known but when the budget got limited and other constraints are also effecting like schedule stress, shipment of project etc. however when companies got limited budget then they prioritize risks to most critical risks and pay most attention on them. Chen et. al. (2002) defines a mathematical to calculate risk exposure that is
RE (f) = P (f) * C (f)
Where;
RE (f) is risk exposure of function f.
P (f) is probability of fault occurrence in function.
C (f) is the cost if a fault is executed in function f.
