Please explain how do we put more load on application and database servers alone with some examples?

A Simple example is when you login to gmail server it touches the application server for validating your request to see if your credentials are authorized and you are a valid user. If you are a valid user then it reaches the DB server to download your data's based upon your network connectivity speed, your mail contents would be downloaded and displayed in your browser. 

Irrespective of the domain, application and the purpose of usage by the end users we would not be in a position to explain you how the load can be generated to the server from the end users perspective in real time. 

Load to the application can be generated in many ways, it is better to generate the load by capturing a sample business scenario with user doing steps like login, create order, logout from the server. You need to do correlation, transactions, add custom codes if required, synchronization points if required, parametrization with multiple values etc run the script fine tune in vugen for 1 vuser with multiple parameter (user) values, run it in controller for multiple parameter values to make sure your load is properly hitting the server. 

At first to you need to find the critical business scenarios for the application, plot it with percentage of users for each scenarios when calculated it should be equal to 100%. WLM (Work Load Metrics) should have Business flows, No of transactions & No of Vusers for each scenarios

You should be good enough in providing think time & pacing to generate proper load in the application, pacing can be calculated by using two parameters they are No of transactions/No of vusers gives you pacing. With out deriving proper pacing you would not be able to generate proper load in to the server, with out proper think time your load to server would be indirect stress testing. No use of testing your performance of application server in that way. You have to be familiar with Run time & Recording settings of Performance Tool either open source & commercial tool. 

If you need to know whether you are generating proper load to the server you can analyze the heap & thread dump using open source & commercial monitoring/tuning tools like Jvisualvm, Jprofiler, Jconsole are used for generating heap dump, tools like jps, jstack tools etc can be used to generate thread dump. 

In case of server slowness/stuck scenario you can generate thread dump, jvm in java/clr in .net application server hang scenario you can generate the heap dump, You can also configure probe into the server to monitor more deeply into the application server/db servers to fine tune the slow moving methods, sql queries, EJB, Weblogic, Hotspots, JVM, Heap, Threads etc. You can fine tune in DB Servers for sequences, indexes, locks, sql queries, ETL, stored procedures, DWH etc. Monitoring tools like HP Diagnostics, HP Sitescope, IBM BMC, Dynatrace, CA Wily Introscope, network monitoring tools like wireshark, fiddler tools used to monitor Latency, Bandwidth. Load balancer (IP enabled, Round Robin, Least Connections), Firewall, Proxy. Overall architecture is to be understood to generate proper load & fine tune the application. 

JVM can classified as heap & non heap further heap can be classified as Eden Space, Survivor Space, Young Generation, Virtual Space I, Virtual Space II, Older Generation (Tenure), Non Heap as Permgen & Pure Native. 

To analyze deep into server to know whether you are generating proper/right amount of users load into server would help my configuring JVM in java by using app server log files 

To generate load you have to go ahead with Baseline Test (1 user load), benchmark test (15 to 20% of actual user load), Actual test run with 100% user load. This is the procedure to generate the load in to the server.

how to handle dynamic boundaries in LoadRunner

Dealing with Dynamic Boundary Parsing

In LoadRunner , we generally hear the term Correlation(Correlation is nothing but capturing dynamic data from the server response to be used as input in further requests).

We generally deal with dynamic data remember not dynamic boundary using following in build function,modify the function as below to include the /IC flag:

.

For Ex---

But now question arises that what should we do if data boundary is dynamic??

Here is the solution for that

Suppose you have the response data as follows, where you are Capturing the string but issue is that the left boundary is always get changing every time. You get the left boundary as Head1,Head2,Head3........HeadX  and so on, where X ranges between 0 and 9, as follows:

Head1=Captured data rb

Head2=Captured data  rb

Head3=Captured data rb

——–

——–

Head9=Captured data rb

You can capture the desired string by putting the following correlation function in place, using the /DIG text flag in combination with LB:

web_reg_save_param(“DynamicCapture”, “LB/DIG=Head#\=”, “RB=rb”, LAST);

If Digits are changing case,

The corresponding place, which you expect to be dynamically filled in with a digit, should be replaced by a pound sign ( # ).

For mulitple Digit you can replace that digit with muliple #### sign,

Head123=Captured data rb

web_reg_save_param(“DynamicCapture”, “LB/IC/DIG=Head###\=”, “RB/IC=rb”, LAST);

If letters are changing case,

Modify the function as below to include the /IC flag:

web_reg_save_param(“DynamicCapture”, “LB/IC/DIG=a#b\=”, “RB/IC=rb”, LAST);

If  digit or a letter are changing case,

Modify the function to use /ALNUM instead of /DIG text flag, and use caret sign(^) instead of # :

web_reg_save_param(“DynamicCapture”, “LB/ALNUM=Head^\=”, “RB/IC=rb”, LAST);

Keep in mind two things:

1.One sign (# or ^) for each place
2.If you have a literal # or ^ sign, it will not be interpreted as a literal, if you use the corresponding flags. E.g. LB/DIG=Head# will not match “Head#”

Can you tell what is the right value for thread peak and how you monitor thread contention?

In my opinion, there is no such thing as "right value for thread peak". It sounds a kind of "what is the right value of cars". 

Threads perform some work and use some resources for that purpose. While a thread waits for a resource being used by another thread, there is a contention for that resource. Waiting is one of thread's states, which depends on an environment in which threads are executed, i.e. POSIX threads / some virtual machine threads which don't have to be aligned with OS threads etc. 

To monitor contention you can check number of threads in specific states / time spent in specific states... but still, you need to look at the numbers within some context (e.g. CPU utilization, number of requests waiting for processing etc.)

Where and what should we check if you are getting poor response time but rest of the other performance parameters are performing well?

Poor Response time but rest parameters are performing fine

You need to focus on time measurements and to drill down the stack. But where should you start? Unfortunately, the test report does not highlight a bottleneck. 
An effective tool to troubleshoot performance of a transaction issued by a single user is a waterfall diagram thanks to its clear visualization. It displays on which requests most of the response time was spent. 
Here's the approach that we use to find bottlenecks by using waterfall diagrams in load tests. Log the entire test traffic. After the test is complete, select a single user / iteration and display its waterfall diagram. It will show bottleneck requests for this VU during a selected iteration with certain load level.

The instrumentation for this approach is built into our load testing tool, but you can generate a waterfall diagrams without it by using Fiddler, a free web debugging proxy as follows: 
1. Install Fiddler on the LR machine 
2. In LR configure proxy on port 8888. Make sure that LR traffic is captured by Fiddler 
3. In LR add a script that inserts a request header "Fiddler-Capture" for one selected VU and iteration (use web_add_header function) 
4. In Fiddler Main Menu > Rules, click Customize Rules. A script editor will open. Find OnBeforeRequest function and add the following script: 
if (oSession.oRequest.headers.Exists("Fiddler-Capture")) 
{ 
//--- Remove the header 
oSession.oRequest.headers.Remove("Fiddler-Capture"); 
} 
else 
{ 
//---- Hide the request from the grid 
oSession.oFlags[Utils.HIDE_FLAG] = "yes"; 
}

5. Run the test. Fiddler will pass-through the entire traffic and will capture sessions with Fiddler-Capture header in the session grid. 
6. Once the test is complete, select the captured sessions in the grid and click Timeline tab that will display a waterfall diagram like this one -http://support.stresstimulus.com/display/doc37/Waterfall+Chart

To select a different VU/iteration, you need to change them in the LR script and rerun the test (our tool does not have this limitation as it stored the entire test log in the queryable database). 
After looking at the diagrams you will know what actions caused the slowness.

Firebug and HTTP Watch

There are many tools that you can use to generate a waterfall diagram of a page. Both, HTTP Watch and Firebug have it. Such tools are used to diagnose performance of a page accessed by one user. If your transaction is slow when used by a single user, then such tools are extremely helpful to pinpoint a bottleneck even without load testing. 
If however your transaction has a scalability problem and slowness is exposed only under load, then using such tools can be tricky. 

The 1st challenge is that you need to generate a waterfall diagram at a time when your server is under load. Because Firebug has no synchronization with LR, you would need to figure out yourself the load level at the moment when the waterfall diagram was created. 
The 2nd challenge is that most of such tools capable of generating a page waterfall diagram, which can be an obstacle if your transaction comprises multiple pages. 
The 3rd challenge is to make sure that the Firebug user operates in a similar environment as VUs in LR, otherwise you will compare apples and oranges. 

We addressed these challenges in our tool. Using Fiddler will also allows to generate a waterfall diagram for a VU participated in the load test so you will compare apples to apples. 
Hope this will help and good luck with your performance troubleshooting.

What is the difference between Iteration and Pacing?

Iteration 

Iteration is nothing but to define the script to iterate for no. of times according to the requirement. It is nothing but repeating the task for multiple times and you need to set to initiate a new user on each iteration in your run time settings which is by default.

Pacing 

Pacing is nothing but the time gap between the iterations. For eg. if you are running your script for multiple iterations, you need to specify the time gap to maintain idleness before starting the next iteration to run.

Eg : If you are running 1 user with 3 iterations and if you pacing time is 3 secs

then iteration 2 will wait for 3 secs after iteration 1 has been completed.

Difference between Performance testing and engineering

Performance Testing: 

Performance Testing consists of purely Testing a application after development of application using any technology, understand the business critical scenarios, run the baseline(1 user load), benchmark(15% to 20% Load), Actual test(100%) 

WLM(Business Scenarios, No of transactions & No of vusers), Pacing, think time, coding knowledge(C, Java, VB) whenever required etc, tool understanding, application feasibility with the performance tool and it is a part of performance engineering 

Performance Engineering: 

Performance engineering includes more of coding knowledge what ever be the technology in which the application is developed(.Net, Java, Flex etc), do all the above steps which you do as part of performance testing 

Apart from this you also take ownership in interacting with various teams(development, qa, infra structure, admin, technology, networking and many more), understanding the architecture of application. Identify the tool which is used for monitoring the application memory which could be of any technology find which probing tool is used for application ex: IBM BMC, HP Diagnostics, HP Sitescope, Dynatrace, Introscope etc. 

Understand the architecture of application helps people find the issues with application using adequate monitoring/profiling tools which helps you understand the issues with application memory, cpu, hotspots, jvm(java), .net(clr), heap, threads, jdbc(database connectivity), EJB, weblogic, parameters in server StuckThreadMaxTime, StuckThreadTimerInterval, PageCheckSeconds, MaxthreadCount, FileDescriptors, CodeOptimization etc 

Understand the main parameters of Web/App/DB servers Web Servers (Web Requests, Web services, Routers), App Servers for memory leakage, threads, jdbc, heap, non heap 
heap can be classified as eden space, survivor, young generation, virtual space I, Older generation(tenure), Virtual space II, Non Heap can be classified as permgen, pure native space etc, Db servers for ETL, Stored Procedures, SQL Queries, Indexes, Sequences etc 

There are various to identify out of memory in jvm in java, clr in .net and other technologies in case of out of memory in jvm you can generate heap dump, In case of out of memory in server(server slowness, server stuck scenario) you can generate thread dump using jstack, Admin->server->monitoring->threads,Dweblogic.Log.Redirectstandtoserverlogenabled=true etc 

You can configure probe tools to monitor infrastructure of application like Nagios, Sprient etc 

Starting from the Initiation, Planning, Development, Testing, Execution, Preproduction(UAT), Go Live(Production) performance engineer is involved as part of building the application 

Performance Engineer is equivalent to system architect and more than that.

What is the difference between Endurance Testing and Failover Testing?

Endurance and Failover testing have very different goals.

Endurance testing is a baseline application load that is applied over a much longer duration (normally at least 5-6 hours) that a standard performance test and is used to find things such as memory / CPU leaks ,threading problems (deadlocks, memory corruption), etc., whereas failover testing is when I apply a load to a system with multiple, redundant servers and arrange for communication to be cut off to one or more environment components to ensure that the remaining load is applied to the running servers. 

I am putting this in short as,
I like to thing of endurance testing as characterizing how long a system can continue to run before a failure whereas failover testing characterizes a system's overall ability to maintain it's operational function. So, endurance testing helps characterize reliability of a system and failover testing helps characterize availability of a system.