Beginner’s Guide to Hyperfoil: part 2

In the previous part we’ve deployed our demo application (Vehicle Market) and exercised some basic requests against that. In this post we will focus on processing responses and user workflow through the site.

Processing responses

We will start with a benchmark that fetches single random page with an offering, without the HTML resources part for brevity:

We have investigated what a browser would do and found out that this page executes a request against to fetch a JSON document. Notice the different port : we will need to add another endpoint to the configuration and start selecting the endpoint in steps:

We have added the another sequence with a second request. When the contains the sequences as a list these are executed in-order; the second sequence is not started until the last step from the previous one completes. While you could keep both requests in one sequence the sequence name is used as the default name for the metric. Therefore a metric with the same name would be reported twice. Moving the request to its own sequence solves the problem.

After receiving the JSON the script would modify the DOM and add images referenced in the JSON. Let’s replicate that in our benchmark:

The scenario does not host a simple list of sequences anymore; we have moved the original sequences under and added another section with sequence . The scenario starts with one instance of , when this completes a single instance of is created. There are no instances at the beginning — hosts only definitions but does not create any instances. For details see the user guide.

In the request in we have registered a handler for response body; This handler applies the query and stores the URLs in an array stored in the session variable . This array has 10 slots; as any other resource in Hyperfoil scenario this array is pre-allocated before the benchmark starts. Therefore we need to limit the size — if there are more images than slots the other URLs are simply discarded.

In the second step in the array is scanned by the foreach step. For each item in the array this steps creates a new instance of the sequence. The next to the sequence name means that there can be at most 10 instances of running concurrently. The number of created sequences is then recorded into variable .

In fact the step stops scanning the array when it finds an unset slot — gaps in the array are not supported. This is irrelevant for our scenario, though — the fills the array from the start without gaps.

The last step in the sequence is not necessary for our scenario (it would be complete after downloading all the images anyway) but shows how to synchronize after all the images are retrieved. In the step we are blocking the completion of the sequence until drops to zero. The counter is decremented after the response with the image is fully received ( handler) using action addToInt.

We have not explained the notation in the path template fetching the image yet. The is called sequence-scoped access: addressing an array with current sequence index. When creates new instances of the same sequence each will get a distinct index (lowest available) that it can use to read/write its private data. Indices for different sequences are not coordinated, though.

Login workflow

Now that you know how to process responses, let’s have a look on another very common part of user workflow: authentication. We’d like to simulate user visiting the front page, clicking on the Login button and filing out the credentials, possibly multiple times to simulate fat fingers.

You need to get list of valid user credentials; Vehicle Market holds a copy of these (regular authentication flow uses hashed passwords) and you can get the list running:

curl localhost:8083/user-loader > /tmp/credentials.csv

Here is the benchmark:

There’s nothing extraordinary in the first sequence, — we retrieve the landing page and decide if we should provide the correct credentials right away or have 1 or 2 failed attempts. We also select credentials using the step. This step picks a random row from a CSV-formatted file, and stores the values into variables. In our case we pick the first column (columns are indexed starting from zero) into variable and second into .

After this we automatically jump to the sequence (even if we’re not supposed to use wrong credentials). The first step there is the conditional step: this step can terminate execution of its sequence prematurely (subsequent steps are not executed) and execute one or more actions. In we use the action that creates a new instance of sequence .

If the condition does not hold the execution of continues with the well known step. This time we are firing a POST request, with a request body that will simulate a submitted HTML form. Hyperfoil will automatically add the header and URL-encode the variables should there be any special characters. In this instance we’re using a constant value for the password that should not match any actual user password.

By default Hyperfoil adds handlers that will mark the response as invalid and stop session execution when the response status is not between 200 and 399. We’re expecting a 401 response with invalid credentials and therefore we disable this default behaviour by setting (we don’t need to disable the other handler, ). Note that this behaviour can be also set globally in the ergonomics.

After receiving the response (the request is synchronous) we decrement the number of failed attempts by 1 using the addToInt step with shorthand syntax. We have used the addToInt action in the previous example: all actions can be used as steps, though steps (such as ) cannot be used as an action. This is not possible because a step can block sequence execution (waiting for an available connection, or until a variable is set…) but an action runs and completes without any delay — this is the main difference between those.

The last step is the step (similar to the action) creating a new instance of the sequence. This step can be used anywhere in a sequence if it creates a different sequence or the sequence has sufficient concurrency limit (we had that in the previous example) however had we added another step after it we would need two instances of running concurrently and the sequence is not marked as concurrent. When we place this as the last step there is a special case when the step only restarts current sequence, not requiring additional concurrent instance.

The sequence does not require much comment, it issues the same request as , only correctly picking the password from session variable. Let’s have a look on the results:

We can now see 2xx responses for and 4xx responses for as we expect. Also the response times for a successful login are somewhat higher, maybe because the server stores a new token in the database.

Looking at browser network log we can see that the webpage captures this token and fetches user profile using that (it will also use this token in the header when talking to other services). Let’s add this to our test, and one more thing: while Hyperfoil can send another login request almost immediately your users would need some time to type these. Therefore we are going to add some user think time:

We have added constant 2-second pause as the last step of , and another pause into using negative-exponential distribution with expected average of 2 seconds but ranging from 500 ms to 10 seconds (the actual average will be about 2044 ms due to these limits).

Then we have added a simple body handler to the successful login request, storing the value in session variable , and a step to the sequence that will start the sequence with single . You can notice that we had to use a new notation in the pattern: . While pasting numbers into the request path is fine, a token might contain some special symbols (such as ), and we need to URL-encode those. Contrary to the form used in the Hyperfoil cannot run the encoding automatically for you since it can’t know if the session variable contents is already URL-encoded (e.g. if you fetched an existing URL into that).

Let’s run this and see the output of command:

In a colorful CLI you’d see all the lines in red and some errors listed below: “Exceeded session limit”, and we did not run all the ~100 index page hits. What happened?

Hyperfoil has a fixed limit for concurrency — number of virtual users (sessions) executed in parallel. By default this limit is equal to user arrival rate (), so in this scenario it was 10 concurrent users. However as with all those think-times the session takes several seconds, we will require more than 10 concurrent sessions, even if the virtual users are idle in their think-time. Average session should take 4 seconds pausing plus some time for the requests, so we can expect little over 40 concurrent users. We’ll add some margin and raise the limit to 60 sessions using the property:

After running this we’ll take a look on stats:

There are no errors and the request numbers are as expected. The throughput is somewhat off because the total duration of the phase was several seconds past — Hyperfoil starts the sessions within the configured 10 seconds, then the phase moves to a state but it won’t complete (state ) until all sessions don’t execute its last step and receive response for the last request.

We can also take a look on number of sessions running concurrently using the command in the CLI:

Our guess that we’ll need 60 concurrent sessions was not too far off as at one moment we had 53 sessions running concurrently. You can also run this command when the test is being executed to see actual number of sessions rather than grand total for the whole phase.

This concludes our second blog post with a deep dive into complex scenarios. In the next article we’ll go through setting Hyperfoil up in an OpenShift cluster.

Software Engineer @ Red Hat, Brno, Czech Republic