How to benchmark a VM

Overview

The ASCAMSO Benchmarking is managed from the ASSESS Tab

The ASCAMSO Benchmarking consists of two major elements:
ASCAMSO-Satellites - small, Java based testing programmes that are downloaded and executed on the VM to be measured. When started they begin measurement of state and performance while transferring data to the platform for further analysis. While the focus is on IaaS environments, "classical", physical hardware settings can be benchmarked as well. The tests are set up in a way they do have a minimal impact on the performance.
ASCAMSO-Scenarios - scenarios organize the scheduling and content of the test runs. scenarios can be run repeatedly for comparison of multiple/regular measurements. As test-data will be stored, analysis and comparison of test results is possible at any time.

To create, configure and operate satellites and scenarios, use the navigation bar on the left. The main window on the right will show details of the corresponding scenarios. Learn how to create and install new satellites in the next section.


     Navigation                                scenario details, option & analysis

 

Create and install new satellites

 

Create a Satellite

To create a new satellite: ensure you are in the ASSESS section, eventually choose 'Assess' in the menu on top of the page.
Ensure "satellites" is selected in the menu bar on the left.
Hit the Plus symbol (plus) in the menu bar on the left. Add the satellites name and a comment in the opening popup. Create or cancel.

In the example below we created a satellite "Testdatacenter 4C 8GB Linux". It is visible now as "Offline".
To configure the satellite see the next section "Configure a satellite". You can download a satellite multiple times from ASCAMSO portal onto different VMs.

Configure a Satellite

to configure the settings of a satellite, ensure you are in the ASSESS section. In the menu on the left choose to manage satellites, find the satellite to configure in the listing below. Klick on the satellites name to open the configuration window.

In the configuration window edit the name and use tags as appropriate. Tags will help to manage even larger numbers of satellites as they are found in the standard search.

Assign provider and tariff to the satellite by typing in the 1st letters. A list of matches is generated. Choose your provider and select a tariff in the adjacent field.

The warning messages you might see are for information purpose only. They will disappear once the satellite has been placed  and was started. 
However, there are limitations for  testing on a Windows server (while all tests will work on a Linux VM). So once the satellite has been placed and was activated, it will detect the operating system. If it is Windows, you'll be noticed on the  restrictions.

Please add path for the read/write files according to your OS/System in the section IO targets.

press save to store the settings and close the window.

 

 

Download the satellite to your computer

In order to deploy the satellite, it will be downloaded to your computer, extracted and uploaded to the VM. For the next steps ensure you are on the ASSESS screen (menu on top) and have chosen to manage satellites (menu on the left).

  1. Find your satellite on the left menu. Pressing the download icon next to the satellites name, will let you choose between ZIP- or TGZ-format. Choose the format you are most familiar with.Start the download to your computer by choosing your option.

  2. Downloading creates a compressed file inside your download folder named Satellite_xxxxx-xxxx-xxxx-xxx.

    This folder contains:

    1. The satellite

    2. The script RunSatellite

    3. A portable interface for gathering system information regardless of the underlying platform.

  3. Please extract the downloaded folder that was created inside the download folder. Move to the new, decompressed folder.

The satellite is now ready for deployment.

 

Deploy the satellite 

In order to run the satellite, it will be transferred to the VM and made executable. We then will ensure the correct Java version is installed. 

Depending on the operating system of your computer and your preference, the approach will be different.
Ensure you have the IP address and the credentials of the VM available.

 

Choose your operating system and transfer the Satellite's folder to the VM

Windows PC -  use PuTTY, a SSH and telnet client, or a similar programme to connect to the VM, port is 22

MAC OS - use an FTP client like FileZilla (open source, free to use) to connect to the VM and upload the folder in a graphic user interface

or use the command line interface to connect to the VM and transfer the folder.

Linux - use the command line interface to connect to the VM and transfer the folder to it.

 

Now the satellite's files should reside in a folder on your VM. Next step ensures the correct Java-version is installed:

Windows PC -  use a terminal programme like PuTTYto connect to the VM and provide a command line interface. Set SSH as protocol, port is 22.

MAC OS - use the terminal or a terminal programme like iTerm to connect to the VM and provide a command line interface

Linux - use the command line interface to connect to the VM

 

After connecting to your VM, install Java SDK from the command line: 

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

 

Make the satellite executable:
chmod +x RunSatellite.sh

 

 Start the satellite interactive

./RunSatellite.sh 

 or start satellite as deamon

sh ./RunSatellite.sh  start

While the satellite is active, test results are transferred continuously. They can be observed in the according scenario that was created by the system with the same name as the satellite. To observe it, choose to manage scenarios in the menu left of the ASSESS screen (menu on top). Scenarios with running satellites are listed as active. Klick on the scenario(s) name to view and edit it.

 

Create and manage scenarios

Create a scenario

Scenarios define test runs and analyze the data captured by the satellites. Use a default monitoring scenario with recommended standard settings or define your own scenario.

default scenario is generated by the system any time a satellite was created. You can not create it, it will be done automatically and state "Default monitoring scenario" in the description.
The settings are fixed and provide a standard test setting we recommend for cloud environments. For instance it will limit the utilization of the VM through benchmarking to 1 % or below.

However, there are settings where you want to define your own scenario which is possible too. To generate a custom scenario, ensure you are in the ASSESS section. In the menu on the left choose scenarios. Hit the Plus symbol in the menu bar on the left to create a custom scenario.

Enter name and description in the following popup, hit "create" to save and exit the window.

The new scenario will be shown as inactive in the menu on the left.

 

Manage and use scenarios

ensure you selected "SCENARIOS" in the sidebar menu on the left of the ASSESS screen (menu on top). Tap on scenario names to show their details in the center window. Tapping again removes them from detail view.

Scenario sidebar menu

The settings button next to the scenarios name let you clone, freeze or delete the scenario. The play button will start the scenario.

Scenario details

The detail window is divided in an upper part with common functions and 5 TABs below that structure the elements of the scenario.

The symbol top left of the detail window allows to toggle between running and stopping of the test run.
Next to it you find the scenario name and the description. Below the number of runs is indicated.

 

Buttons in the top right of the detail view allow to:

Duplicate - duplicate your scenario

Freeze - locks the settings of a scenario, so they can not be amended. Once locked, you can not unlock a scenario again. Still, you can clone it and amend the clone.

Suggestions  - will query our tariff database and come up with alternative tariffs that match the performance measured.

Download - Extract the data captured for further analysis in CSV or Excel-format.

 

 

TAB Overview

5 TABs structure the management of scenarios:

DIAGRAM - powerful but easy to use analytic tool to view, analyze, compare or export the data captured in interactive graphs. 

REPORT - displays data captured in statistical relevant performance figures 

GENERAL - define specifics of the scenario 

SATELLITES - define which satellite(s) are part of the scenario

EDGES - benchmark inside your virtual datacenter by measuring the performance of its components

Find more on Function and content of the tabs below.

 

DIAGRAM

powerful but easy to use analytic tool to view, analyze, compare or export the data captured in interactive graphs. 

The data displayed is configured by selecting the chart-type and a test run.

 

Chart-type: The options are:

Processing - Capacity Measurement: see ASCs and total CPU utilization

Memory - memory utilization in %

CPU - total and per-core CPU utilization

Throughput - in Kilobyte per second

Latency - in milliseconds

IO - in MB per second

 

Current Test Run: select the test run you want to analyze here. The most recent run is pre-selected. 
Tick the 'compare test runs' box to add a second test run for comparison.

 

Compared Value: compare the ASCs of the VM with the performance of different CPUs we benchmarked. Selecting a CPU draws the value into the diagram. This will work for "processing" charts only.

 

For scenarios that had at least a single run, the processing chart of the most actual run will show up as soon as you open the diagram tab. 

You can manipulate the diagrams in a number of ways:

ZOOM: to zoom into a section of the diagram, klick and hold the left mouse-botton and move the mouse to define the range you wish to deep dive into. Release the button when done and the chart will be redrawn with the defined range. 
To reset, klick the 'Reset zoom'-button in the upper right corner of the chart.

 

Show all / hide all: some charts can become pretty crowded. You may focus on certain elements by selecting or unselecting single values in the legend right beside the diagram. To 'show all' or 'hide all' values, use the according button. 

 

Print or download the graph: klick on the symbol below the chart legend. You are offered to print or download the chart in different formats.

 

Show report: add statistical figures to your chart for a deeper understanding. Min, max, average, median, quartiles, variation, standard deviation and mean absolute deviation (MAD) will be calculated.

 

REPORT

view the selected test run in statistical values derived from the data captured.

See Minimum, maximum and average, quartiles and the median.

The spread of the data is calculated as variation, standard deviation and mean absolute deviation (MAD) to describe the variation in performance.

 

GENERAL

define specific settings, manage tags or add a description to your scenario.

Note specifics of your scenario in the DESCRIPTION field.

Add TAGS to ease search and selection, especially with lager numbers.

INTERVALS between CPU test runs, throughput tests and IO tests limit the utilization of the system tested. The standard settings are 5000 ms between CPU tests and 30000 between IO tests. Throughput has no intervals (=0). The values that can be changed as desired.

Specify the tests to be performed in your test run by selecting/deselecting CPU test, throughput test, Latency test and IO test.

SATELLITES

After creating a custom scenario from scratch, here you would attach a satellite to it.
See the available satellites on the left. Klick on the arrow in the middle to add a satellite to the scenario. Added Satellites move to the right side. To remove them, click x on the right.
But there is more: the ASCAMSO test suite provides a benchmark of the network performance inside your virtual datacenter or between internal and external components.
To monitor network latency and network throughput, two satellites are placed as "sender" and "receiver" of the data.
For this network test you may select two satellites in the SATELLITES tab and move onto the tab EDGES to specify the interaction between them. 

EDGES

Edges allow to benchmark network latency and network throughput inside your virtual datacenter or in an interaction with external components.

This is done with two satellites that are chosen in SATELLITES tab. In the EDGES tab you do the connect between them.
To define the test, klick on "+" in the center of the screen.

 

Choose a satellite as "sender" of the data. Receiver is already populated with "Base Control", allowing a test with one VM and our platform as counterpart.
In any other scenario, select a "receiver" satellite.

After selection of the RECEIVER, two additional fields show up: Receiver IP and Port.
Choose an internal address if sender and receiver are part of the same network, otherwise choose external. Specify the port.
Check mark to store the definition. Hit X to delete it. Hit + to create another line.

 

Collecting and spending credits

We at ASCAMSO are keen to help our customers getting the best out of the cloud. Therefore we continuously invest into building tools and services dedicated to ease business use of cloud services. 

These services are charged in "credits". Your account comes with 50 credits that can be used for the services offered. Beyond that you can purchase additional credits. See all details here.

 

Have more questions? Submit a request

Comments