pws_upgrade_load.html.md.erb

---
title: Pivotal Web Services Performance During Upgrade
owner: CloudOps
---

<strong><%= modified_date %></strong>

This topic provides sample performance measurements of a Cloud Foundry installation undergoing the workload associated with an upgrade. 

To obtain these measurements, Pivotal repaved its production Pivotal Web Services (PWS) deployment. The repave process simulates system load that would be incurred when performing a rolling upgrade of Diego cells.

Use the measurements and configuration values published in this document as guidance when ensuring you have adequate file storage hardware prior to a platform upgrade.

For more information on the impact of upgrade on file storage performance, see [Upgrade Considerations for Selecting File Storage in Pivotal Cloud Foundry](./storage_considerations.html).

## <a id="configuration"></a> Platform Configuration

The following table details the starting parameters and configuration of PWS.

<table border="1" class="nice">
    <tr>
     <th>Configuration</th>
     <th>Value</th>
     <th>How to Locate</th>
    </tr>
    <tr>
     <td><b>IaaS</b></td>
     <td>Amazon Web Services</td>
     <td>Refer to your Ops Manager Director configuration or BOSH deployment manifest.</td>
    </tr>
    <tr>
     <td><b>File Storage</b></td>
     <td>AWS EBS (External with some elastic capacity)</td>
     <td>Refer to your Elastic Runtime configuration or BOSH deployment manifest.</td>
    </tr>
    <tr>
      <td><b>Version of CF</b></td>
      <td>v252</td>
      <td>Refer to your Ops Manager Director and Elastic Runtime configuration or BOSH deployment manifest.</td>
    <tr>
     <td><b>Number of Diego Cells</b></td>
     <td>218</td>
     <td>To view the number of Diego cell instances currently running in your deployment, see the <b>Resource Config</b> section of your Elastic Runtime tile or consult your Diego deployment manifest.</td>
    </tr>
    <tr>
     <td><b>Maximum Number of Started Containers</b></td>
     <td>250</td>
     <td>See <a href="../adminguide/max-container-starts.html">PCF</a> or <a href="http://docs.cloudfoundry.org/adminguide/max-container-starts.html">Cloud Foundry</a> documentation for configuration information.
     </td>
    </tr>
    <tr>
      <td><b>max_in_flight Configuration for Diego Cells</b></td>
      <td>6</td>
      <td>To retrieve the existing <code>max_in_flight</code> value for the Diego Cell job in Ops Manager Director, use the Ops Manager API. See the Ops Manager API documentation. If you are running open source CF, consult your BOSH deployment manifest.
      </td>
    <tr>
      <td><b>Number of Availability Zones (AZ)</b></td>
      <td>2</td>
      <td>Consult your Elastic Runtime or BOSH deployment AZ configuration.</td>
    </tr>
    <tr>
     <td><b>Number of App Instances</b></td>
     <td>16231</td>
     <td><code>datadog.nozzle.bbs.LRPsRunning</code></td>
     </tr>
    <tr>
     <td><b>Number of Application Security Groups (ASGs)</b></td>
     <td>43</td>
     <td>As admin user, run the <code>cf security-groups</code> command. For more information, see <a href="../concepts/asg.html">Understanding Application Security Groups</a>.</td>
    </tr>
 </table>

## <a id="during_upgrade"></a> System Performance Measurements During Cell Repave

This table presents performance measurements taken during the Diego cell repave. 

<p class="note"><b>Note</b>: These measurements indicate the peak cumulative values of the entire system (250 Diego cells, ~15,000 application instances, and 2 AZs.)</p>

Use these measurements as a baseline for expected system load during Diego cell upgrade.

 <table border="1" class="nice">
    <tr>
     <th>Measurement</th>
     <th>Value</th>
     <th>Metric Used</th>
    </tr>
    <tr>
     <td><b>Cell CPU Consumption</b></td>
     <td>36%</td>
     <td><code>bosh.healthmonitor.system.cpu.user</code></td>
    </tr>
    <tr>
     <td><b>Cell Memory Consumption</b></td>
     <td>~50%</td>
     <td><code>bosh.healthmonitor.system.mem.percent</code></td>
    </tr>
     <tr>
     <td><b>Cell I/O Consumption (Read) During Normal Operations</b></td>
     <td>43 Read I/O Operations per second
     </td>
     <td><code>aws.ebs.volume\_read\_ops</code></td>
    </tr>
    <tr>
     <td><b>Cell I/O Consumption (Read) During Upgrade</b></td>
     <td>1,943 Read I/O Operations per second
     </td>
     <td><code>aws.ebs.volume\_read\_ops</code></td>
    </tr>
    <tr>
     <td><b>Cell IO Consumption (Write) During Normal Operations</b></td>
     <td>2,166 Write I/O Operations per second</td>
     <td><code>aws.ebs.volume\_write\_ops</code></td>
    </tr>
    <tr>
     <td><b>Cell IO Consumption (Write) During Upgrade</b></td>
     <td>21,000 Write I/O Operations per second</td>
     <td><code>aws.ebs.volume\_write\_ops</code></td>
    </tr>
    <tr>
     <td><b>Cell Network Consumption (Network Out) During Normal Operations</b></td>
     <td>~1.25 GB per minute
     </td>
     <td><code>aws.ec2.network\_out</code></td>
    </tr>
    <tr>
     <td><b>Cell Network Consumption (Network Out) During Upgrade</b></td>
     <td>~1.25 GB per minute (no significant change)
     </td>
     <td><code>aws.ec2.network\_out</code></td>
    </tr>
     <tr>
     <td><b>Cell Network Consumption (Network In) During Normal Operations</b></td>
     <td>2.11 GB per minute</td>
     <td><code>aws.ec2.network\_in</code></td>
    </tr>
    <tr>
     <td><b>Cell Network Consumption (Network In) During Upgrade</b></td>
     <td>16.75GB per minute</td>
     <td><code>aws.ec2.network\_in</code></td>
    </tr>
 </table>


## <a id="graph"></a> Sample Performance Graphs

These DataDog graphs represent a timeline visualization of read and write operations during the repave event. 

### <a id="graph_readops"></a> Read I/O Operations

The read I/O operations sample was taken over 115 VMs and represent the number of read operations over 300 seconds for a single Diego cell.

<%= image_tag("pws_read_ops_upgrade.png") %>

### <a id="graph_writeops"></a> Write I/O Operations

The write I/O operations sample was taken over 115 VMs and represent the number of read I/O operations over 300 seconds for a single Diego cell.

<%= image_tag("pws_write_ops_upgrade.png") %>

## <a id="summary"></a> Summary

During the repave process, 250 Diego cells were updated. The repave process took 6 hours overall or about 3 hours for each Availability Zone.