zData Labs is currently leveraging Pivotal GemFire for the zData Big Data Pilot Program, which runs on a cloud platform. Pivotal GemFire is an in-memory, distributed data store that provides a SQL interface to table data. It is designed for applications that require very low-latency transactions, high availability and maximum scalability.

The purpose of this initial benchmark was to gain insight on proper sizing, expansion planning and management of GemFire client/server architectures on a cloud platform using a midrange cluster configuration.

As part of this effort, the Yahoo Cloud Serving Benchmark (YCSB) was used to test the performance of the system using a Core Workload which best emulates real-time analytics and OLTP-oriented applications. YCSB has become widely recognized as the industry standard for benchmarking cloud applications.

The main focus points of the benchmark were sustained throughput and associated latency, horizontal and vertical scaling, and performance/management characteristics. The results indicate that GemFire performs well on relatively small cluster configurations on a cloud platform. It displayed a healthy increase in throughput while maintaining low latencies in both vertical and horizontal scaling. In regards to performance and management, GemFire proved to be a lightweight architecture with low maintenance requirements that lends itself well to a cloud platform. Increasing and decreasing the size of the cluster and per-node resources was relatively seamless, and performance tuning was not as complex as traditional RDBMS systems.

*This benchmark used Pivotal GemFire v.1.4

order cheap viagra online canadian pharmacy GemFire Overview

  • Pivotal GemFire is a memory-optimized, distributed data store and data management platform.
  • Use cases include the following:
    • Real-time transactional applications, such as airline reservation systems, which need to process a large number of parallel transactions.
    • In-memory processing of data sets with very high output and quick response time requirements, such as risk analysis.
    • Enterprise application data caching for distributed computing systems to improve performance.
    • High volume stream processing such as those associated with IoT raw sensor events or market data feeds.
  • With GemFire, data can be managed entirely using in-memory tables, persisted to tables on local disk capable of storing large amounts of data or to a Hadoop Distributed File System (HDFS)
    • GemFire provides an optimized local disk persistence mechanism with a non-flushing algorithm to maintain high performance in applications that require stable, long-term storage.
  • It provides a low-latency SQL interface to in-memory table data, local disk store files and data stored in HDFS.
    • Applications can also use GemFire to actively cache table data from a traditional disk-based RDBMS.
  • A GemFire distributed system can be scaled out using commodity hardware to support a large number of concurrent clients
  • GemFire is built upon the previous VMWare/Pivotal product vFabric SQLFire with the following features added:
    • HDFS-persistence
    • Off-heap memory storage for table data
    • Federated JMX MBean architecture
    • Monitoring using the GemFire Pulse web application.
  • GemFire shares many internal features with the vFabric SQLFire product, such as the SQL language extensions and query capabilities, system tables, local disk store persistence files, product directories, and the gfxd command-line utility.
  • See Overview of Pivotal GemFire  and GemFire Compared to Other Data Management Systems for more detail.

 

cheap cialis Benchmark Scope

Gaining insight on proper sizing, expansion planning and management of GemFire client/server architectures on a cloud platform were the primary goals of this initial benchmark. It focused on the following three categories:

  • Sustained throughput capabilities and associated latency
  • Horizontal and vertical scalability
  • Cloud platform performance and management characteristics

A single workload was introduced that best represented a real-time, in-memory analytics or OLTP-oriented application workload.

  • Measuring sustained throughput and associated latency was achieved by incrementally increasing the number of workload client threads and target operations per second until the point of resource saturation.

Horizontal scalability was evaluated by expanding the cluster from 4 to 8 nodes.

  • The number of records loaded was adjusted in conjunction with physical cluster expansion.
  • Non-persistent (In-memory) and persistent (On disk) data structures were implemented in both cluster configurations as part of the scalability test.

The benchmark was executed on the Amazon Web Services (AWS) Elastic Compute (EC2) environment, which is one of the cloud platforms where zData hosts customer environments.

  • Resource usage patterns were observed include memory, compute, storage and network traffic patterns.
  • One of the objectives was to observe the performance characteristics of solid state storage in the data persistence portion of the benchmark.
    • The AWS Elastic Compute (EC2) instances were configured with General Purpose SSD Elastic Block Storage (EBS) volumes.
    • Magnetic storage or HDDs (Hard Disk Drives) are being deprecated by cloud vendors in favor of SSDs (Solid State Disks) which provide higher and predictable IOPS, throughput and response times.
  • The cloud resources used represented a midrange GemFire client/server deployment on AWS EC2 memory-optimized R3 instances. Details are in the Benchmark Configuration section.

 

Yahoo! Cloud Serving Benchmark (YCSB) Project

YCSB 0.1.4 (Release notes) was chosen as the benchmark tool suite.

  • YCSB consists of the following components:
    • The YCSB Client, an extensible workload generator
    • The Core workloads, a set of workload scenarios to be executed by the generator
  • Of the Core workloads, “Workload A” was used for the benchmark
    • This is an update heavy workload that consists of a 50/50 mix of reads and writes
    • This workload best represents real-time analytics or OLTP-oriented applications.
    • See the YCSB Github Wiki for more details.

 

buy cheap sildenafil citrate Benchmark Configuration

GemFire Cluster Configuration

A 4-node AWS EC2 cluster using the CentOS 6 Linux distribution per GemFire requirements was built for the benchmark and then scaled up to an 8-node cluster. Memory-optimized EC2 R3 instances were used, and each of the R3 instances was backed by 4x 410 GB Elastic Block Storage (EBS) SSD volumes.

These volumes were used to create a Linux RAID0 array which was formatted with the XFS filesystem. The RAID0 volume was mounted on /data and served as storage for the the GemFire locators and data stores. All of the GemFire  cluster instances were EBS-optimized. The EBS volumes used were General Purpose SSD. In a Linux RAID0 array, they have the ability to burst to 4920 total IOPS, with a base performance of 3 IOPS/GiB.

See Figure 1 below for more detail on the cluster node configuration:
Figure 1 - AWS Cluster Node

The GemFire cluster was set up in a client/server deployment for this benchmark. GemFire supports three basic types of deployment: client/server, embedded peer-to-peer, and multi-site (WAN). GemFire can also be deployed using elements from each topology.

For the workload drivers, 2x EC2 R3 instances were used of the same type as the cluster nodes. They were launched into a Amazon Virtual Private Cloud (VPC) within the same EC2 Placement Group. This was necessary to enable EC2 Enhanced Networking between the instances. The YCSB package was pre-compiled with the GemFire JDBC thin client driver on a separate CentOS 6 system and copied to driver2 and driver3.

In general, distributed compute clusters require the fastest possible interconnect fabric between nodes, and GemFire is no exception to the rule. As part of an EC2 Placement Group, the workload driver nodes had access to the same high speed, low-latency fabric as the cluster nodes to insure the benchmark results would not be influenced by slower network performance.

Finally, a general purpose m3.large EC2 instance was used as a benchmark control host. The Massh Parallelized Environment Control software package was installed here, as well as the GemFire client and other cluster control tools.

See Figure 2 and 3 below for more detail on the 4- and 8-node clusters:

Figure 3 - GemFire XD 4-node Cluster

Figure 2 - GemFire XD 8-node Cluster

YCSB Benchmark Database

YCSB is a highly extensible benchmark suite that allows you to change many settings and parameters, as well as develop custom workloads. The default table called usertable was used with 10 fields, each containing 100 bytes. By using the default table, the results can be more easily compared to similar in-memory processing engines and data stores since many of them have been benchmarked against the same table.

Tables in GemFire can be partitioned or replicated. A replicated table keeps a copy of its entire data set locally on every GemFire server in its server group. A partitioned table manages large volumes of data by partitioning it into manageable chunks and distributing those chunks across all members in the table’s server group.

The benchmark table usertable was created both in-memory with no persistence to disk and with persistence enabled. The data set was loaded with different record counts as part of the throughput and scaling tests. By default, GemFire will only persist the data dictionary for the tables and indexes created to disk. In the following DDL used to create the in-memory table, there is no PERSISTENT keyword in the statement so the table will be empty if the system is restarted.

For this benchmark the focus was on partitioning in GemFire, and the benchmark table was partitioned by the primary key YCSB_KEY. Note that unless PARTITION BY is specified at table creation, the table will be replicated by default:

CREATE TABLE usertable
(
YCSB_KEY VARCHAR(100) NOT NULL PRIMARY KEY,
FIELD1 VARCHAR(100),
FIELD2 VARCHAR(100),
FIELD3 VARCHAR(100),
FIELD4 VARCHAR(100),
FIELD5 VARCHAR(100),
FIELD6 VARCHAR(100),
FIELD7 VARCHAR(100),
FIELD8 VARCHAR(100),
FIELD9 VARCHAR(100),
FIELD10 VARCHAR(100)
)
PARTITION BY PRIMARY KEY;

The following is output from 3 queries run on the benchmark table which contains 150000000 records and has partitioning enabled on an 8-node cluster. Column 8 in the output from the third query shows the number of rows per host in the partitioned table:

gfxd> select tablename, datapolicy from sys.systables where tableschemaname=’APP’;
TABLENAME |DATAPOLICY
————————————————————————————–
USERTABLE |PERSISTENT_PARTITION

1 row selected

gfxd> select count(*) from usertable;
1
———–
150000000

1 row selected

gfxd> SELECT host, int(constant_overhead), int(entry_size), int(key_size), \ int(value_size), int(value_size_offheap), int(total_size), int(num_rows) \ FROM sys.memoryanalytics order by host;

HOST |2 |3 |4 |5 |6 |7 |8
——————————————————————————-
gfxd1.zdatainc.local |2 |1519006 |0 |18504262 |0 |20023272 |17261439
gfxd2.zdatainc.local |2 |1517836 |0 |18490010 |0 |20007848 |17248143
gfxd3.zdatainc.local |2 |1635154 |0 |19919154 |0 |21554312 |18581300
gfxd4.zdatainc.local |2 |1635697 |0 |19925768 |0 |21561468 |18587471
gfxd5.zdatainc.local |2 |1752580 |0 |21349620 |0 |23102204 |19915691
gfxd6.zdatainc.local |2 |1751798 |0 |21340094 |0 |23091896 |19906805
gfxd7.zdatainc.local |2 |1635734 |0 |19926214 |0 |21561950 |18587886
gfxd8.zdatainc.local |2 |1752191 |0 |21344876 |0 |23097070 |19911265

8 rows selected

GemFire Disk Store and LRU Row Eviction

For the persistence-to-disk portion of the benchmark, a disk store was created called STORE1. AUTOCOMPACT was set to false so that automatic compaction of old DML operation logs would not impact benchmark processes. As mentioned earlier, data can be persisted to tables on local disk or to HDFS. In this benchmark the GemFire locators and data store servers used the EBS volumes mounted on the local host per the GemFire Cluster Configuration section.

The benchmark table usertable was then created with least recently used (LRU) row eviction. With LRU, GemFire keeps a table’s data use under a specified level by removing the least recently used data to free memory for new data. Table eviction settings can be configured based on entry count, the percentage of available heap, or the absolute memory usage of table data. For the benchmark, entry count (LRUCOUNT) was chosen because the table record sizes were homogeneous.

With partitioning enabled, GemFire maintains LRU entry information on a bucket-by-bucket basis to avoid the performance penalty of managing data across the entire partitioned table:

CREATE TABLE usertable
(
YCSB_KEY VARCHAR(100) NOT NULL PRIMARY KEY,
FIELD1 VARCHAR(100),
FIELD2 VARCHAR(100),
FIELD3 VARCHAR(100),
FIELD4 VARCHAR(100),
FIELD5 VARCHAR(100),
FIELD6 VARCHAR(100),
FIELD7 VARCHAR(100),
FIELD8 VARCHAR(100),
FIELD9 VARCHAR(100),
FIELD10 VARCHAR(100)
)
PARTITION BY PRIMARY KEY
EVICTION BY LRUCOUNT 2 EVICTACTION OVERFLOW
persistent ‘STORE1’ asynchronous;

 

YCSB Parallel Data Set Load

The capability to perform parallel loading is built into the YCSB client load values insertstart and insertcount. For example, 150 million records were loaded in 15 minutes by the 8-node cluster using 2 workload driver nodes connecting to 4 locators, which act as load balancers between the hosts serving as data stores.

In terms of the YCSB workload, the transaction set (the read and write operations against the database) was the primary focus of the benchmark. However, the data load speed was respectable and also allowed observation of the GemFire Locator service load balancing features and network traffic patterns that occurred between the cluster hosts. Figure 3 below is a chart from AWS CloudWatch Metrics which shows network traffic patterns between the cluster hosts and the 2 workload driver instances during a 150 million record data set load to the 8-node cluster:

 

Figure 3 - NetIn/NetOut During Data Load

 

YCSB Transaction Set Run

The transaction section of the workload provides the average operations per second and read/write latency data needed to illustrate throughput characteristics. To determine scalability and performance of the GemFire system, the transaction set run was executed on the following 6 configurations:

  1. 150 million records (~150 GB in memory)
    • 4-node cluster
    • No persistence configured
  2. 300 million records (~300 GB on disk)
    • 4-node cluster
    • Persistence configured
  3. 150 million records (~150 GB in memory)
    • 8-node cluster
    • No persistence configured
  4. 300 million records (~300 GB in memory)
    • 8-node cluster
    • No persistence configured
  5. 300 million records (~300 GB on disk)
    • 4-node cluster
    • Persistence configured
  6. 600 million records (~600 GB on disk)
    • 4-node cluster
    • Persistence configured

 

In both physical cluster configurations, records were loaded until available memory and disk storage were at operational capacity. In the 4-node cluster, this was 150 million records for non-persistent, in-memory data storage and 300 million records for the persistent table. In the 8-node cluster, it was 300 million records for non-persistent and 600 million for the persistent table.

The YCSB client thread and target values were adjusted in increments after a test run on each benchmark configuration above to determine required ratios to achieve thread saturation and reach the performance plateau of the system. Also, operationcount was set at a high value to collect a sufficient sample for sustained throughput measurement. The value set for operationcount ranged between 20 million for a thread value of 10 and a target value of 10000, and 100 million for a thread value of 800 and a target value of 150000.

In all configurations, heap memory storage was used for table data. The heap-size value was set to 57 GB and critical-heap-percentage was set to 99 on each cluster host. Each of the EC2 r3.2xlarge instances serving as the benchmark cluster hosts has 61 GB of memory. Pivotal’s recommendation is that each cluster host have at least 150 GB of memory to take advantage of an off-heap memory configuration.

 

Monitoring with GemFire Pulse

GemFire Pulse is a web application that provides a graphical dashboard for monitoring real-time health and performance of GemFire clusters, members, and tables.

GemFire Pulse was used in embedded mode on cluster host gfxd1 to assist in monitoring during the benchmark. It proved to be very useful, and consumed negligible resources on the EC2 instance. The ability to view CPU, memory and disk usage in real time was helpful in pinpointing bottlenecks and other issues during the benchmark. Figure 3 and 4 below are Pulse Cluster and Data view on an 8-node cluster with 150 Million records loaded with persistence disabled:

 

Figure 3 - Pulse Cluster View

 

Figure 4 - Pulse Data View

 

Figure 5 below is the Pulse Data view with persistence enabled for 300 million 100 Byte records (308.82 GB on disk):

 

Figure 5 - Pulse Data View - Persistence On

Benchmarking Results

GemFire 4-Node Cluster – 150 Million Records (Non-Persistent)
YCSB Workload A Results Chart

YCSB-Workload

GemFire 4-Node Cluster – 300 Million Records (Persistent)
YCSB Workload A Results Chart

YSCB-Workload-2

GemFire 8-Node Cluster – 150 Million Records (Non-Persistent)
YCSB Workload A Results Chart

YCSB-Workload---3

GemFire 8-Node Cluster – 300 Million Records (Non-Persistent)
YCSB Workload A Results Chart

YSCB Workload - 4

GemFire 8-Node Cluster – 300 Million Records (Persistent)
YCSB Workload A Results Chart

YSCB Workload-5

GemFire 8-Node Cluster – 600 Million Records (Persistent)
YCSB Workload A Results Chart

YSCB Workload 6

YCSB Workload A Results Overview Chart

GemFireChart

 

AWS CloudWatch CPU Utilization, Workload A Transaction Set Run
160 Threads, 10K Target Ops/sec, 10 Million Ops, 300 Million Records (Persistence On), 8 Cluster Hosts

AWS Cloudwatch - 1.png

AWS CloudWatch Network In, Workload A Transaction Set Run
160 Threads, 10K Target Ops/sec, 10 Million Ops, 300 Million Records (Persistence On), 8 Cluster Hosts

AWS Cloudwatch 2

AWS CloudWatch Network Out, Workload A Transaction Set Run
160 Threads, 10K Target Ops/sec, 10 Million Ops, 300 Million Records (Persistence On), 8 Cluster Hosts

AWS Cloudwatch 3

AWS CloudWatch Volume Write Ops, Workload A Transaction Set Run
160 Threads, 10K Target Ops/sec, 10 Million Ops, 300 Million Records (Persistence On), 8 Cluster Hosts

AWS Cloudwatch 4

Summary

As mentioned earlier, this benchmark tested certain aspects of GemFire’s performance using a single synthetic workload and was part of a complete profiling of the test architecture. It provided valuable insight into sizing and performance tuning GemFire for client’s applications with OLTP-like workloads on a cloud platform. Following are some of the most interesting observations made during the benchmark.


Sustained Throughput and Latency

Even with a relatively small client/server architecture consisting of 8x r3.2xlarge EC2 instances with no off-heap memory configured, sustained throughput of 65000 operations per second could be maintained with under 1 ms read and write latencies when usertable was populated with 300 million records with persistence enabled (300+ GB on disk, ~37.5 million rows per node).

With 600 million records and persistence enabled (600+ GB on disk, ~75 million records per node), sustained throughput of just under 100,000 operations was achieved with read and write latencies under 2 ms.

A maximum sustained throughput of over 120,000 operations per second was seen on the 8-node cluster with 150 million records in-memory (~18.75 million records per node).

 

Vertical Scaling

There was an ~18% increase in maximum sustained throughput from the 150 million record in-memory transaction run to the 300 million record transaction run with persistence enabled in the 4-node cluster.

In the 8-node cluster, the highest throughput was on the 150 million record in-memory configuration as mentioned above. A combination of a network traffic bottleneck to the workload driver host running the YCSB transaction set and saturated links to the EBS volumes on the cluster hosts were interfering with the higher throughput workloads on the 300 and 600 million record persistent configurations. This was one of the expected outcomes when the higher thread and target values were introduced in the workload transaction set runs on the 8-node cluster as it was loaded to operational capacity.

With GemFire, methods can be used to avoid or minimize these type of issues in cloud architectures. Dependent upon the application, these include strategies such as hybrid peer-to-peer and client/server architecture, server grouping and strategic co-location of data.


Horizontal Scaling

A maximum throughput increase of up to 25% was seen after expanding the cluster from 4-nodes to 8-nodes for the same configuration, and up to 46% when the number of records doubles. Details are below:

150 Million records, 4-node to 8-node, No Persistence

  • 18% Max throughput increase

150 Million records 4-node to 300 Million records 8-node, No Persistence

  • 46% Max throughput increase

300 Million records, 4-node to 8-node, Persistence enabled

  • 24% Max throughput increase

300 Million records, 4-node to 8-node, Persistence enabled on 8-node

  • 24% Max throughput increase

300 Million records 4-node to 600 Million records 8-node, Persistence enabled

  • 25% Max throughput increase


Cluster Expansion

GemFire’s scalable, lightweight architecture lends itself well to cloud deployment. It doesn’t have the dependencies and overhead of traditional RDBMS platforms which have been overhauled to fit into the in-memory distributed computing space. Unlike many of these platforms, nodes can be added and removed at any time with minimal impact to the system.

For example, the cluster was expanded from 4-nodes to 8-nodes with 300 million records loaded into the partitioned benchmark table with persistence enabled, which was a little over 300 GB. Once the additional GemFire members were added to the cluster, the rebalance operation was executed with the system procedure call sys.rebalance_all_buckets();. The rebalancing operation runs asynchronously and took several hours to complete, but this is by design so data movement will not impact other operations on the system.

Partitioned tables can be accessed while rebalancing is in progress, and Workload A was ran during the middle of the operation to determine the impact on performance. The results showed that rebalancing had very little effect on throughput or latency in a run with identical target and thread values set.

Workload A (threads 10 target 20000)
Configuration: 300 Millon records, Persistence enabled, 8-node cluster, Rebalance in progress:

[OVERALL], Throughput(ops/sec), 16228.023199581967
[UPDATE], AverageLatency(us), 512.2296022361539
[READ], AverageLatency(us), 689.8959474089838

Workload A (threads 10 target 20000)
Configuration: 300 Millon records, Persistence enabled, 8-node cluster, No rebalance in progress:

[OVERALL], Throughput(ops/sec), 17942.78119
[UPDATE], AverageLatency(us), 451.9638103
[READ], AverageLatency(us), 627.9891231

The benchmark was performed using GemFire client/server cluster configurations that contained a single copy of the data in a partitioned table. In a production scenario, high availability for partitioned tables could be enabled, allowing a redundant copy of the partitioned table to be distributed in storage buckets across the entire cluster. In GemFire, a bucket is the container for data that determines its storage site, or multiple sites in cases where redundancy is enabled. It is also the unit of migration for used for rebalancing.