We are excited to announce that on September 12th, 2014, we will be migrating the CTO blog to a new version of the blogging platform.

Posts Tagged ‘ hadoop ’

joshsimons

Hadoop Performance on vSphere 5.1

April 19, 2013
By
Hadoop Performance on vSphere 5.1

We’ve just published a third Hadoop performance paper, written by VMware performance expert Jeff Buell, which looks in detail at the relative performance of a bare-metal 32-node Hadoop cluster compared to a range of virtual clusters with up to 128 VMs. The executive summary is that while we saw a 13% performance degradation in a head-to-head comparison of a 32-node physical cluster against a 32-VM virtual cluster (one VM per host) running on the same hardware and running the same tests, virtualized performance can be increased significantly — to the point where virtualized Hadoop actually runs a bit faster than physical — by increasing the number of VMs per host. We’ve seen this effect before with Hadoop...

Read more

joshsimons

HPC Performance in the Cloud: Status and Future Prospects

October 1, 2012
By
HPC Performance in the Cloud: Status and Future Prospects

I spoke last week at ISC Cloud 2012 in Mannheim, Germany about the performance of HPC applications in the cloud, citing results from several studies. I have summarized the talk below and my PDF slide deck is available here . For full details of ISC Cloud 2012, I recommend the coverage at  HPC in the Cloud . I gave my talk to share some specific data about HPC performance in virtual environments, but I first described what new capabilities can be gained by virtualizing an HPC environment. I felt this was necessary because in HPC discussions it is often the case that...

Read more

steveherrod

GigaOm Structure – Big Data in the Cloud

June 20, 2012
By

I attended GigaOm Structure this week in San Francisco, and it was another successful event with cloud bursting at the seams! I gave a talk about the real impact of Big Data when it meets cloud computing, which is an important topic for our customers.

Read more

@richardmcdougll

Project Serengeti: There’s a Virtual Elephant in my Datacenter

June 12, 2012
By
Project Serengeti: There’s a Virtual Elephant in my Datacenter

Introduction There’s no question that the amount of value being extracted from data is increasing – almost every customer I speak with is building new technology to gain new or competitive insights from tapping large volumes or rates of data. In the last few posts, I have introduced VMware technologies and products that provide data services to new applications. We see four major axes along which data requirements are stretching the limits of traditional approaches to data analysis: Big Data – The need to store and compute against hundreds of gigabytes of unstructured or semi-structured data Fast Data – The increasing need for low latency interactions with large sets of data, often driven by today’s mobile and social apps. Flexible Data – The need...

Read more

@richardmcdougll

Cetas (VMware) receives prestigious 2012 TiE50 award!

May 18, 2012
By
Cetas (VMware) receives prestigious 2012 TiE50 award!

  As a great validation of VMware’s decision to acquire Cetas , that team has been honored with the TiE50 award in the Software category. This award acknowledges their achievements and underscores the recognition we are getting in the Big Data Analytics space. We are thrilled to receive this prestigious award and it is a solid recognition of all the innovative and hard work done by the team. TiE is a leading entrepreneur-focused organization doing a tremendous job of fostering creativity and innovation amongst entrepreneurs while identifying the technologies and companies that rise to the top in their respective focus areas. More and more business users are demanding instant...

Read more

joshsimons

Video: Why Hadoop Can Run Faster on vSphere than Native

April 11, 2012
By

In this video interview, Jeff Buell, one of VMware’s senior performance engineers, shares his perspectives on the importance of Big Data for customers and for VMware. He also addresses the issue of why his Hadoop testing showed significantly better performance running Hadoop on vSphere when compared to native in some instances. The video (running time 6:20) is here .   The full white paper detailing Jeff’s Hadoop performance testing done in partnership with AMAX and Mellanox is here .

Read more

joshsimons

SC11: Virtualized Hadoop

November 21, 2011
By

Abhinav Chawade from AMAX gave an excellent talk on virtualized Hadoop performance in Intel’s booth theatre on the exhibit floor during SC11. He covered the results detailed in Jeff Buell’s whitepaper, A Benchmarking Case Study of Virtualized Hadoop Performance on VMware vSphere 5 , which was created as a collaborative effort between AMAX, VMware, and Mellanox.   Big Data was a big part of SC this year with numerous papers on Hadoop and related technologies. Several papers discussed running Hadoop in a cloud environment, perhaps most notably Purlieus: Locality-aware Resource Allocation for MapReduce in a Cloud , which I found very interesting. The abstract is produced below. “We present Purlieus, a MapReduce cloud resource allocation system aimed at...

Read more

joshsimons

Virtualizing Big Data

October 26, 2011
By

Analysis of large-scale, often unstructured data is becoming increasingly important within both the Enterprise and the HPC community. This is perhaps one of the most apparent areas where the convergence of HPC and Enterprise requirements can be seen as the tools and algorithmic approaches required are often the same or very similar. I imagine, for example, that the large-scale, graph-oriented “social network” analyses done by companies like Facebook are quite similar to the “anti-social network” analyses done by Homeland Security and the Intelligence community. Unsurprisingly, many VMware customers are interested in running Big Data workloads and are looking for guidance about how best to do this in a virtual environment. To help, we have published a whitepaper that examines Hadoop performance using local...

Read more

joshsimons

Our Joint VMware / AMAX HPC Collaboration

April 27, 2011
By
Our Joint VMware / AMAX HPC Collaboration

I am excited we have now started our joint HPC exploration with our partner, AMAX . Based on an initial meeting on the show floor at VMworld in San Francisco last year, we decided to work together to examine several aspects of virtualized HPC of mutual interest. Areas where we see converging requirements between HPC and Enterprise customers are of particular interest to VMware as an Enterprise software company looking at broader markets and to AMAX as a dynamic computing solutions provider to HPC and Enterprise , and now Cloud customers. We are starting with Hadoop since scale-out data analytics is rapidly becoming an important workload in the Enterprise while Data Intensive Computing is simultaneously rising...

Read more