1-705-797-2105 |
Avetti Commerce on Skype
|
Chat | Talk to an Expert
|
Espanol |
...
Menu

Ecommerce Performance Benchmarking

Overview

We are testing all the Java open source ecommerce solutions that we can find and one that has a non open source license with partially available code. We also tested 3 of the leading PHP solutions.

Why does speed matter?

Google advises that rankings are partially based on site speed. Amazon found that every 100ms a site is slow costs 1% in sales. You can cluster to get more speed but every server you add adds cost and the real bottleneck eventually will be the database so at the most you can do 10 servers hitting 1 database.

Therefore with more speed you have less costly hosting, higher organic search rankings and more sales.

Testing Methodology

We tested using Blitz.io which reads one page only and reads only the HTML of the page. It does not run Ajax or pull images, but it can simulate hundreds of unique users hitting that one page on the site which is a good test since a common ecommerce use case is sending an email campaign or launching an advertising campaign which bring many users to one particular landing page on a site at one time. Ecommerce load is normally spikey so having the ability to stay fast is critical. Images should come from a content delivery network so normally would not come from the website anyway so the fact that Blitz.io does to read the images is not an issue.

The times below are the best times we found before timeouts started occurring. We basically looked at the hit rate chart and found the highest value before time outs occurred and recorded the 'Best Results' as the hit rate (pages per second) and the number of concurrent users that generated it, then we checked on the response time chart to see the corresponding response time for that hit rate. An assumption we make is that one should be able to get that level of throughput continually. For some of the solutions we ran tests and got timeouts then reduced the number of concurrent users until no timeouts occurred and then used that test for the results.

All tests were done on an AWS (amazon web services) cloud 64 bit Centos Linux server with 3.7 GB of RAM and 2 ECU (1 vCPU with EC2 compute units - an indicator of processor speed).

uname -a
Linux xxx.avetti.ca 2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Oct 6 19:24:09 BST 2011 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue
CentOS release 6.4 (Final)

cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2650 0 at 2.00GHz
stepping : 7
cpu MHz : 1795.672
cache size : 20480 KB
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc up rep_good aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat epb xsaveopt pln pts
bogomips : 3591.34
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

We also tested a few on an m1.large as well.

Performance Results

Here are the Best Throughput Results. All tests are on an m1.medium unless otherwise noted.

PHP Ecommerce Solutions

  • Magento Commerce 1.7.0.2 - 6 item page, with facets
    • m1.medium = 1 page per second, 1 user at 350ms
  • Prestacart 1.5 - 10 item page
    • m1.medium = 2 pages per second, 1 user at 500ms
  • Open Cart - 1.5.5.1 - 6 item page, with facets
    • m1.medium = 11 pages per second, 14 users at 350ms

Java - Spring/Hibernate - Ecommerce Solutions

  • Broadleaf Commerce v2.2 - 18 item page, with facets
    • m1.medium = 6.5 pages per second, 9 users at 800 ms;
    • m1.large = 13 pages per second, 17 users at 300ms
    • And for the 4 item home page
    • m1.medium = 26 pages per second, 39 users at 576ms
  • Yes-Cart v1.0 - 10 item page, with facets
    • m1.medium = 13 pages per second, 22 users at 600ms
  • Avetti Commerce v8.3.3 - 12 item page, with facets
    • m1.large = 260 pages per second, 260 users at 50ms
    • (We can probably crank up the load higher on the m1.large as we got 280 on an m1.medium).
    • m1.medium = 280 pages per second, 380 users at 140 ms

Java Struts/Jsp - Ecommerce Solutions

  • Konakart. - 6.5.1.2 - 4 item page, with facets.
    • 32 pages per second, 46 users at 250ms
    • Unfortunately we cant find a page with more items and the number of items per page will affect performance
    • Also Konakart is not an open source license and you don't get all the source code.
  • soft slate - 3.2.7 - 9 item page, no facets
    • 16 pages per second, 26 users at 600ms
    • Shopizer v1.6
      • not updated since may 2011
      • could not get it to compile
      • dead?
    • Jadasite
      • not updated since may 2012
      • seems no demo store is included
      • dead?

Java - Other - Ecommerce Solutions

  • Apache OFBiz - v12 - 13 item page.
    • 1 page per second, 1 user at 500ms.
    • Strange how the checkouts have options to keep addresses and phone numbers private.
    • The last thing a merchant wants to do is make it too easy for a customer to never be contacted. I can't see how this feature can help sales.
    • Note we had to run this using its internal Tomcat as we could not figure out how to make a war we could install.
    • This might have affected its speed.

Conclusions

First, Java is clearly faster than PHP. It's also surprising how slow some well known PHP solutions are.

If you want to start with an open source license and then have the option of moving to get enterprise level support for larger customers that demand it, then you basically have only 3 choices:

  • Magento Commerce
    • Enterprise License is $15,000 per year
    • Includes Support
  • Broadleaf Commerce
    • Enterprise license is $20,000 to $30,000 per year
    • Includes Support
  • Avetti Commerce
    • Enterprise license is $5,000 per year
    • Support is $5,000 per year
  • Konakart
    • Enterprise License £3,600 (about $5,000) per year
    • Support is $5,000 per year
    • Source code is limited, even with the enterprise edition to: the Struts action classes and forms, the JSPs, the payment modules, order total modules, shipping modules and the GWT One Page Checkout code. Also, it's not clear if that source is on another license or still on the LGPL license which means any changes you make are not private.

If you need speed then you are down to:

Avetti Commerce

Which also:

  1. Is Java based on the modern spring/hibernate/Solr framework.
  2. Is fully featured
  3. Has available SLA support, includes PCI Secure software, and Easy ERP integration software.

Performance Testing Details

If we're wrong on any point in our load testing results, please email me at ecommerce@avetti.com so we can correct this.

Every test was done on using Apache Tomcat for Java solutions and MySQL on a Linux based 3.7 GB, 2 ECU m1.medium AWS server. If available, we turned on caching in the application to get as much speed as possible.

Broadleaf 2.2

The URLs in the images say broadleaf3 but as BL3 is their dev branch and not stable we could not get that branch to compile on the day we tried. We'll try again later. We ended up putting broadleaf 2.2 (latest stable version) on the the server we named broadleaf3.avetti.ca.

We tested the home page of their demo store, which has only 4 items on it and is very simple.

Got 7.6 pages per second.

We increased the number of caching items to 100,000 and improved performance from 7.6 pages per second and now get 13 pages per second, again on the very simple home page.

Best Result: Home page with 4 items. 26 pages per second, 39 users at 576ms.

Now lets test a page that has more than 4 items on it. Their hot sauces category in the demo store has 15 items, plus 3 featured products.

Lets try 20 concurrent users over 60 seconds:

From the above graph, there are about 7 pages per second at about 12 users and 700 ms per page before we got timeouts.
Lets lower the number of concurrent users.

We lowered it to 10 concurrent users and are getting 2.71 hits per second.

Best Result: 18 item page from the graph. We got 6.5 pages per second at 9 users and 800ms latency before we got timeouts.

On an m1.medium, from the charts, it looks like anything more than 6.5 concurrent users and Broadleaf starts giving timeouts for this more complex page.

We next tried on an m1.large, which has 4 ECU of processing power and 8 GB of RAM.

Best Results: On an m1.large, 13 pages per second, 17 users at 300ms. So an m1.large helped.

Prestashop 1.5.4.1

http://broadleaf2.avetti.ca/prestashop/index.php?id_category=3&controller=category&id_lang=1

Note that we installed Prestashop on this broadleaf2 URL.

The Prestashop store is much more complex, but again only 3 items on the page, plus 1 special on the right.

The test showed we got 0.5 pages per second for 1 concurrent user. Extremely slow.

Best Results: From the charts, the best we got is about 2 pages per second for 1 user at 500ms.

Magento 1.7.0.2 Community Edition

We used their demo store and chose the cell phones category that has 6 products on the page and some search facets.

Best Result: 1 page per second, 1 user at 350ms.

Note: We tried higher numbers of users, but got timeouts so we used this test for our results.

Yes-Cart 1.0.0

We used the keyboards and desktops category from their demo store that has 5 products, 5 related products and some search facets. We got 6.53 pages per second.

Best Result: 13 pages per second, 22 users at 600ms on a 10 item page with facets.

Avetti Commerce 8.3.3

Load tested on an m1.large. This is a 4 ECU system, which is about twice as fast as an m1.medium.

We chose our camcorders category as it's complex and has as about 12 products with many search facets.

Here are the Blitz.io load testing results:

We got 137 pages per second averaging 36 ms.

Best Result: m1.large (8 GB of RAM and 4 ECU) - From the charts, the best we got was 260 pages per second with 260 users at 50ms.

The above test was done on an Amazon m1.large.

Next here is the same test using a clone of that server on an m1.medium (3.7 GB of RAM, and 2 ECU).

Best Result: m1.medium, 220 pages per second, 220 users at 50ms. 12 item page with facets on an m1.medium.

Max Hit Rate was 222 pages per second.
Max Response Time was 51ms at 242 users.

Lets try a test with 350 unique users:

The Max Hit Rate was: 280 hits per second.
The Max Response Time was: 176 MS at 338 users.

Now lets try 450 users and we see we have a timeout when we have 321 concurrent users.

The Max Hit Rate was: 297 hits per second.
The Max Response Time was: 432 ms at 435 users.

Best Results: m1.medium, 280 pages per second, 380 users at 140ms.

We next tried with a c3.xlarge which is a new SSD based server from Amazon. 30 cents an hour vs 24 cents so not much more but it has much faster IO.

We still have the same speed but the response rate is much much better. 7ms vs 53 ms at 280 users. We now recommend the c3.xlarge on all new installations of Avetti Commerce on EC2.

Best Results: c3.xlarge, 279 pages per second, 300 users at 6ms response rate.

Konakart 6.5.1.2

17 pages per second, at 273ms on a 4 item page (seems no page in their demo store has more than 4 items).

Best Result: 32 pages per second, 46 users at 250ms on a 4 item page with facets.

Unfortunately we cant find a page with more items and the number of items does make a difference. Also, Konakart is not an open source license and you don't get all the source code.

Softslate 3.2.7

We installed the demo store and are showing 9 products in a camcorders category we created. There are no search facets.

Here is the page we hit:

Here are the test results:

10 pages per second at 569 ms per page and 35 users. At 40 users, we got timeouts.

Best Result: 16 pages per second, 26 users at 600ms

OpenCart 1.5.5.1

Here are the results:

2.56 pages per second at 104ms and 5 users. 6 items on the page with some search facets.

We increased the users to 10 and still no timeouts. Got 4.23 pages per second at 188ms. Seems OpenCart is the fastest php solution.

Increased to 15 concurrent users and now we get timeouts around 12 users.

We got 6.02 pages per second at 200ms which is still very good.

Best Results: 11 pages per second, 14 users at 350ms.

OFBiz 12.04.01

We installed the OFBiz binary, ran it using the tools/startofbiz.sh script and proxied to it from Apache. The script runs an embedded copy of tomcat.

We are surprised to see so much logging occured when we hit the gizmos category page.

Also, the results are not great so I am not sure if we are doing something wrong.

Here are the results:

Best Result: 1 user, 1 page per second at 500ms. Surprisingly slow.