Shipping API Monitor – What Does 1 Year Tell Us?

One of our projects here at LexiConn – www.ShippingAPIMonitor.com – just had a birthday.  The website recently passed 1 year of monitoring the FedEx, UPS, and USPS shipping APIs, and we thought – why don’t we take a look at some of the stats and see what we’ve tracked over the last 12 months?

What Does It Do?

ShippingAPIMonitor.com provides real-time tracking of the shipping API systems for rates being returned from the big three – FedEx, UPS, and USPS. It monitors for their system being up and returning valid rates every minute from 2 geographically diverse locations. Any outage is displayed on the site, and after 3 consecutive rate failures, we mark the shipping API as down.

For this post, we’ve summarized the outages from July 1, 2013 through June 30th, 2014.

How Long & How Many?

In the 1 year of tracking we monitored a total of 1,559 minutes of downtime across all 3 APIs, which is just under a total of 26 hours.  26 hours may seem like a lot, but since that is over 1 year it’s just 0.3% of overall downtime.

Luckily most of that was at night, but you don’t want any downtime when it means a customer could fail to place an order and you therefore lose a sale.  Let’s take a closer look at our big 3:

FedEx had 173 outages comprising 1,308 minutes (21.8 hrs) of downtime.  61 of those outages totaling 1,188 minutes (19.8 hrs) were over 3 minutes long, resulting in a “Down” rating on ShippingAPIMonitor.com.

They had 6 outages last longer than an hour, with the longest being 4-1/2 hours.  That outage of was one of their planned maintenance windows, lasting from 10pm Eastern time on 8/3/2013 until 2:30am the following morning.

USPS had 130 outages comprising 250 minutes (4.2 hrs) of downtime.  18 of those outages reached our 3 minute threshold, which accounted for 123 minutes (2 hrs) of their downtime.  Their longest outage was 32 minutes, during the afternoon of 3/26/2014.

UPS had 1 outage of 1 minute in length.  That was it!  It was not even long enough to trigger a “Down” rating on our site, just a “Warning” notice for 1 minute until the next test completed as passed.

What Have We Learned?

We started this project to help our customers when they found shipping errors in their shopping carts, but we never knew how often the shipping APIs really went offline until we started tracking.  During the first month of going live (July 2013) we tracked 18 minutes of outages and then August followed with 2 scheduled maintenance outages by FedEx and a variety of smaller outages which allowed us to tweak our code when needed and confirm everything was monitoring correctly.

We quickly learned that FedEx has a lot of little outages, mostly going unnoticed by merchants since they are short and if a customer did receive a shipping error but tried again they would have been able to submit their order without ever letting the merchant know an error occurred.

USPS also has a good number of outages, but with so many being 2 minutes or less they mostly go unnoticed.

It took over 5 months before we finally tracked an outage with UPS.  What does UPS do that is different than FedEx and UPS?  We can only assume they are load-balancing their API servers with a method where at least 1 location is always (almost!) up and serving rates, as having just 1 minute of downtime in a whole year of tracking is remarkable.

Want to know when shipping APIs go down for your store?  Visit ShippingAPIMonitor.com to view stats or sign up for the newsletter to receive emails anytime an API goes down.  You can even follow us on Twitter for notifications as well.

7 Common Reasons Why Products Don’t Display in Magento

In this post, we will be reviewing some of the more common reasons Magento products do not display in the frontend.

1.  Product updates have invalidated the cache

When using the Magento cache option, a refresh is required after product updates.

System > Cache Management
 
(click to enlarge)

(click to enlarge)

The invalidated cache will be displayed in orange.

Also, if you are running FPC (Full Page Cache), be sure that it has flushed the page where the product is expected to appear.  Some modules require a complete manual flush of the FPC, while others are more automated/selective.

2.  Product is not assigned to a website

Catalog > Manage Products > Website (tab)
 
product-website

3.  Product is not assigned to a category

Catalog > Manage Products > Category (tab)
 
(click to enlarge)

(click to enlarge)

4.  Product is out of stock

Catalog > Manage Products > Inventory (tab)
 
(click to enlarge)

(click to enlarge)

By default, products with a ‘Stock Availability’ of ‘Out of Stock’ (OOS) are not displayed on category pages.  Though, they can be accessed directly via the products URL.

You can change the default setting to display OOS products System > Configuration > Catalog > Display Out of Stock Products (change value to Yes).

(click to enlarge)

(click to enlarge)

 

Once the setting is changed, Indexes need to be rebuilt in System->Index Management.

Also, remember to update cache (System -> Cache Management, etc…).

Reminder: When making changes to system defaults, be sure the scope is set correctly.  This is especially critical if you have multiple sites, stores and/or views…

global_setting_scope_change

5.  Product status is ‘Disabled’

Catalog > Manage Products> General (tab)

Set Status to ‘Enabled’

product-status

6.  Product visibility does not include ‘Catalog’

Catalog > Manage Products > General (tab)

Set Visibility to either ‘Catalog, Search’ or ‘Catalog’

product-visiblity

7.  Category page Display Mode is set to ‘Static block only’ (and the assigned static block does not contain support for product display)

Catalog > Manage Categories > Display Settings (tab)
 
(click to enlarge)

(click to enlarge)

Set Display Mode to either ‘Products only’ or ‘Static block and products’

If you have frequently experienced other scenarios where products have not displayed on Magento catalog pages, feel free to send us the details!

Top Ecommerce Blog Posts and Articles for June 2014

Four great articles worth a read for any ecommerce merchant…

Recipe for Responsive Ecommerce Product Pages in 2014Practical Ecommerce
Some good tips about creating a great responsive website

Deconstructing E-Commerce Search: The 12 Query TypesBaymard Institute
How customers actually search a website

The Landing Page Optimization Guide You Wish You’ve Always HadSearch Engine Journal
A must read for anyone with a landing page from a paid ad

How to Retain at Least 95% of Your Organic Traffic After a Site RedesignQuickSprout
Redoing your website? Read this before going live!

Bounced Emails – Common Issues and What To Do

It happens to everyone once in a while – you send an email and it bounces back undeliverable.  You end up with a cryptic email which may have 30-40 lines of strange information in it, but what do you do to fix it?  Luckily the most common issues are easily resolved and the details of the issue are in the bounce message.

It’s just a matter of knowing what to look for.  Don’t let the long gibberish of the bounce email scare you!  Giving it a quick review will usually show the cause of the bounce:

Quota Exceeded

This is the most common issue and is usually the easiest to determine  as it generally appears near the top of the bounced email:

procmail: Quota exceeded while writing "/var/mail/johnsmith"

You can guess the issue here – the “johnsmith” email account is full.  If that’s your own account  or one of your users then you can login to webmail and delete old email from the inbox, or set the email program to delete old email when it downloads.  Either solution will clear up the issue, but setting the email program to delete old email will keep it clean automatically.

If you were sending to someone outside of your hosting account with LexiConn, then they would need to clean up their inbox with their mail provider.

User or Host Unknown

Very common, and usually easy to find in the bounce message:

550 5.1.1 <webmaster@domain.com>... User unknown

If you see “User unknown” then the receiving email server may have a problem, or it could simply be a typo in the email address.  Check to make sure the email address is correct.  You may also see “Host unknown” which points to an issue with their mail server itself or just the domain name is wrong on the email address.  For example if you send an email to someone at gmail.com but typed in gmaill.com you may see a “Host Unknown” bounce come back.

Large Emails

A less common issue, but usually easy to determine from the text of a bounce message is:

552 5.2.3 Message size exceeds fixed maximum 
message size (20000000)

The number at the end may vary in size, but the issue is still the same – the email was too large for the email server.  In this example the max size allowed is 20MB.  A normal text email would never reach this size, so there must have been a file attached.  The file itself may have been smaller than 20MB, but due to email encoding the total size of the message was over 20MB.

If you’re sending a variety of files in 1 email, the solution may be to simply split your attachments over 2-3 emails, assuming no single file is too large by itself.  Another option would be a forwarding service such as the one by HighTail below.  Their free option allows files up to 250MB to be sent.  The sender can upload a file to the website and it in turn emails a link  to the receiver to download the file:

https://www.hightail.com/lite

The Dreaded Block List

This issue appears a large variety of ways, but the premise of the error is the same – your email was blocked due to the IP address being in a block list with either an ISP or a black list provider.  Here’s a sample of what it can look like in the bounce message:

550 OU-002 (BAY004-MC6F8) Unfortunately, messages from 10.11.12.13
 weren't sent. Please contact your Internet service provider since part of
 their network is on our block list. You can also refer your provider to
 http://mail.live.com/mail/troubleshooting.aspx#errors.

When you see an issue like this, the first thing to do is check to see if the IP address is your own.  To check your IP visit the following website:

http://www.whatismyip.com/

They will show your IP address on the screen.  If it matches the one shown in the bounce message then your local IP is on the list.  If you have a business-class internet service you may be able to contact your ISP and have them work on getting the block cleared.  However if you’re emailing from home and have a regular residential internet service provided by someone like Comcast or Verizon there is usually no chance of ever getting the IP removed.

A better solution is to change your email program to send through your hosting account with us and not through your ISP.  When sending through your ISP your email goes out with your local IP and is then seen by all the email servers.  If you send email through your hosting account then your server’s IP with us is seen instead.

We have a section in our Knowledge Base here with articles for how to configure various email programs:

http://support.lexiconn.com/kb/categories/E-Mail/Configuring+Email+Programs/

Basically you want to check to make sure your email program sends using “mail.your-domain.com” with your email username and password.  A setting usually called “Outgoing Authentication” or “My outgoing server requires authentication” must be on which tells your email program to login when sending email.  This will allow your email to be sent through your hosting server.

Unfortunately not all email providers look at the sending IP correctly.  Even though you may have configured your program to send through us, some blacklists still look at the original source IP and block it.  If this is happening for you or if the IP in the bounce doesn’t match your local IP please forward us the bounce message and we will research it.

These are just a few of the many reasons emails may bounce.  Luckily the most common reasons are simple to resolve, but if you’re hosted with LexiConn and have not been able to find a cause for the bounce please don’t hesitate to forward it to us and we’ll take a look.  Be sure to include a copy of the bounced error as it will help us track down the cause, and (if it’s not already in the bounce email) including details such as the sending email address, the to email address, and the date/time the email was sent will help as well.

Magento – Faster Than a Speeding Bullet

speeding-bulletMagento is a popular, powerful ecommerce solution. It’s also quite slow out of the box.

How slow?

The metaphor “as slow as a snail” comes to mind. (and that’s being kind…)

However, with a few changes and some know-how, Magento can be as fast as any other ecommerce application, and likely faster than most.

Here’s what we do for our hosting clients to make Magento fly…

Out of the box vanilla install

To put this into perspective, we first installed a Magento Community 1.9.0.1 version along with the sample data for a fully functional demo store. For this install, we did not perform any optimizations, just installed it as-is on a regular server.

We tested the “Sale” page of the demo, which is a category page with 4 products on it. This is a good representative page for any ecommerce store.

  • The initial page load came in at 3.9 seconds.
  • The repeat view had a page load time of 3.5 seconds.
  • The Time to First Byte (TTFB) was 1.8 seconds

Here’s the Webpagetest.org results:

Unoptimized Results (click image to enlarge)

Unoptimized Results (click image to enlarge)

The load time of nearly 4 seconds is very slow to render a web page. The TTFB of almost 2 seconds is an eternity. Ideally you want to be under 0.25 seconds.

As you can see, Magento out of the box with no optimizations is a dog.

Optimized Magento Demo on a Virtual Private Server (VPS)

The speeds you saw above with the vanilla install are what you might find at a host that does not specialize in Magento. In fact, that may be considered “fast” on some hosting servers (which is sad).

What could you expect if you were to host a Magento store with us? To answer that, we installed the same Magento 1.9.0.1 version with sample data on a standard VPS (that has other live Magento clients on it). We applied our standard optimizations that we do for any Magento store we host.

Here’s what we did:

  • Latest Apache web server running in threaded mode for the most efficient environment to scale well.
  • Latest PHP 5.4 with proper optimized settings running as php-fpm for the best performance.
  • APC Op-code caching installed and optimized for Magento (size, speed, fragmentation monitoring).
  • MySQL 5.5 with optimized Query Caching and InnoDB settings.
  • .htaccess adjustments for proper gzip compression and proper client side caching
  • Magento admin settings adjusted for caching, indexing, ideal cron, and log rotations.
  • Two extensions installed for a Full Page Cache and Site Optimizer (combining css/js files, compressing images, minifying html code)

Optimized Results

  • The initial page load was 0.9 seconds
  • The repeat view page load time was 0.38 seconds
  • Initial Time to First Byte (TTFB) was 0.16 seconds

Here’s the Webpagetest.org results:

Optimized VPS (click image to enlarge)

Optimized VPS (click image to enlarge)

We were able to make the initial page view 400% faster. On a repeat view, this increased to almost 1,000% faster!

TTFB dropped by a factor of 10.

Essentially, you can browse this ecommerce Magento site with pages loading in well under a second. Almost seamless browsing from page to page.

Feel free to try the site yourself: magento-demo.lexiconn.com

Magento can (and should) be fast

We were able to take a fully populated demo of Magento and make every page fully load in under a second. Each page is rendered viewable in under 0.2 seconds.

That is how it should be.

Unfortunately, many web hosts don’t offer this type of speed for their Magento clients. Overloaded servers, improperly tuned environments, and poorly executed set ups can contribute to a Magento store that can take up to 20 seconds for pages to load.

If your store is not running as fast as you’d like, get in touch with us. We can help you migrate your Magento store with NO downtime, no re-work needed, and we can enable your store to complete more sales due to a vastly improved experience for your customers. All of this in a PCI compliant environment.

No two Magento stores are alike. We personalize each environment to meet the needs and specific requirements of each installation. This is “the magic” that can set your store apart from the rest, and allow it to be “faster than a speeding bullet”.

photo credit