Introduction

Time to first byte (TTFB) is how long it takes a server to generate HTML of a requested page. 1-2 seconds is an acceptable TTFB of an uncached page. Anything above it could be a sign of a performance issue.

In this guide I’ll show you how to use a PHP profiler to improve TTFB of an Adobe Commerce site (or any other PHP based site). As a Magento 2 developer I used this technique to optimize numerous slow pages and it helped a lot.

What is Tideways and how to use it

Tideways is a commerce version of an abandoned XHProf PHP profiler. Once it’s set up it gives you a trace diagram with function call names and respected times it took to execute them.

Tideways has a free trial period, you don’t need to provide credit card details to sign up. All you need is an email address. It gives you 14 days of trial use but the trial period length might change in the future, check with their website.

Once you registered for a trial, you’d need to install a php extension and a daemon. There are instructions on tideways website but you might need expert help.

Run a manual trace

Turn off Magento 2 full page cache and find a page that you think is too slow. Head over to Traces > Trigger trace and click on Generate Token:

(click to enlarge)

Copy the token, append it to the page URL and fetch it with a browser. Wait 2-3 seconds and your trace is going to be available for analyzing.

Analyze trace output

Diagram has several tabs: Timeline, Summary, Callgraph and Bottlenecks:

(click to enlarge)

TTFB is 3.32 seconds. Let’s see if there is any room for improvement.

Analyzing a Callgraph

Callgraph provides a waterfall diagram.

(click to enlarge)

Scroll down and analyze every function call, there should be something that contributes to that 3+ seconds TTFB.

In our case, Topmenu::getMegamemuhtml needs further investigation. This function generates top menu html and its output could be cached to save 0.6+ seconds.

Here is a screenshot of a trace with mega menu cached:

(click to enlarge)

TTFB is 1.48s now.

There is another function to improve – Homepage::getProduct, it’s selected on the screenshot above. The function takes 0.8s to be executed and it’s all because of 27 load model calls. Model loads are expensive and should be substituted with direct attribute fetching whenever possible.

There might be other slow calls, you just need to go over a tree diagram and inspect every node.

There could be slow SQL queries. You need to enable SQL profiling in General Settings:

(click to enlarge)

After that any slow queries will be visible under the Bottlenecks tab.

Summary

A good code profiler can help you speed up a site and save time and money. From my experience 14 days is more than enough to figure out most performance issues.

The profiler can be used on a live (production) site, it doesn’t slow it down or cause any problems, it works with PHP 8. Let it run for a couple of hours and then inspect traces under Project > Traces.

…

About the author: Konstantin Gerasimov is a Magento certified developer with Goivvy.com. He specializes in speed optimization, migration and general Magento support.

According to Google, around 32% of visitors abandon a page that takes more than three seconds to load. That alone is a huge blow to your potential conversions.

We are organizing an exclusive invite-only masterclass on the “Ultimate 3-Step Process on How to Achieve a Page Speed Score of 100/100” in partnership with Growisto, a Digital Growth company that specializes in technology and marketing services.

Key learnings:

• What is Google Page Speed Score and Core Web Vitals? Why do they matter?
• What are the different stages that matter in this process? A step-by-step plan of action for page speed score improvement
• What is the right approach to increase the page speed of my website?
• What is a good score to target? At what stage should I focus on this?
• What kind of teams, effort, cost, and time it will take to get results?
• How does it depend on my current technology stack? Should I think about upgrading my tech stack?
• How to manage the requirements of different teams such as, marketing, branding and others while optimizing core website vitals and page speed score?
• How to track progress and improvements? How long does it ideally take to start noticing the improvements?
• How can we measure the ROI of the overall efforts and measure the business outcome?

Schedule:

Date:  27 April 2023, Thursday
Time:  3 Pm EST/12pm PST
Where – Zoom (Webinar link will be shared over email on successful registration)

This webinar is an exclusive session for Founders, CXOs, Directors, Head eCommerce Director/VP Technology, CTOs, CPOs, IT managers, eCommerce managers, Tech leads, Product managers, and Engineering leads with first-in-time bookings only.

Save Your Spot Now! 

—– Register Here—-

About the Speakers:

Robert Mangiafico is the CTO of LexiConn, a fully managed hosting provider that specializes in e-commerce hosting with Magento and WooCommerce. He has previously created a platform for email reception and delivery at-sea for US Navy submarines. His expertise lies in web servers, database servers, networking, e-commerce, internet security, ShopSite, and Magento.

Gunjan Agrawal is the Chief Product Officer at Growisto and has 15+ years of experience in technology and product development. He has served as the Director of Products at Oyo Rooms and has successfully built and exited three ventures – Logic roots, Menufry.com, and Career Point Edutech Ltd.

Pritesh Mittal is the COO & Co-founder of Growisto, a Digital Growth company that specializes in technology and marketing services. A second-time entrepreneur, Pritesh in last 8 years has helped 150+ brands increase their digital growth and online sales. He has experience in Tech and Marketing Consulting, Product Marketing, SEO, Digital Marketing, and Conversion Optimization.

When it comes to page load speed, every second counts. In fact, Google research shows that when loading times increase from one to three seconds, the probability of a bounce (the visitor leaving right away) increases by 32 percent. Hence, it is important to optimize your site speed in order to enhance customer experience and lower your bounce rates. A faster page speed would also impact your search engine rankings. One of the most recommended ways to check and optimize your page speed is through Google Page Speed Insights.

For one of our clients, who is a leading manufacturer of safety signs and labels in the US, we achieved a perfect score of 100 on Google Page Speed Insights. The client’s website consisted of around 150k products and over 300k pages. Both teams worked meticulously to figure out the target areas, key requirements, and created a step-by-step roadmap to work efficiently. Our experts implemented each step with precision to take the page speed score of the website to 100.

In this article, we will be discussing some key steps that the Growisto team implemented to achieve a perfect score of 100:

1. Migration to Headless Commerce: We migrated the traditional monolithic architecture of the website to headless solutions using Magento backend & ReactJS frontend. Adopting a headless approach gave us a lot of flexibility to enhance customer experience, improve parameters of the page speed score insights, and to increase personalization. The migration to the headless system itself helped us push the page speed score above 85.
   .
2. Implement full-page caching: Full-page caching improves response time and reduces the load on the server. Without caching, each page might need to run blocks of code and retrieve information from the database. However, with full-page caching enabled, a fully-generated page can be read directly from the cache. For most websites, it will be a significant performance boost to enable the full-page cache.

   However, if your website has ten thousand or more pages, then full-page caching becomes very complex and generally not possible. In those cases, you should have a fallback caching mechanism where you just cache the data. In such cases, less frequently visited pages will not have full page caching, but the data for those will be cached.

   Now that you are done with caching, there might be some changes you want to make to the data. To make this process simpler, you should automate your caching mechanism. Specifically, if you have a large number of products and categories. Otherwise, updating the data would be a very difficult task for the product team.
   .

3. Set Up Content Delivery Network (CDN): A CDN simply collects all your site data like images, CSS, JS and saves it on their servers spread across the world. So when a user makes a request, it can deliver it from the nearest server with respect to the user. Having a CDN with a good network in the region where your customers are, is always better. Also, CDN should help with image handling, as mentioned in the points below, which is also very helpful. Moreover, you can also implement full-page caching in the CDN itself, so your page becomes faster.
   .
4. Add placeholders and set up explicit dimensions for all images: Images make up a large portion of the page elements, so using heavy images increases the page weight and results in slow loading speed. In order to optimize your images, you need to ensure that they are properly sized for your page and efficiently coded.

   Apart from this, you need to set explicit dimensions for all your images. If you set these dimensions beforehand, the other content on your page won’t be affected, and the image will have sufficient space to load while the user reads the rest of the data. This will help drastically reduce CLS.
   .

5. Implement on-the-fly resizing and optimal format of the images: Another important aspect to keep in mind is the automatic resizing and compression of the images. In case an image is larger than the standard aspect ratio, it should be able to resize itself and fit in the area provided. For example, if the aspect ratio set is 400*300 and the original image has size of 400*400. Then it should be able to automatically resize and deliver to the frontend with the size of 300*300 in order to maintain the actual image ratio. This way the structure of the entire page would remain intact.

   Along with resizing, the images should be delivered in the optimal format, such as WebP or other latest formats based on browser support.

   Both of these can be done easily through CDN, so if your CDN supports them, we should configure them in CDN only.
   .

6. Add lazy loading for below-the-fold content: This means that instead of making the browser load every content on a page before displaying the above-the-fold content, it will only load the ones that are immediately visible. Less loading before the page becomes visible means better performance, which is why Google recommends this method.
   .
7. Optimize the SVG icons used on the headers: Merge the smaller icons to create a bigger file in order to get a decent size. It enables you to create an entire file for the icons and not individually save and load them separately. This will reduce multiple network calls and thus save time loading the page.

Other Google Suggestions: Apart from the ways listed above, Google Page Speed Insights also recommends several ways to optimize your site speed. All you have to do is, put your page/site link in the page speed bar and hit analyze. Once you do that, you will see several recommendations such as, Defer offscreen images, remove unused CSS, etc, pop up. You can also follow these steps to achieve a higher page speed.

Moreover, Google Page Speed Insights takes into consideration a few metrics to analyze the page speed of a website. To reach an optimal page speed, you need to move all your metrics towards the green region from the red one (as shown in the picture below). Once all your metrics are green, you have achieved a good page speed for your website.

All in all, mentioned above are some of the most important points that we kept in mind in order to optimize the page speed for our client and achieve a score of perfect 100 on Google Page Speed Insights. Some of these might seem difficult to implement, but, with the right technology partner, you can surely increase your website speed and improve your user experience.

…

About the author: Growisto is a Digital Growth company that specializes in technology and marketing services. Based out of US, and India they have worked with over 300+ brands across a gamut of sectors, including Puma, Tata Cliq, and Victorinox. Their experienced team of 125+ members and technology-first approach has helped them provide high-impact solutions to their clients. They have expertise in custom product development, website development, and digital marketing.

Magento 2 sites make their platform version public. You can visit ‘domain.com/magento_version’ and get a message like this:

Magento/2.4 (Community)

You can go to your site, append /magento_version and wonder why Magento developers team made it public.

I consider it a security risk and here is how to fix it.

All you need to do is to disable Magento_Version extension:

php bin/magento module:disable Magento_Version

You might want to recompile afterwards:

php bin/magento deploy:mode:set production

That’s it. No more Magento version being displayed on your website. 

…

About the author: Konstantin Gerasimov is a Magento certified developer with Goivvy.com. He specializes in speed optimization, migration and general Magento support.

If you have been seeing very slow pages in your admin panel lately for Magento 2 store owners, or random spinning icons while waiting for pages to load in the admin, it may be related to a change Adobe made to some of their JavaScript files.

Specifically, if you have the feature on for “Admin Usage Tracking”, which is defined as:

Grants permission for Adobe to collect usage data to improve the experience of using the Admin, and related products and services. Allowing data collection also enables In-Product Guidance which is designed to bring interactive content such as help, tool tips, walk-through guides, onboarding information, feature announcements, and more to the Admin. Individual administrators are not identified in usage data. Options:
Yes – Allows data collection and enables In-Product Guidance.
No – Does not allow data collection nor enable In-Product Guidance.

You will want to disable this feature under:

Stores -> Settings -> Configuration -> Advanced -> Admin

Once you do that and flush the Magento cache, it should fix these randomly slow loading pages until Adobe can fix their local JavaScript issues.

Adobe / Magento released an emergency patch on Sunday February 13th for all Magento 2.3.3-p1 and higher stores. The Security Bulletin (APSB22-12) describes a remote code execution (rce) vulnerability for Magento 2 stores (both Commerce and Open Source).

An RCE allows a hacker to run code or upload a file on a store without needing any special privileges. It is the worst kind of vulnerability for ecommerce software. Adobe indicated that it was already being exploited “in the wild”.

The patch is a small 2 file adjustment that stops the vulnerability through better sanitizing inputs. The fix is seamless and does not cause any issues in M2 stores.

All Magento 2.3.3-p1 and higher stores hosted with LexiConn have been patched against this vulnerability. We have not detected any active exploits that used this entry point on our network.

If you have any questions about this patch, please let us know.

If you are using Magento 2 Commerce / Adobe Commerce and make use of the Gift Card feature, there is a simple fix that can speed up the use of gift cards in the cart (both checking the balance and applying them).

The database for gift cards, specifically the table magento_giftcardaccount, is missing a crucial index that can cause very slow gift card usage if your store has a lot of gift cards.

The column “Code”, which is the gift card redemption code, does not have an index on it. There are queries that run when a customer checks the card balance, or attempts to apply the card in the cart of the format:

SELECT COUNT(*) FROM `magento_giftcardaccount` AS `main_table` WHERE ((`code` = 'XYZ'))

SELECT `main_table`.* FROM `magento_giftcardaccount` AS `main_table` WHERE ((`code` = 'XYZ')) LIMIT 1

Without an index, a full table scan must be performed for these 2 queries. If your store has thousands (or millions) of gift cards, these queries can take 3-5 seconds per query, which results in a delay of 6-10 seconds in the cart when the customer tries to use a gift card!

Adding a simple index to this column (or even a UNIQUE index as no 2 codes should be the same) will make these queries complete almost instantly. This means when a customer enters a gift card in the cart, the result is immediate, which is a win for usability!

We added an index to Code for one client who makes extensive use of gift cards, and dropped the time in the cart when applying a gift card from 8 seconds to 0.004 seconds.

A simple fix to drastically speed up the gift card feature in Magento 2 Commerce edition.

The latest zero-day Apache log4j vulnerability has surprised the online community with how pervasive this issue is in many systems and software packages. It allows hackers to run commands on any server that is running this vulnerable software.

Here at LexiConn, we have patched all vulnerable applications like SOLR and Tomcat, or have verified the versions in use are not vulnerable to this issue.

For our Magento clients, Elasticsearch is not susceptible to the remote code execution vulnerability, so there are no issues related to the use of Elasticsearch. We will be updating all Elasticsearch v 6/7 installs to the newly released versions just to be sure there are no future issues.

If you have any questions about this vulnerability, please reach out to us.

E-commerce sites can suffer from hackers injecting pieces of JavaScript to sniff credit card or other sensitive data.

It’s usually a checkout page with payment forms where JS code could be placed.

There is a simple yet effective way to catch it.

You store the original checkout page HTML and then regularly compare your current checkout page against ‘the gold standard’.

In this article I’ll describe one way of automating this security check using a $10 linux box.

1. Test Case

We will use a Magento 2 e-commerce site as an example.

The script will visit the website, add an item to the cart, proceed to checkout and compare the checkout page HTML against the template we have stored on the server.

If there is a mismatch, it indicates the page has been altered and needs to be reviewed.

2. Installation

We will use a 2G RAM linux box running Debian 10.

1.1 Facebook/Webdriver

Firstly, we need to install facebook/webdriver. That’s a PHP wrapper to work with a browser.

sudo apt update
sudo apt-get install composer
mkdir /home/username/security && cd /home/username/security
sudo apt-get install php-cli php-zip unzip wget php-curl
composer require facebook/webdriver

1.2 Google Chrome Browser

Next, we will install a google chrome web browser:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt-get install ./google-chrome-stable_current_amd64.deb

1.3 ChromeDriver

ChromeDriver is a standalone server that can talk to Google Chrome browser.

Versions of ChromeDriver and Google Chrome browser should match.

wget https://chromedriver.storage.googleapis.com/92.0.4515.107/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo cp chromedriver /usr/bin/

1.4 Xvfb

Xvfb is a virtual X server. We need it to run Google Chrome properly as we don’t have a monitor connected to our server.

apt-get install xvfb

1.5 PHP script to visit a checkout page

Below is the script that visits the Magento 2 website, adds an item to the cart, proceeds to the checkout and compares the checkout page HTML to what we have stored on the server.

/home/username/security/check.php:

namespace Facebook\WebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
require_once('vendor/autoload.php');
$host = 'http://localhost:4444/';
$capabilities = DesiredCapabilities::chrome();
$driver = RemoteWebDriver::create($host, $capabilities);
$driver->get('https://domain.com/');
$driver->wait()->until(
     WebDriverExpectedCondition::titleContains('Professional Grade Security')
);
$driver->wait()->until(
     WebDriverExpectedCondition::elementToBeClickable(WebDriverBy::cssSelector('.action.tocart.primary'))
);
$nextButton = $driver->findElement(
     WebDriverBy::cssSelector('.action.tocart.primary')
);
$nextButton->submit();
$driver->wait()->until(
     WebDriverExpectedCondition::visibilityOfElementLocated(WebDriverBy::cssSelector('.add-to-cart-dialog'))
);
$driver->get('https://domain.com/checkout/');
$driver->wait()->until(
     WebDriverExpectedCondition::visibilityOfElementLocated(WebDriverBy::cssSelector('.checkout-shipping-method'))
);
$driver->get('view-source:'.$driver->getCurrentUrl());
$paymentPage = $driver->getPageSource();
$old = file_get_contents('/home/username/security/payment.html');
if($old != $paymentPage)
echo 'hack';
$driver->quits();

1.6 Bash wrapper script

Here is a final bash wrapper script that starts chromedriver server, Xvfb and executes the checkout PHP script.

/home/username/security/script.sh:

#!/bin/bash
/usr/bin/Xvfb :0 -ac -screen 0 1024x768x24 &
DISPLAY=:0 /usr/bin/chromedriver --port=4444 &
sleep 5
echo 'running the test'
response=`/usr/bin/php /home/username/security/check.php`
echo $response
if [[ $response = "hack" ]]
then
echo 'not good'
fi
kill -9 `pidof chromedriver`
kill -9 `pidof Xvfb`

Instead of ‘echo ‘not good’’, a simple sendmail command could be placed to send a notification email.

…

About the author: Konstantin Gerasimov is a Magento certified developer with Goivvy.com. He specializes in speed optimization, migration and general Magento support.

During the COVID-19 Pandemic in late March, one of our Magento 2 Commerce clients had some massive traffic and sales. In partnership with Creatuity, the development agency for the client, we worked to scale up their site to handle this surge in visitors…

This blog post will look at the steps we took on the hosting end to meet this demand.

Identifying Bottlenecks

Visitors to the client’s website went through the roof from news exposure and marketing. From one night to the next, concentrated visits increased 5 to 10 times, which was an issue.

The hosting cluster set up for their store was designed to handle large amounts of traffic and orders. Scaling up php nodes would allow for increased traffic. But the surge in traffic all at once when commercials aired nationwide, combined with the media attention, and everyone at home watching TV was quite the challenge. We found that bottlenecks were quickly reached.

Here is what the New Relic graph looked like when these traffic spikes occurred initially:

(click image to enlarge)

As you can see, performance dropped precipitously. We quickly identified that Redis was the first bottleneck we had to overcome (see the next section for specific fixes we implemented). After Redis, we found that a single MySQL database for everything in Magento was the next hurdle. In between Redis and MySQL were also many other tweaks and improvements that were made.

Implementing Structural Changes

Redis:

We quickly discovered that Redis uses A LOT of data transfer under heavy traffic. In fact, it completely saturated the 1 Gbps private network card (which led to php and website actions backing up quickly). We had to split Redis off to its own server (away from MySQL), and more importantly, upgrade the private network to a 10 Gbps system.

We did this quickly, and Redis was now bursting in the 1-2.5 Gbps range comfortably at peak traffic. No more slowdowns due to Redis reaching network saturation.

MySQL:

Once Redis was running optimally, we found that when traffic spiked all at once, MySQL was having a hard time processing regular Ajax web requests, customer logins, add to cart actions, and checkouts all at the same time. It was quickly determined that splitting the database into three databases (Main, Cart, and Checkout) with Magento Commerce would pay big dividends.

We worked with Creatuity to migrate the code to Commerce and split out the databases. Once that was done, MySQL could comfortably scale with high levels of traffic and orders without getting bogged down during peak traffic. It also now supports replicated databases for read-only access to further increase throughput.

Misc. Changes:

In addition to the big changes with Redis and MySQL, we worked on improving performance in a number of areas:

• Tweaking network parameters (TCP/IP level) to improve throughput and private network performance (bare metal servers and network is a distinct advantage in this area)
• Memory adjustments to MySQL, Redis, etc… to make more efficient use of in-memory utilization
• PHP-FPM configuration changes to perform faster at scale
• Load Balancer adjustments to even out performance across the web nodes
• Better use of the CDN for static assets to offload NFS and load balancer utilization

The End Result:

After all of these changes were made in a short period of time, we were able to easily handle over 12,000 simultaneous visitors on the site. The infrastructure was able to support a peak order volume of 5 to 6 orders per second! The New Relic graph now shows a system that is working as it should:

(click image to enlarge)

Monitoring is Key

All of these changes would not have mattered if we didn’t have an effective monitoring system in place for each component in the cluster. Being able to detect an issue before it balloons into a major problem helps keep things running smoothly. Things such as:

• NFS monitoring for speed, performance, and errors
• TCP/IP monitoring for packet issues and saturation problems
• Load Balancer monitoring looking for errors and saturation conditions
• Web nodes being monitored for performance, error free, and serving content quickly
• Continuous monitoring of add to cart and checkout performance to pinpoint a drop in speed

Not only are things such as this monitored, but pro-active action is taken if any “tripwires” exceed a defined threshold. This has been the key to keeping things running smoothly for the hosting cluster.

…

It’s a good problem to have when you’re scaling up to handle tens of thousands of shoppers in a short period of time. Both LexiConn and Creatuity rose to the challenge and helped this Magento Commerce deal with unprecedented demand for their products.

If you are looking for a web host that can help your Magento store scale, get the most out of the software, and to have a hosting partner that can help you sleep well at night, don’t hesitate to reach out to us.