- Running huge imports with WP All Import reliably (yes, even if you use Cloudflare!) - September 19, 2023
- New SQL enhancement for Scalability Pro to fix WooCommerce long-running query in the product-hero block - August 31, 2023
- Create a static favicon.ico to avoid surplus PHP requests for rush traffic - July 25, 2023
If you know me, you know I love WP All Import. It’s the best import tool for WordPress on the market and they still offer lifetime licenses even to this day. You’ll also probably know I’m a big fan of Cloudflare for how much protection it can add to your WordPress website for free, and as a direct result how much pressure it can alleviate from your server.
If you’re dealing with huge imports with WP All Import on your WordPress website with data, you might have encountered performance bottlenecks. The good news is, our cutting-edge plugin, Scalability Pro, is your silver bullet for fixing import speeds and overall website performance.
Table of Contents
- Speed up your imports with Scalability Pro first
- Why do Cloudflare and WP All Import not work well together?
- What solutions are available to dependably run huge imports with WP All Import?
- How to bypass Cloudflare using PHP-CLI to get huge imports to complete reliably every time
- Constructing the PHP command to call WP All Import reliably from your cron
- Testing and Troubleshooting
- Summary
Speed up your imports with Scalability Pro first
Scalability Pro optimizes your WordPress database, making it more efficient and less prone to slowdowns, even when you’re importing large datasets. In particular, Scalability Pro optimises lookups so that when your import is checking if an item already exists it gets that answer more quickly.
Additionally, it defers the term recounts until after the import has complete – normally, recounts happen after every single import which is incredibly wasteful. These recounts give you the little numbers of published/draft/trash items in your wp-admin pages. They’re not as important as getting your import to complete quickly.
There are many other optimisations for imports in our Scalability Pro plugin which have been collected over the years of completing performance analysis work for our clients and you get the benefit to transform your WordPress site into a high-performance import machine.
If you don’t yet have Scalability Pro, you should add that first and you probably won’t need this guide at all. However, there are some of you out there importing products with 1000s of variations and since WP All Imports minimum-item-per-batch only operates at the parent product level, if you have enough variations in your import it’s possible to hit server or Cloudflare timeouts even with Scalability Pro installed.
Why do Cloudflare and WP All Import not work well together?
Most people running imports run them from the WordPress wp-admin pages. When you’re loading pages there, your pages are loading through Cloudflare and are susceptible to their 100s timeout limit.
Add on to this the ‘batch size’ you use with WP All Import – the larger batch you run, the faster your imports will be, but then you’ll be more likely to run into this 100s timeout limit and your import gets halted.
Even if you run your import from cron, you’re probably fetching the import URL so you’re *still* going through Cloudflare.
Add on to this the variable product issue – some of your products in your import might only have a few variations and can handle a 100 or 200 batch limit with no issues, but then some other products might have 100,000 variations and now we have a problem. In this case, even dropping your batch size down to 1 product at a time can fail.
What solutions are available to dependably run huge imports with WP All Import?
One way to fix this would be to create a separate subdomain for your wp-admin pages and have that bypass Cloudflare completely. I don’t like this solution because then your most important pages, from a security point of view, are bypassing Cloudflare’s security tools.
Another way would be to split your import files out so that you have simpler (< 100 variations) products and complicated (>= 100 variations) products in separate files – then you can have separate batch sizes for each. This is not great either because it’s a faff to have to split the files up and how to even do that?
Finally, instead of calling the URL, you can call the PHP file directly from your server using PHP-CLI. PHP-CLI is like PHP-FPM except no output buffering, everything gets echoed immediately, and no timeouts. More importantly, by calling the file directly from your server you bypass Cloudflare completely.
How to bypass Cloudflare using PHP-CLI to get huge imports to complete reliably every time
When you look at WP All Import cron Scheduling Options, they’ll give you 2 URLs per import – the trigger URL to start the job and the processing URL to do the actual work.
For example:
- wp-load.php?import_key=your_key&import_id=9&action=trigger
- wp-load.php?import_key=your_key&import_id=9&action=processing
Notice there is a file – wp-load.php and then 3 URL parameters:
- import_key
- import_id
- action (either trigger or processing)
Most people when they are creating a cron will simply create a command like below and stick that in their crontab and then wonder why their problems are not over:
wget https://yourdomain.com/wp-load.php?import_key=...
Instead of calling the URL, you want to load the PHP file directly on the server. Some notes:
- It’s important you don’t do this as root or a malicious plugin could take over your entire server
- You might need to review your php.ini – PHP-CLI has its own php.ini which is different from PHP-FPM which serves web pages up. Probably not though, PHP-CLI defaults to stupidly high rates for memory, execution time etc
To construct the php command to call wp all import directly you need the following additional information from your server (additional to the url parameters above):
- Web username – what user your website runs files as normally
- Webroot folder – where your websites files are located – the top/root folder of your website
Often the web user will be called www-data. You can see the owner if you connect to your sites webroot folder with SFTP (I use Filezilla), or you can SSH to your server (since you’ll need to anyway) and navigate to your web folder then run:
ls -l
You’ll see 2 usernames, the first is the owner, the other is the group – they might be the same like on my sites where it’s www-data:www-data. You need to know the owner.
You also need the webroot folder – normally this is located under /var/www/domain.com/{maybe some extra folder} but in that folder you will see the file wp-load.php. Alternative locations include ~/applications/xyz and more. If you are struggling to find it you can run these 2 commands to locate them:
cd / find . -name wp-load.php
Constructing the PHP command to call WP All Import reliably from your cron
Once you know the owner and webroot folder, you want to edit the crontab for that web user. Crontab is the task-scheduler on web servers. It’s important you switch to the correct users tab because previously you were relying on your web server providing security. You don’t have that here because you are running a file directly on the server, so BE CAREFUL!
sudo -u www-data crontab -e
The above will open your crontab editor for the www-data user.
Now, construct your 2 commands – the trigger command and the processing command. Like this:
php -r '$_GET["import_key"] = "your_import_key"; $_GET["import_id"] = "your_import_id"; $_GET["action"] = "trigger"; require("/the/path/to/your/website/wp-load.php");'
You need to replace your_import_key, your_import_id and /the/path/to/your/website/
The only difference in your 2nd command will be replacing the word “trigger” with “processing”.
crontab uses a 5 column system for their schedules – minute, hour, day of the month, month, day of the week. Each of these can be a number or * or */number.
For example:
0 0 * * * yourcommand
The above would run at the 0 minute of the 0 hour i.e. midnight, then every day of the month, every month of the year, every day of the week.
* * * * * yourcommand
The above would run yourcommand every minute of every day.
*/2 * * * * yourcommand
The above would run yourcommand every 2nd minute of every day etc.
You want to pick a suitable schedule for your trigger command – e.g. once per week or every night – and then your processing command you can either run every minute, or every 2nd minute. WP All Import say that you should do it every 2nd minute, but they have a built in system to detect if another job is already processing so I’ve never had any issues running the processing job every minute.
However, through my tests I found that a single call to the processing command is enough to run the entire import. When run from php-cli it will simply continue until complete, or until interrupted.
Put it all together and we have:
0 0 * * * php -r '$_GET["import_key"] = "your_import_key"; $_GET["import_id"] = "your_import_id"; $_GET["action"] = "trigger"; require("/the/path/to/your/website/wp-load.php");'
*/2 * * * * php -r '$_GET["import_key"] = "your_import_key"; $_GET["import_id"] = "your_import_id"; $_GET["action"] = "processing"; require("/the/path/to/your/website/wp-load.php");'
You’ll need to have 2 lines like these for each of the imports you want to process. It sounds like a pain I guess, but do you want your imports to complete or not!?
Testing and Troubleshooting
You can always go back to WP All Imports Manage Imports page and view progress there, or if you want to just make sure your command is configured properly you can run your trigger command from the command line followed by your processing command and read the output.
If you have logged onto your server as root, be careful – don’t just run the raw command, instead prefix your command with sudo -u www-data, e.g:
sudo -u www-data php -r '$_GET["import_key"] = "your_import_key"; $_GET["import_id"] = "your_import_id"; $_GET["action"] = "trigger"; require("/the/path/to/your/website/wp-load.php");'
Then repeat for your processing command. You should see output on the screen telling you what’s happening.
If your server tells you php command not found then you can ask your hosts to install it and all the WordPress dependencies, or if you’re on Ubuntu/Debian then you can use a command like this:
sudo apt-get install php8.1-cli php8.1-common php8.1-mysql php8.1-zip php8.1-gd php8.1-mbstring php8.1-curl php8.1-xml php8.1-bcmath php8.1-xmlrpc
When you then manage to run your php command manually to trigger the job, if it tells you there’s a critical error, you should enable debug logging and then from your wp-content folder run:
tail debug.log -n 100
Note: On some hosts, your debug.log might be in a logs folder one level up from your website home folder.
For example, you might discover a missing library – if you do, you can comment here and I’ll update the article for others.
Once you’ve successfully run your trigger php command you should get a message like this:
{"status":200,"message":"#9 Cron Job started"}
Then you can run your processing command manually to test that too and, with a delay at the beginning to process your file and split it into chunks, you’ll eventually start seeing message spat out to the screen similar to what you would see if you were running this import through the wp-admin web interface.
You can wait until this batch has completed and then go back and enable this processing command through your cron job.
Summary
If you want your imports never to fail again, this is the way to go. You can have a batch size of 200 and even if one product takes an hour to import because of 1000s of underlying variations it will still complete.
Using PHP-CLI for WordPress imports is a reliable and flexible solution that bypasses Cloudflare and other timeout-related limitations. It’s especially useful for e-commerce sites that need to handle large, frequent inventory imports. This method improves scalability and performance, helping you avoid server timeouts and Cloudflare restrictions.
