100x Faster Shopify Product Import
Shopify bulk imports have cut our sync time from 18 hours to 3 minutes for 50,000+ product variants. Here’s how I implemented it using TypeScript.
I recently worked on a Shopify customer project where the product catalog is managed by an ERP system (Bits), which is quite common in the German shoe and fashion retail industry. However, it does not have a Shopify connection to synchronize products and orders. The product catalog consists of about 10,000 products with over 50,000 product variants. The existing Shopify connection was a custom-developed script that used a local database, REST API calls, and had to deal with Shopify API rate limits, etc. It was a classic grown script. In the end, it took over 18 hours to synchronize the entire product catalog and was not very reliable. So we decided it was time to rewrite it using GraphQL and Shopify bulk imports.
The time dropped from 18 hours to 3 minutes!
We were blown away when we saw the first results after the rewrite. Here is a diagram of the structure of the newly created program.
The bulk import consists of 2 parts:
- The product images are created via a photo box and are automatically synced with Amazon S3.
- The product catalog is exported from the ERP system and is synced with Shopify.
In this article, I will focus on the node.js CLI program. It is written in TypeScript and uses the Shopify GraqhQL API. The article is separated into the following sections:
- Demo CLI Program: It is a simplified version of the program I created for my client. You can run the demo program on a Shopify development store.
- Implementation walk through
Demo CLI Program
The source code of the fully functional demo program is freely available on Github, and can be run on a Shopify development store.
npx markustripp-shopify-bulk-import@latestRequirements
- Shopify Development Store
- Shopify Access Token
- Shopify Location for your inventory
- Shopify Online Store Sales Channel set to autopublish: true
- (optional) Resend API Key
- Configuration added to .env file in execution directory
Shopify Access Token
On your development store, create an access token with the access scopes shown in the screenshot below and add the access token to the .env file.
Shopify Location
The sync program updates the inventory. Since Shopify can manage multiple locations, you need to specify the Shopify location where the inventory will be updated. You can easily get the location ID with this GraphQL query run in the Shopify GraphQL Explorer.
query locations {
locations(first: 10) {
edges {
node {
id
name
address {
formatted
}
}
}
}
}Autopublish for Shopify Online Store Channel
When creating new products using the Shopify productSet GraphQL API, the product is not automatically published to any sales channel (e.g., online store, POS, Facebook, Google, etc.).
If you don’t want to make subsequent API calls, you’ll need to configure at least one sales channels to autopublish. This can be done through the Shopify GraphQL Explorer.
query publications {
publications(first: 10, catalogType: APP) {
nodes {
id
catalog {
... on AppCatalog {
apps(first: 10) {
nodes {
handle
}
}
}
}
}
}
}
mutation publicationUpdate($id: ID!, $input: PublicationUpdateInput!) {
publicationUpdate(id: $id, input: $input) {
publication {
autoPublish
id
}
userErrors {
field
message
}
}
}
Variables:
{
"id": "gid://shopify/Publication/271974269195",
"input": {
"autoPublish": true
}
}Resend and React Email
Since the script runs automatically, I want to get a status email when the program finishes or an error occurs. I love using Resend and React Email to send beautifully formatted emails.
Tip: If you want to design your own React email template, install the react-email-starter app, and clone one of the example templates (e.g., Linear, Notion, Slack, etc.) and implement your custom design.
.env
# required
SHOPIFY_GRAPHQL_URL=https://<store-name>.myshopify.com/admin/api/2025-04/graphql.json
SHOPIFY_ACCESS_TOKEN=
SHOPIFY_LOCATION_ID=gid://shopify/Location/999
# optional: Send email via https://resend.com
RESEND_API_KEY=
RESEND_TO=name@example.comCLI
The CLI program is implemented using TypeScript and can be published to NPM. I used the following technologies and tools:
- Bunchee: Typescript bundler which creates the executable.
- Resend with React Email and Tailwind CSS.
- Commander and ora: Creates command line input prompts and beautiful prints to the terminal
- nodefetchline: Reads JSONL lines from URL
CLI Program Structure
The program is structured into 5 steps:
- Parse (XML, JSON, CSV) from the ERP export
actions/step-01-erp.ts - Bulk Fetch Products
actions/step-02-products.ts - Prepare JSONL Import
actions/step-03-prepare.ts - Execute Bulk Import (create/update)
actions/step-04-import.ts - Send Status Email
actions/step-05-email.ts
I didn’t copy all the source code into this article, but it’s available for free on Github.
Bunchee, the TypeScript bundler I’m using, requires the executable files located in bin/index.ts. This file just validates the environment variable settings (.env), and calls the main program (actions/sync.ts).
Main Program
Github: actions/sync.ts
// Shopify has a limit of 20MB for file uploads incl. JSONL
const MAX_JSONL_LINES = 1000
export const sync = async (options)
) => {
const start = Date.now()
// 1. Parse export from ERP
const erpProducts = await stepParseErp(options.file as string)
// 2. Bulk fetch all products from Shopify and parse JSONL results
const productsUrl = await stepBulkFetchProducts('tag:sync')
const shopifyProducts = await stepFetchProductResults(productsUrl)
// 3. Prepare JSONL for the import to Shopify (create or update)
const jsonl = await stepPrepareImport(erpProducts, shopifyProducts)
// 4. Upload the JSONL to Shopify, call the bulk import, and get the results
const totalErrors: string[] = []
for (let i = 0; i < jsonl.length; i += MAX_JSONL_LINES) {
const chunk = jsonl.slice(i, i + MAX_JSONL_LINES)
const importUrl = await stepBulkImport(chunk)
const errors = await stepBulkImportResult(importUrl, chunk)
totalErrors.push(...errors)
}
// 5. Send a status email via Resend
const duration = formatDuration(Date.now() - start)
await stepSendEmail({
numberOfProducts: erpProducts.size,
numberOfErrors: totalErrors.length,
duration,
})
ora(`Finished. ${totalErrors.length} errors. (${duration})`).succeed()
}Parse XML, JSON, CSV Export From ERP
Github: actions/step-01-erp.ts
I usually prefer real-time online interfaces. But sometimes older ERP systems only allow you to export certain resources to XML, CSV, etc. For my client project, we used an XML export from the ERP system of all products with the tag “available in the online shop.”
I used a SAX XML parser to parse the export and extract all the data I need for the Shopify import. This part is specific to your system. In the demo app I used a simple JSON data file to emulate this step.
Bulk Fetch Products
Github: actions/step-02-products.ts
Bulk fetching the products is a 2 steps process:
- Shopify bulk operation
- Fetch and parse the JSONL results
The Shopify bulk operation is asynchronous, because it can take some time. Therefore, the CLI app checks every second if the bulk process has been finished. I implemented it in a simple while loop.
while (bulkOperation.status !== 'COMPLETED') {
spinner.text = `Shopify bulk operation ${bulkOperation.status} ${bulkOperation.id}`
// wait for 1 second
await new Promise((resolve) => setTimeout(resolve, 1000))
const json = await fetchShopify(getQueryCurrentBulk(false))
bulkOperation = json.data.currentBulkOperation
}
...
return bulkOperation.url
...
export const getQueryCurrentBulk = (mutation: boolean) => `
query {
currentBulkOperation(type: ${mutation ? 'MUTATION' : 'QUERY'}) {
id
status
errorCode
createdAt
completedAt
objectCount
fileSize
url
partialDataUrl
}
}When the process is finished successfully, I return the URL to get the JSONL results (bulkOperation.url).
For reading the JSONL results into my products Map, I use the nodefetchline library.
export const stepFetchProductResults = async (url: string | null) => {
const spinner = ora('Shopify bulk fetch product results').start()
const lineIterator = url ? fetchLine(url) : null
const productsMap = await parseProducts(lineIterator)
spinner.succeed(`Shopify Products loaded. ${productsMap.size} products.`)
return productsMap
}Prepare JSONL Import
Github: actions/step-03-prepare.ts
Bulk imports are more expensive than bulk fetch calls. My goal was to minimize the number of API calls when creating or updating the products. Fortunately, there is one Shopify GraphQL API function that can perform all create and/or update operations in a single request: productSet. This is one of the most powerful operations of the entire Shopify GraphQL API.
Shopify bulk operations require the paramaters passed in JSONL format. It is simply an array of strings created via JSON.stringify(input).
{"input":{"handle":"the-collection-snowboard-oxygen","title":"The Collection Snowboard Oxygen", ... }}
{"input":{"id":"gid://shopify/Product/999","handle":"the-multi-location-snowboard","title":"The Multi-location Snowboard", ...}}If the JSON input line has an “id” field, productSet will execute an update, otherwise it creates a new product.
Execute Bulk Import (create/update)
Github: actions/step-04-import.ts
After the JSONL has been created, the CLI program can call the bulk import. This is a 4 steps process:
- Create a staged upload object via GraphQL
- Use the staged upload URL and parameters to upload the JSONL string via an HTTP POST request using FormData.
Note: Shopify has a limit of 20MB for file uploads incl. JSONL - Execute the Shopify bulk operation
- Wait for the operation to finish and then check the results to see if there were any errors during the import.
Please check the source file (actions/step-04-import.ts) for the implementation details.
Send Status Email
Github: actions/step-05-email.ts
I love the simplicity of Resend and React Email.
export const stepSendEmail = async (params: StatusEmailParams) => {
const spinner = ora('Sending Email').start()
const resend = new Resend(process.env.RESEND_API_KEY)
await resend.emails.send({
from: 'Resend <onboarding@resend.dev>',
to: process.env.RESEND_TO!,
subject: 'Shopify Bulk Import Status',
react: await ResendStatusEmail(params),
})
spinner.succeed('Email sent')
}Please be aware that the ResendStatusEmail component must be a default export.
export type StatusEmailParams = {
duration: string
numberOfProducts: number
numberOfErrors: number
}
export const ResendStatusEmail: React.FC<StatusEmailParams> = ({
numberOfProducts,
numberOfErrors,
duration,
}) => {
return (
<Tailwind>
<Html>
<Head />
<Preview>Shopify Bulk Import Status</Preview>
<Body className="p-4 text-gray-800 bg-white">
<Container>
<Heading className="text-2xl font-bold">
Shopify Bulk Import Status
</Heading>
<Section className="my-8 px-8 py-3 bg-gray-100 rounded-lg">
<Text className="text-gray-600">
{numberOfProducts} products from ERP.
<br />
{numberOfErrors} errors.
<br />
Duration: {duration}
</Text>
</Section>
</Container>
</Body>
</Html>
</Tailwind>
)
}
export default ResendStatusEmailConclusion
I’ve seen many Shopify projects in the past that still use the old REST API. Or they already use the GraphQL API but don’t benefit from the bulk operations. It takes a little more effort, but you don’t have to deal with Shopify’s rate limits and create weird workarounds.
I hope this article helps you get started on your next Shopify bulk fetch/import project or helps you migrate existing scripts.
About Me
I’m Markus, a freelance developer and consultant. I have worked on some of the biggest Shopify Plus stores worldwide. In addition to Shopify and Liquid, I love to work on React, TypeScript and Next.js projects. I live in Salzburg, Austria.
