Scrape H&M with MultiOn

This example combines MultiOn step and retrieve to scrape the H&M website catalog.

Project setup

TypeScript

Python

Initialize project

Create a new project by running the following command in your terminal:

npm init

Install package

Install the multion package by running the following command in your terminal:

npm install multion

Import library

Create a new file called index.ts and import the required library for the example:

import { MultiOnClient } from 'multion';

Initialize client

Initialize the MultiOn client with your API key.

const multion = new MultiOnClient({ apiKey: "YOUR_API_KEY" });

Run script

Run your script by running the following command in your terminal:

node index.ts

Scrape first page

To scrape the first page of the H&M catalog, we can simply call retrieve.

const retrieveResponse = await multion.retrieve({
  url: "https://www2.hm.com/en_us/men/products/view-all.html",
  cmd: "Get all items and their name, price, colors, purchase url, and image url.",
  fields: ["name", "price", "colors", "purchase_url", "image_url"]
});
const data = retrieveResponse.data;
console.log(data);

However, you might notice that while the first few items are complete, the rest are incomplete and some are even broken—especially images. This is because H&M dynamically loads the images as the user scrolls down the page.

To help with this, we can use renderJs to ensure image links are included and scrollToBottom to scroll down the page.

const retrieveResponse = await multion.retrieve({
  url: "https://www2.hm.com/en_us/men/products/view-all.html",
  cmd: "Get all items and their name, price, colors, purchase url, and image url.",
  fields: ["name", "price", "colors", "purchase_url", "image_url"],
  renderJs: true,
  scrollToBottom: true
});
const data = retrieveResponse.data;
console.log(data);

If we only want 10 items from the page, we can use maxItems to speed up the request.

const retrieveResponse = await multion.retrieve({
  url: "https://www2.hm.com/en_us/men/products/view-all.html",
  cmd: "Get all items and their name, price, colors, purchase url, and image url.",
  fields: ["name", "price", "colors", "purchase_url", "image_url"],
  renderJs: true,
  scrollToBottom: true,
  maxItems: 10
});
const data = retrieveResponse.data;
console.log(data);

Scrape multiple pages autonomously

To scrape multiple pages autonomously, we can use retrieve with step to navigate to next page. To do this, we must first create a session.

const createResponse = await multion.sessions.create({
  url: "https://www2.hm.com/en_us/men/products/view-all.html"
  // Can set useProxy to true to circumvent IP block
});
const sessionId = createResponse.sessionId;
console.log("Session created: ", sessionId);

Then, we can create a while loop that will keep running until the last page. At each iteration, the agent will retrieve data and step to navigate to the next page.

let hasMore = true;
while (hasMore) {
  const retrieveResponse = await multion.retrieve({
    sessionId: sessionId,
    cmd: "Get all items and their name, price, colors, purchase url, and image url.",
    fields: ["name", "price", "colors", "purchase_url", "image_url"],
    renderJs: true,
    scrollToBottom: true
  });
  console.log("Data retrieved: ", retrieveResponse.data);
  const stepResponse = await multion.sessions.step(sessionId, {
    cmd: "Keep clicking on the next page button.",
    mode: "fast",
  });
  console.log("Navigating to next page: ", stepResponse.message);
  hasMore = !stepResponse.message.includes("last page"); 
  // Can implement better way of checking if more pages to scrape
}

Scrape multiple pages in parallel

To massively speed up the scraping process, we can call retrieve for each page simultaneously. This works for H&M because the URL is numbered for each page.

const pagePromises = Array.from({ length: 10 }, (_, i) => i + 1).map(async (i) => {
  const retrieveResponse = await multion.retrieve({
    url: `https://www2.hm.com/en_us/men/products/view-all.html?page=${i}`,
    cmd: "Get all items and their name, price, colors, purchase url, and image url.",
    fields: ["name", "price", "colors", "purchase_url", "image_url"],
    renderJs: true,
    scrollToBottom: true
  });
  console.log(`Data retrieved for page ${i}: `, retrieveResponse.data);
  return retrieveResponse.data;
});
await Promise.all(pagePromises);