> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.multion.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.multion.ai/_mcp/server.

# Scrape H&M with MultiOn

> Learn to use step and retrieve to scrape structured data

This example combines MultiOn [step](/step-api/sessions/step) and [retrieve](/autonomous-api/retrieve) to scrape the H\&M website catalog.

## Project setup

#### TypeScript

### Initialize project

Create a new project by running the following command in your terminal:

```bash
npm init
```

### Install package

Install the `multion` package by running the following command in your terminal:

```bash
npm install multion
```

### Import library

Create a new file called `index.ts` and import the required library for the example:

```ts
import { MultiOnClient } from 'multion';
```

### Initialize client

Initialize the MultiOn client with your API key.

```ts
const multion = new MultiOnClient({ apiKey: "YOUR_API_KEY" });
```

### Run script

Run your script by running the following command in your terminal:

```bash
node index.ts
```

#### Python

### Install package

Install the `multion` package by running the following command in your terminal:

```bash
pip install multion
```

### Import library

Create a new file called `main.py` and import the required library for the example:

```python
from multion.client import MultiOn
```

### Initialize client

Initialize the MultiOn client with your API key.

```python
client = MultiOn(api_key="YOUR_API_KEY")
```

### Run script

Run your script by running the following command in your terminal:

```bash
python main.py
```

## Scrape first page

To scrape the first page of the H\&M catalog, we can simply call retrieve.

#### TypeScript

```ts
const retrieveResponse = await multion.retrieve({
  url: "https://www2.hm.com/en_us/men/products/view-all.html",
  cmd: "Get all items and their name, price, colors, purchase url, and image url.",
  fields: ["name", "price", "colors", "purchase_url", "image_url"]
});

const data = retrieveResponse.data;
console.log(data);
```

#### Python

```python
retrieve_response = client.retrieve(
    url="https://www2.hm.com/en_us/men/products/view-all.html",
    cmd="Get all items and their name, price, colors, purchase url, and image url.",
    fields=["name", "price", "colors", "purchase_url", "image_url"]
)

data = retrieve_response.data
print(data)
```

However, you might notice that while the first few items are complete, the rest are incomplete and some are even broken--especially images. This is because H\&M dynamically loads the images as the user scrolls down the page.

To help with this, we can use `renderJs` to ensure image links are included and `scrollToBottom` to scroll down the page.

#### TypeScript

```ts
const retrieveResponse = await multion.retrieve({
  url: "https://www2.hm.com/en_us/men/products/view-all.html",
  cmd: "Get all items and their name, price, colors, purchase url, and image url.",
  fields: ["name", "price", "colors", "purchase_url", "image_url"],
  renderJs: true,
  scrollToBottom: true
});

const data = retrieveResponse.data;
console.log(data);
```

#### Python

```python
retrieve_response = client.retrieve(
    url="https://www2.hm.com/en_us/men/products/view-all.html",
    cmd="Get all items and their name, price, colors, purchase url, and image url.",
    fields=["name", "price", "colors", "purchase_url", "image_url"],
    render_js=True,
    scroll_to_bottom=True
)

data = retrieve_response.data
print(data)
```

If we only want 10 items from the page, we can use `maxItems` to speed up the request.

#### TypeScript

```ts
const retrieveResponse = await multion.retrieve({
  url: "https://www2.hm.com/en_us/men/products/view-all.html",
  cmd: "Get all items and their name, price, colors, purchase url, and image url.",
  fields: ["name", "price", "colors", "purchase_url", "image_url"],
  renderJs: true,
  scrollToBottom: true,
  maxItems: 10
});

const data = retrieveResponse.data;
console.log(data);
```

#### Python

```python
retrieve_response = client.retrieve(
    url="https://www2.hm.com/en_us/men/products/view-all.html",
    cmd="Get all items and their name, price, colors, purchase url, and image url.",
    fields=["name", "price", "colors", "purchase_url", "image_url"],
    render_js=True,
    scroll_to_bottom=True,
    max_items=10
)

data = retrieve_response.data
print(data)
```

## Scrape multiple pages autonomously

To scrape multiple pages autonomously, we can use retrieve with step to navigate to next page. To do this, we must first create a session.

#### TypeScript

```ts
const createResponse = await multion.sessions.create({
  url: "https://www2.hm.com/en_us/men/products/view-all.html"
  // Can set useProxy to true to circumvent IP block
});

const sessionId = createResponse.sessionId;
console.log("Session created: ", sessionId);
```

#### Python

```python
create_response = client.sessions.create(
    url="https://www2.hm.com/en_us/men/products/view-all.html"
    # Can set useProxy to True to circumvent IP block
)

session_id = create_response.session_id
print("Session created: ", session_id)
```

Then, we can create a while loop that will keep running until the last page. At each iteration, the agent will retrieve data and step to navigate to the next page.

#### TypeScript

```ts
let hasMore = true;
while (hasMore) {
  const retrieveResponse = await multion.retrieve({
    sessionId: sessionId,
    cmd: "Get all items and their name, price, colors, purchase url, and image url.",
    fields: ["name", "price", "colors", "purchase_url", "image_url"],
    renderJs: true,
    scrollToBottom: true
  });
  console.log("Data retrieved: ", retrieveResponse.data);
  const stepResponse = await multion.sessions.step(sessionId, {
    cmd: "Keep clicking on the next page button.",
    mode: "fast",
  });
  console.log("Navigating to next page: ", stepResponse.message);
  hasMore = !stepResponse.message.includes("last page"); 
  // Can implement better way of checking if more pages to scrape
}
```

#### Python

```python
has_more = True
while has_more:
    retrieve_response = client.retrieve(
        session_id=session_id,
        cmd="Get all items and their name, price, colors, purchase url, and image url.",
        fields=["name", "price", "colors", "purchase_url", "image_url"],
        render_js=True,
        scroll_to_bottom=True
    )
    print("Data retrieved: ", retrieve_response.data)
    step_response = client.sessions.step(
        session_id=session_id,
        cmd="Keep clicking on the next page button.",
        mode="fast",
    )
    print("Navigating to next page: ", step_response.message)
    has_more = "last page" not in step_response.message
    # Can implement better way of checking if more pages to scrape
```

## Scrape multiple pages in parallel

To massively speed up the scraping process, we can call retrieve for each page simultaneously. This works for H\&M because the URL is numbered for each page.

#### TypeScript

```ts
const pagePromises = Array.from({ length: 10 }, (_, i) => i + 1).map(async (i) => {
  const retrieveResponse = await multion.retrieve({
    url: `https://www2.hm.com/en_us/men/products/view-all.html?page=${i}`,
    cmd: "Get all items and their name, price, colors, purchase url, and image url.",
    fields: ["name", "price", "colors", "purchase_url", "image_url"],
    renderJs: true,
    scrollToBottom: true
  });
  console.log(`Data retrieved for page ${i}: `, retrieveResponse.data);
  return retrieveResponse.data;
});

await Promise.all(pagePromises);
```

#### Python

```python
# We have to import AsyncMultiOn instead of the normal MultiOn client
from multion.client import AsyncMultiOn
client = AsyncMultiOn(api_key=os.getenv("MULTION_API_KEY"))

async def retrieve_page(client, page_number):
    retrieve_response = await client.retrieve(
        url=f"https://www2.hm.com/en_us/men/products/view-all.html?page={page_number}",
        cmd="Get all items and their name, price, colors, purchase url, and image url.",
        fields=["name", "price", "colors", "purchase_url", "image_url"],
        render_js=True,
        scroll_to_bottom=True,
    )
    print(f"Data retrieved for page {page_number}: ", retrieve_response.data)
    return retrieve_response.data

async def retrieve_all_pages():
    tasks = []
    for i in range(1, 11):
        tasks.append(retrieve_page(client, i))

    results = await asyncio.gather(*tasks)
    return results

asyncio.run(retrieve_all_pages())
```