> ## Documentation Index
> Fetch the complete documentation index at: https://docs.declaw.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraping with Network Policy

> Run a web scraper inside a Declaw sandbox with a network allow-list that restricts outbound access to a single domain, then prove that all other domains are unreachable.

## What You'll Learn

* How to create a sandbox with `network={"allow_out": [...]}` for domain-level scraping control
* How to upload and run a scraping script using Python's `urllib` (stdlib — no pip installs needed)
* How to verify that allowed domains are reachable and blocked domains are not
* The pattern for containing scrapers that should only access specific sources

## Prerequisites

* Declaw running locally or in the cloud (see [Deployment](/deployment/overview))
* `DECLAW_API_KEY` and `DECLAW_DOMAIN` set in your environment
* Outbound network access from your Declaw instance to `httpbin.org`

<Note>
  This example is available in Python. TypeScript support coming soon.
</Note>

## Code Walkthrough

### 1. Create the sandbox with a network allow-list

```python theme={null}
from declaw import Sandbox

sbx = Sandbox.create(
    template="python",
    timeout=300,
    network={"allow_out": ["httpbin.org"]},
)
```

Only traffic destined for `httpbin.org` is allowed. All other outbound connections — including DNS for other domains and direct IP connections — are blocked by the TCP proxy.

### 2. The scraper script

The scraper uses Python's built-in `urllib` — no third-party packages required. The sandbox's base Ubuntu image already has Python 3 installed:

```python theme={null}
SCRAPER_SCRIPT = """\
import urllib.request
import json

url = "http://httpbin.org/get"
print(f"Fetching: {url}")

req = urllib.request.Request(url, headers={"User-Agent": "Declaw-Sandbox/1.0"})
with urllib.request.urlopen(req, timeout=10) as resp:
    body = resp.read().decode("utf-8")
    data = json.loads(body)
    print(f"Status: {resp.status}")
    print(f"Origin IP: {data.get('origin', 'unknown')}")
    print(f"Headers sent: {json.dumps(data.get('headers', {}), indent=2)}")
    print("SUCCESS: Allowed domain is reachable")
"""
```

### 3. Prove blocked domains are unreachable

Use a TCP socket test rather than an HTTP request — the block applies at the TCP layer, so even raw socket connections to blocked IPs are refused:

```python theme={null}
BLOCKED_SOCKET_TEST = """\
import socket

target = "93.184.216.34"  # example.com IP
port = 80
try:
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.settimeout(5)
    s.connect((target, port))
    s.close()
    print("CONNECTED")
except Exception as e:
    print(f"BLOCKED: {e}")
"""
```

### 4. Upload and run both tests

```python theme={null}
try:
    # Test 1: Scrape the allowed domain
    sbx.files.write("/home/user/scraper.py", SCRAPER_SCRIPT)
    result = sbx.commands.run("python3 /home/user/scraper.py 2>&1")
    print(result.stdout)

    # Test 2: TCP socket to blocked domain (example.com)
    sbx.files.write("/home/user/blocked_test.py", BLOCKED_SOCKET_TEST)
    result2 = sbx.commands.run("python3 /home/user/blocked_test.py 2>&1")
    print(result2.stdout)
finally:
    sbx.kill()
```

## Expected Output

```
--- Creating Sandbox with Network Policy ---
  Policy: allow only httpbin.org outbound
  Sandbox created: sbx-abc123

--- Scraping Allowed Domain (httpbin.org) ---
Fetching: http://httpbin.org/get
Status: 200
Origin IP: 203.0.113.42
Headers sent: {
  "Host": "httpbin.org",
  "User-Agent": "Declaw-Sandbox/1.0"
}
SUCCESS: Allowed domain is reachable

--- TCP Socket Test: Blocked Domain (example.com) ---
Attempting TCP connection to 93.184.216.34:80 (example.com)...
BLOCKED: [Errno 110] Connection timed out

--- Network Policy Summary ---
  httpbin.org:  ALLOWED (network policy permits this domain)
  example.com:  BLOCKED (not in the allow list)
```

## Use Cases

**Price monitoring:** Allow only the target retailer's domain. The scraper cannot exfiltrate data to other servers or call home.

**News aggregation:** Allowlist a set of news site domains. Even if the scraped page contains malicious JavaScript or links, the sandbox cannot follow them to unauthorized destinations.

**Competitive intelligence:** Restrict the scraper to a defined list of competitor domains. Any unexpected outbound connection is blocked automatically.

## Domain Allowlist vs IP Allowlist

The network policy uses domain names, not IP addresses. The proxy resolves the domain to an IP at connection time and enforces the rule at the TCP layer. This means:

* `allow_out: ["httpbin.org"]` permits connections to any IP that `httpbin.org` resolves to
* Direct IP connections (like `93.184.216.34`) are blocked unless the IP resolves to an allowlisted domain at the time of the connection
* CDNs and load balancers that share IPs across domains are handled correctly — the proxy checks the SNI (TLS) or Host header (HTTP) rather than just the IP

For CIDR-based rules or more fine-grained control, see [Network Policies](/security/network-policies).