crawlfox

# crawlfox (Elixir)

Official Elixir SDK for the [CrawlFox](https://crawlfox.io) API: scrape, search (Google, Bing, or DuckDuckGo), batch scrape, map site links, crawl multi-page jobs, and logs.

Get a key from the [dashboard](https://crawlfox.io). Set `CRAWLFOX_API_KEY`.

Hex once published:

```elixir
{:crawlfox, "~> 0.2.0"}
```

Until Hex lists it:

```elixir
{:crawlfox,
 git: "https://github.com/Automote-LLC/crawlfox-integrations.git",
 sparse: "packages/elixir"}
```

```elixir
client = Crawlfox.new()

page = Crawlfox.scrape(client, "https://example.com", %{
  "formats" => ["markdown", "links"]
})

hits = Crawlfox.search(client, "rust async tutorial", %{
  "engine" => "google",
  "num" => 10
})

batch = Crawlfox.batch(client, ["https://example.com/", "https://example.org/"], %{
  "formats" => ["markdown"]
})
```

Formats: `markdown`, `html`, `rawHtml`, `json`, `links`, `images`, `emails`. CSS selectors live under `jsonOptions` when you request `json`.

## Map

`POST /v1/map`. One credit per call.

```elixir
mapped = Crawlfox.map(client, "https://example.com/", %{limit: 10})
```

## Crawl

```elixir
job = Crawlfox.crawl(client, "https://example.com/", %{
  limit: 3,
  scrapeOptions: %{formats: ["markdown"]}
})

status = Crawlfox.get_crawl(client, job["id"])
done = Crawlfox.wait_for_crawl(client, job["id"])
Crawlfox.cancel_crawl(client, job["id"])
errors = Crawlfox.get_crawl_errors(client, job["id"])
```

Credits: scrape 1 per page, search 1 per 10 requested results, map 1 per call, crawl 1 per page per format.

```bash
mix deps.get && mix test
```

Docs: https://docs.crawlfox.io