Packages

crawlie

0.1.0
1.0.0 0.6.0 0.5.1 0.5.0 0.4.1 0.4.0 0.3.1 0.3.0 0.2.0 0.2.0-alpha2 0.2.0-alpha1 0.1.1 0.1.0

A simple Elixir web crawler library powered by GenStage.

HexDocs

Current section

Files

Jump to
Readme
13 Versions
2 Dependencies
Dependants
Files
Activity
crawlie lib crawlie.ex
Raw

lib/crawlie.ex 

defmodule Crawlie do



  alias Crawlie.Options



  @spec crawl!(Stream.t, module, Keyword.t) :: Stream.t



  def crawl!(source, parser_logic, opts \\ []) do

    client = Options.get(opts, :http_client)



    results = source

      |> Stream.map(&client.get(&1, opts))

      |> Stream.map(&elem(&1, 1))

      |> Stream.map(&parser_logic.parse("fake_url", &1))

      |> Stream.map(&parser_logic.extract_data(&1))



    results

  end



end