Packages

scrape

3.0.3
3.1.0 3.0.3 3.0.2 3.0.1 3.0.0 2.0.0 1.2.7 1.2.6 1.2.5 1.2.4 1.2.3 1.2.2 1.2.1 1.0.4 1.0.3 1.0.2 1.0.1 1.0.0 0.2.0 0.1.1 0.1.0 0.0.4 0.0.3 0.0.2 0.0.1

Scrape any website, article or RSS/Atom feed with ease!

HexDocs

Current section

Files

Jump to
Readme
25 Versions
8 Dependencies
Dependants
Files
Activity
scrape lib scrape source http get.ex
Raw

lib/scrape/source/http/get.ex 

defmodule Scrape.Source.HTTP.Get do

  @moduledoc false



  @opts [

    follow_redirect: true,

    timeout: 33_000,

    recv_timeout: 30_000,

    ssl: [{:versions, [:"tlsv1.2"]}]

  ]



  @headers [

    "user-agent":

      "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36",

    accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"

  ]



  def execute(url, http_headers \\ @headers, http_opts \\ @opts) do

    HTTPoison.get(url, http_headers, http_opts)

  end

end