robots

v1.1.2

A parser for robots.txt.

HexDocs HexPreview

Current section

1 Dependant

Jump to

Readme 4 Versions 0 Dependencies 1 Dependant Activity

Packages depending on robots

1 package

treewalker v0.4.1

A web crawler that respects `robots.txt`.

Published about 6 years ago

65

recent downloads

total downloads: 1 107

1 package of 1 total

Checksum

Dependency Config

mix.exs

rebar.config

Gleam

erlang.mk

Package Details

Downloads Last 30 days, all versions

this version

11 060

yesterday

0

last 7 days

0

all time

22 876

Last Updated

Nov 21, 2023

License

BSD-3-Clause

Build Tools

rebar3

Publisher

Links

Github

Owners

antoine_gagne