Released spidr 0.7.0. Added a `Spidr.domain` method for spidering the domains and any sub-domains.
https://github.com/postmodern/spidr/blob/master/ChangeLog.md#070--2022-12-31
https://github.com/postmodern/spidr
#ruby #webspider #webcrawler #spidering

Coming up with the options for a web spider command and which options are mutually exclusive, is really difficult. Like obviously such a common should print the URLs by default. However, what if the user also wants to scrape HTML nodes out of each webpage using an XPath? Should you print the URLs and the matched content, or disable printing of URLs if --xpath is specified, or have a separate option called like --no-print-urls to explicitly disable printing the URLs if you only want to pipe the matched HTML into some other util.
#webspidering #webspider #spidering #cli #recon

What would be some good infosec/bughunter web spidering enumeration functions? So far I have hooks for enumerating every cert, favicon, hostname, JavaScript source, and JS strings. Trying to think of more.
#infosec #bugbounty #spidering #recon