How To Download From An Open Directory

Open Directory Downloader

Indexes open up directories listings in 100+ supported formats, including FTP(Due south), Google Drive, Bhadoo, GoIndex, Go2Index (alternatives), Dropbox, Mediafire, GoFile, GitHub.

Written in C# with .NET (Core), which means information technology is cantankerous platform!

Downloading is not (yet) implemented, merely is already possible when you use the resulting file into another tool (for most of the formats).

Downloading with wget:
wget -x -i theurlsfile.txt

Downloading with aria2c (Does not support directory construction..):
aria2c -i theurlsfile.txt

If you take improvements, supply me with a pull request! If y'all have a format not yet supported, please let me know.

Releases / Binaries

For builds (64-bit) for Windows, Linux and Mac, or ARM/ARM64 builds for Pi:

https://github.com/KoalaBear84/OpenDirectoryDownloader/releases

When using the cocky-contained releases yous don't need to install the .NET (Core) Runtime.

Prerequisites

When y'all are NOT using the cocky-independent releases, you lot need to install the latest/electric current Runtime version of .NET 7:

https://dotnet.microsoft.com/download/dotnet/seven.0/runtime

Usage

Command line parameters:

Curt	Long	Description
`-u`	`--url`	Url to scan
`-t`	`--threads`	Number of threads (default 5)
`-o`	`--timeout`	Number of seconds for timeout
`-w`	`--wait`	Number of seconds to await between calls (when single threaded is likewise fast..)
`-q`	`--quit`	Quit later on scanning (No "Press a cardinal")
`-c`	`--clipboard`	Automatically copy the Reddits stats one time the scan is washed
`-j`	`--json`	Save JSON file
`-f`	`--no-urls`	Do not save URLs file
`-r`	`--no-reddit`	Practise not bear witness Reddit stats markdown
`-l`	`--upload-urls`	Uploads urls file
`-e`	`--exact-file-sizes`	Exact file sizes (WARNING: Uses Caput requests which takes more time and is heavier for server)
	`--fast-scan`	Only use sizes from HTML, no Caput requests, fifty-fifty if the approx. size cannot exist extracted from the HTML
`-s`	`--speedtest`	Does a speed test later on indexing
`-a`	`--user-agent`	Utilise custom default User Agent
	`--username`	Username
	`--countersign`	Password
	`--github-token`	GitHub Token
`-H`	`--header`	Supply a custom header to use for each HTTP asking. Can be used multiple times for multiple headers. See below for more info.
	`--output-file`	Output file to use for urls file
	`--proxy-address`	Proxy address, like "socks5://127.0.0.1:9050" (needed for .onion)
	`--proxy-username`	Proxy username
	`--proxy-password`	Proxy password
	`--no-browser`	Disallow starting Chromium browser (for Cloudflare)

Case

Windows

OpenDirectoryDownloader.exe --url "https://myopendirectory.com"

Linux

./OpenDirectoryDownloader --url "https://myopendirectory.com"

If you desire to learn more or contribute, see the following paragraphs!

Custom Headers

Headers need to exist provided in the following format:

              <Header Name>: <Header Value>

This syntax is uniform with e.g. scroll, then that you can copy the headers from a whorl command and re-utilise them with OpenDirectoryDownloader.

This ways you lot can hands "fake" a browser request:

On the page/site you want to index, open your browsers dev tools (F12 or CTRL + SHIFT + i)
Go to the Network tab
Reload the folio
Correct-click on the outset asking/detail in the network tab and select Copy > Copy as cURL (bash) (might be called differently, depending on your browser)
The copied command ends with lots of headers (-H '<something>' -H '<something else>'). Re-create but this office of the command and append it to your OpenDirectoryDownloader control, like and so: OpenDirectoryDownloader --url "https://myopendirectory.com" -H 'header-proper noun-i: header-value-1' -H 'header-name-2: header-value-2' ...
You can of course likewise use other options with this or omit the --url choice to use the prompt instead.

Setting some options similar --username or --user-agent might override some headers, as explicit options accept precedence. Option order does not matter (this applies to OpenDirectoryDownloader in full general).

Copying on Linux

When y'all want to copy (C key or -c flag) the stats at the cease on Linux yous need to take xclip installed.

Linux distros

On some distros yous need extra dependencies. For Alpine: https://docs.microsoft.com/en-us/dotnet/core/install/linux-alpine

For others see: https://docs.microsoft.com/en-us/dotnet/core/install/linux

Cloudflare

EXPERIMANTAL!! READ THIS FIRST!

It WILL NOT ALWAYS WORK!

There is experimental back up for Cloudflare. When it detects a Cloudflare issue it volition download a Chromium browser, start it, in which the Cloudflare protection can be solved. Sometimes this is a captcha which the user (y'all) needs to solve. For each browser session you take sixty seconds to complete. After that the browser volition exist killed and you can retry on next request.

Cloudflare does somehow observe that it is not the normal Chromium/Chrome browser and therefore it sadly will non always work. A skillful tip is move your mouse every bit soon as possible in the browser.

Sometimes it fails and pops up a browser for every request, and too kills information technology almost immediately when Cloudflare sees that there is no trouble with the session. If this happens, kill the indexer!

If anybody have more than info how to get Cloudflare to work amend, let me know!

GitHub

By default GitHub has a rate limit of 60 asking per 60 minutes, which is plenty for 20 repositories with less than 100.000 items. You tin increase this limit to 5000 per hr by creating a (personal) token:

Go to https://github.com/settings/tokens/new
Add together a name similar "OpenDirectoryDownloader"
Y'all don't have to select any scopes!
Click "Generate token"
Get-go OpenDirectoryDownloader with --githubtoken

Docker

Every release volition automatically push an image to the Docker Hub:

https://hub.docker.com/repository/docker/koalabear84/opendirectorydownloader

Run it like:

docker run --rm -v c:/Scans:/app/Scans -information technology koalabear84/opendirectorydownloader --quit --speedtest

It will save the URLs files onto C:\Scans (windows), or supersede with a custom folder on other Os-ses.

* You can as well run information technology without -5 c:/scans:/app/Scans if yous don't desire to save the results on your host.

Google Colab / Jupyter Notebook

Open up https://colab.research.google.com/github/KoalaBear84/OpenDirectoryDownloader/blob/primary/OpenDirectoryDownloader.ipynb
Run step ane to setup the environment and install the latest OpenDirectoryDownloader
Fill in the Url
Run stride ii
Wait until indexing is completed
Urls file can be found in Scans binder (encounter Folder icon on the left sidebar)

Onion / Tor back up

Make certain the Tor is running on your motorcar
Utilize the right proxy accost notation, default for Tor is: "socks5://127.0.0.i:9050"
Start it with --proxy-address parameter

OpenDirectoryDownloader.exe --url "http://*.onion/" --proxy-address "socks5://127.0.0.one:9050"

Getting the code

For Visual Studio (Windows)

Install Visual Studio: https://visualstudio.microsoft.com/vs/community/

With workload: ".Internet Core cross-platform development"
With individual components: Lawmaking tools > Git for Windows and Lawmaking tools > GitHub extension for Visual Studio

Exist sure to install Git: https://git-scm.com/downloads
Clone the repository by clicking "Clone or download" and click "Open in Visual Studio"

For Visual Studio Code

Download Visual Studio Code: https://code.visualstudio.com/download
Be certain to install Git: https://git-scm.com/downloads
Clone the repository: https://code.visualstudio.com/docs/editor/versioncontrol#_cloning-a-repository
More help: https://docs.microsoft.com/en-us/dotnet/core/tutorials/with-visual-studio-code

Edifice

Install the newest .NET seven SDK.
git clone https://github.com/KoalaBear84/OpenDirectoryDownloader
cd OpenDirectoryDownloader/OpenDirectoryDownloader
dotnet build .
cd bin/Debug/net7.0
./OpenDirectoryDownloader --url "https://myopendirectory.com"

For Linux (Might non be needed since .Cyberspace seven):
Then, if you need to package it into a binary, you tin can use warp-packer

When you have cloned the code, you lot can also run it without the SDK. For that, download the "Runtime" and do "dotnet run ." instead of build.

Google Drive

For Google Bulldoze scanning y'all need to become a Google Bulldoze API credentials file, information technology's free!

You can use a many steps manual option, or the 6 steps 'Quickstart' workaround.

Transmission/customized:

Go to https://console.deject.google.com/projectcreate
Fill up in Project Proper noun, like "opendirectorydownloader" or so, go out Location unchanged
Change Project ID (optional)
Click "CREATE"
Wait a couple of seconds until the project is created and open it (click "VIEW")
On the APIs pane, click "Go to APIs overview"
Click "ENABLE APIS AND SERVICES"
Enter "Drive", select "Google Drive API"
Click "ENABLE"
Go to "Credentials" menu in the left bill of fare bar
Click "CONFIGURE CONSENT SCREEN"
Cull "External", click "CREATE"
Make full in something like "opendirectorydownloader" in the "Application name" box
At the bottom click "Save"
Go to "Credentials" menu in the left carte bar (again)
Click "CREATE CREDENTIALS"
Select "OAuth client ID"
Select "Desktop app" equally "Awarding type"
Modify the name (optional)
Click "Create"
Click "OK" in the "OAuth customer created" dialog
In the "OAuth 2.0 Client IDs" section click on the merely create Desktop app line
In the tiptop bar, click "DOWNLOAD JSON"
You volition go a file like "client_secret_xxxxxx.apps.googleusercontent.com.json", rename information technology to "OpenDirectoryDownloader.GoogleDrive.json" and supercede the ane in the release

Wow, they actually fabricated a mess of this..

Alternative method (easier):

This will 'corruption' a 'Quickstart' projection.

Go to https://developers.google.com/drive/api/v3/quickstart/python
Click the "Enabled the Drive API"
"Desktop app" will already be selected on the "Configure your OAuth client" dialog
Click "Create"
Click "DOWNLOAD CLIENT CONFIGURATION"
You will get a file like "credentials.json", rename it to "OpenDirectoryDownloader.GoogleDrive.json" and replace the one in the release

On the showtime use, y'all will go a browser screen that y'all demand to grant access for it, and because we haven't granted out OAuth consent screen (This app isn't verified), we get an extra warning. Yous can utilize the "Avant-garde" link, and use the "Go to yourappname (unsafe)" link.

Support

If yous similar OpenDirectoryDownloader, please consider supporting me!

❤️ Sponsor

Contact me

Reddit https://www.reddit.com/user/KoalaBear84