HTTPGETN

Synopsis

HTTPGETN [/?] [/REFERER string] [/HOST string] [/USERAGENT string] [/ACCEPT string] [/PROXY string] [/BYIP[+|-]] [/SRV_SUPERDOMAINS[+|-]] [/SRV_SELF[+|-]] [/IMPLICIT[+|-]] [/IP6ADDR[+|-]] [/CLIENTIP[+|-]] [/CLIENTPORT[+|-]] [/LOWER number] [/UPPER number] [/NWIDTH number] [/DONEFILE string] domain remote-prefix remote-suffix local-prefix local-suffix

Description

The HTTPGETN command retrives multiple file, following a fixed numbered pattern, from a content HTTP server and stores them in local files with the same numbered pattern.

The remote and local files all have a central numeric part, with a fixed width. HTTPGETN iterates all (positive) integer values from the lower to the upper bound, generating the appropriate URLs and local filenames, and issues one HTTP request per value.

The URLs of the files on the server follow the form http://domainremote-prefixNremote-suffix. This is because the remote-prefixNremote-suffix portion is defined, by the protocol specification, to always be an absolute path, that begins with a forward slash character.

Note: Because a forward slash is a command-line option character, one must bypass standard option recognition when invoking HTTPGETN. Conveniently, the double slash method of doing this is syntactically similar to a URL. For example:

[c:\]httpget // example.com /absolute/path/name ext localname ext

HTTPGETN sends HTTP/1.1 GET requests. It is capable of supporting both the chunked transfer encoding and the vanilla HTTP/1.0 transfer encoding.

HTTPGETN will attempt to preserve file timestamps from the server. If there is a Last-Modified: header in a response, HTTPGETN will attempt to set the last modified date of the file to the same value.

HTTPGETN will not touch existing files if they exist, but will skip them in the numeric sequence. This allows interrupted prior transfer batches to be resumed. However, it will not resume the transfer of a file whose prior transfer was interrupted partway through. It is necessary to delete (or rename) the local file in such cases if it is desired to attempt another transfer.

HTTPGETN attaches a description (.SUBJECT) extended attribute to each file comprising the URL from which it was downloaded.

HTTPGETN looks up domain to find the content HTTP server to contact. It uses the same DNS lookup process for locating content HTTP servers as the proxy HTTP server dæmon does. It is SRV resource record aware and will respect any such records if found.

By default, domain is also the hostname sent in the GET request.

Command-specific options

/REFERER: Send a Referer: header, with the specified content, in the GET request.
/HOST: Send a Host: header, with the specified content, in the GET request.
/USERAGENT: Send a User-Agent: header, with the specified content, in the GET request.
/ACCEPT: Send a Accept: header, with the specified content, in the GET request.
/PROXY: Connect to a specified proxy HTTP server instead of directly to the content HTTP server. HTTPGETN makes no changes to the actual request sent if a proxy server is specified.
/LOWER: Specify the lower numeric bound. The default is 0.
/UPPER: Specify the upper numeric bound. The default is 0.
/NWIDTH: Specify the format width for the numeric portion of the URLs and filenames. The default is 0.
/DONEFILE: If a filename is specified with this option, HTTPGETN will write a count of the number of files successfully transferred to this file when it has finished iterating over all numeric values.