[MathJax on]

9 Web Downloads

The Download operation has been written by Thomas Breuer, incorporating a number of suggestions from Max Horn, for version 0.77 of Utils. It implements downloading a file from within GAP. It can use the IO or curlInterface packages, or wget or curl, if installed, and it can be extended with other download methods quite easily. It is envisaged that, once other packages have started to use it, and any problems have been addressed, that the functions will be transferred to the main GAP library.

9.1 Functions for downloading files from the web

9.1-1 Download

This function downloads the file with the web address url, which must be a string.

The result is a record which has at least the component success, with value true if the download was successful and false otherwise. In the former case, the component result is bound, whose value is a string that contains the contents of the downloaded file. In the latter case, the component error is bound, whose value is a string that describes the problem.

The function calls the methods stored in the global list Download_Methods until one of them is successful. Currently there are methods based on the GAP functions DownloadURL (curl: DownloadURL) and SingleHTTPRequest (IO: SingleHTTPRequest), and methods based on the external programs wget and curl.

An optional record opt can be given. The following components are supported.

maxTime

If this component is bound then its value must be a nonnegative integer n, meaning that the function gives up after n seconds.

A zero value of n means that no timeout is set, the method will never give up in this case.

The default for n is given by the value of the user preference DownloadMaxTime (see 9.1-3).

target

If this component is bound then its value must be a string that is a local filename, and the function writes the downloaded contents to this file; the returned record does not have a result component in this case.

If the download fails then a file created by the attempt is not left behind, unless resume is set. A file that was already there before the call is not removed, since it is not ours to delete – but note that a failed attempt may well have overwritten it, so its old contents are not guaranteed either.

resume

If this component is bound and has the value true, and target is given, then a partially downloaded file is continued rather than fetched again, and it is kept if the download fails again.

Methods that cannot resume decline the request, since they would discard the partial file that a method which can resume needs; currently the methods based on the external programs curl and wget resume. If none is available the download fails, and the caller can retry without resume.

It is the caller's responsibility that an existing target really is a prefix of what url delivers; otherwise the two get concatenated.

If the server does not support range requests then no data is lost: wget fetches the file again from the start, and curl fails and leaves the partial file alone.

verifyCert

If this component is bound and has the value false then those download methods that are based on curl or wget will omit the check of the server's certificate.

The same effect is achieved for all Download calls by setting the user preference DownloadVerifyCertificate (see 9.1-2) to false and omitting the verifyCert component from opt.

gap> url:= "https://www.gap-system.org/index.html";;
gap> res1:= Download( url );;
gap> res1.success;
true
gap> IsBound( res1.result ) and IsString( res1.result );
true
gap> res2:= Download( Concatenation( url, "xxx" ) );;
gap> res2.success;
false
gap> IsBound( res2.error ) and IsString( res2.error );
true

9.1-2 User preference DownloadVerifyCertificate

The value true (the default) means that the server's certificate is checked in calls of Download (9.1-1), such that nothing gets downloaded if the certificate is invalid.

If the value is false then download methods are supposed to omit the check of the server's certificate (this may not be supported by all download methods).

One can set the value of the preference to be val via SetUserPreference (Reference: SetUserPreference), by calling SetUserPreference( "utils", "DownloadVerifyCertificate", val ), and access the current value via UserPreference (Reference: UserPreference), by calling UserPreference( "utils", "DownloadVerifyCertificate" ).

We recommend leaving this preference at its default value true. Sometimes it can be necessary to change it, e.g. to work around issues with old operating systems which may not be able to correctly verify new certificates. In general it is better to update such a system, but if that is not an option, then disabling certificate checks may be a good last resort.

9.1-3 User preference DownloadMaxTime

The value 0 (the default) means that no timeout is set in calls of Download (9.1-1). If the value is a positive integer n then those download methods that support a timeout will give up after n seconds.

One can set the value of the preference to be val via SetUserPreference (Reference: SetUserPreference), by calling SetUserPreference( "utils", "DownloadMaxTime", val ), and access the current value via UserPreference (Reference: UserPreference), by calling UserPreference( "utils", "DownloadMaxTime" ).

generated by GAPDoc2HTML