1/*  Part of SWI-Prolog
    2
    3    Author:        Jan Wielemaker
    4    E-mail:        J.Wielemaker@vu.nl
    5    WWW:           http://www.swi-prolog.org
    6    Copyright (c)  2007-2016, University of Amsterdam
    7                              VU University Amsterdam
    8    All rights reserved.
    9
   10    Redistribution and use in source and binary forms, with or without
   11    modification, are permitted provided that the following conditions
   12    are met:
   13
   14    1. Redistributions of source code must retain the above copyright
   15       notice, this list of conditions and the following disclaimer.
   16
   17    2. Redistributions in binary form must reproduce the above copyright
   18       notice, this list of conditions and the following disclaimer in
   19       the documentation and/or other materials provided with the
   20       distribution.
   21
   22    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
   23    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
   24    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
   25    FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
   26    COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
   27    INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
   28    BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
   29    LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
   30    CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
   31    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
   32    ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
   33    POSSIBILITY OF SUCH DAMAGE.
   34*/
   35
   36:- module(http_stream,
   37          [ http_chunked_open/3,        % +Stream, -DataStream, +Options
   38            stream_range_open/3,        % +Stream, -DataStream, +Options
   39            multipart_open/3,           % +Stream, +DataStream, +Options)
   40            multipart_open_next/1,      % +DataStream
   41
   42                                        % CGI Stream interaction
   43            cgi_open/4,                 % +Stream, -DataStream, :Hook, +Options
   44            cgi_property/2,             % +Stream, -Property
   45            cgi_set/2,                  % +Stream, -Property
   46            cgi_discard/1,              % +Stream
   47            is_cgi_stream/1,            % +Stream
   48            cgi_statistics/1            % ?Statistics
   49          ]).   50
   51:- use_foreign_library(foreign(http_stream)).   52:- public http_stream_debug/1.          % set debug level
   53
   54:- meta_predicate
   55    stream_range_open(+,-,:).       % onclose option is module sensitive

 http_chunked_open(+RawStream, -DataStream, +Options) is det

Create a stream to realise HTTP chunked encoding or decoding. The technique is similar to library(zlib), using a Prolog stream as a filter on another stream. Options:

close_parent(+Bool)

If true (default false), the parent stream is closed if DataStream is closed.

max_chunk_size(+PosInt)

Define the maximum size of a chunk. Default is the default buffer size of fully buffered streams (4096). Larger values may improve throughput. It is also allowed to use set_stream(DataStream, buffer(line)) on the data stream to get line-buffered output. See set_stream/2 for details. Switching buffering to false is supported.

Here is example code to write a chunked data to a stream

        http_chunked_open(Out, S, []),
        format(S, 'Hello world~n', []),
        close(S).

If a stream is known to contain chunked data, we can extract this data using

        http_chunked_open(In, S, []),
        read_stream_to_codes(S, Codes),
        close(S).

The current implementation does not generate chunked extensions or an HTTP trailer. If such extensions appear on the input they are silently ignored. This is compatible with the HTTP 1.1 specifications. Although a filtering stream is an excellent mechanism for encoding and decoding the core chunked protocol, it does not well support out-of-band data.

After http_chunked_open/3, the encoding of DataStream is the same as the encoding of RawStream, while the encoding of RawStream is octet, the only value allowed for HTTP chunked streams. Closing the DataStream restores the old encoding on RawStream.

Errors

- io_error(read, Stream) where the message context provides an indication of the problem. This error is raised if the input is not valid HTTP chunked data.

  136                 /*******************************
  137                 *             RANGES           *
  138                 *******************************/

 stream_range_open(+RawStream, -DataStream, +Options) is det

DataStream is a stream whose size is defined by the option size(ContentLength). Closing DataStream does not close RawStream. Options processed:

size(+Bytes)

Number of bytes represented by the main stream.

onclose(:Closure)

Calls call(Closure, RawStream, BytesLeft) when DataStream is closed. BytesLeft is the number of bytes of the range stream that have not been read, i.e., 0 (zero) if all data has been read from the stream when the range is closed. This was introduced for supporting Keep-alive in http_open/3 to reschedule the original stream for a new request if the data of the previous request was processed.

  158                 /*******************************
  159                 *            MULTIPART         *
  160                 *******************************/

 multipart_open(+Stream, -DataSttream, +Options) is det

DataStream is a stream that signals end_of_file if the multipart boundary is encountered. The stream can be reset to read the next part using multipart_open_next/1. Options:

close_parent(+Boolean)

Close Stream if DataStream is closed.

boundary(+Text)

Define the boundary string. Text is an atom, string, code or character list.

All parts of a multipart input can be read using the following skeleton:

process_multipart(Stream) :-
      multipart_open(Stream, DataStream, [boundary(...)]),
      process_parts(DataStream).

process_parts(DataStream) :-
      process_part(DataStream),
      (   multipart_open_next(DataStream)
      ->  process_parts(DataStream)
      ;   close(DataStream)
      ).

license

- The multipart parser contains code licensed under the MIT license, based on node-formidable by Felix Geisendoerfer and Igor Afonov.

 multipart_open_next(+DataStream) is semidet

Prepare DataStream to read the next part from the multipart input data. Succeeds if a next part exists and fails if the last part was processed. Note that it is mandatory to read each part up to the end_of_file.

  202                 /*******************************
  203                 *           CGI SUPPORT        *
  204                 *******************************/

 cgi_open(+OutStream, -CGIStream, :Hook, +Options) is det

Process CGI output. OutStream is normally the socket returning data to the HTTP client. CGIStream is the stream the (Prolog) code writes to. The CGIStream provides the following functions:

• At the end of the header, it calls Hook using call(Hook, header, Stream), where Stream is a stream holding the buffered header.
• If the stream is closed, it calls Hook using call(Hook, data, Stream), where Stream holds the buffered data.

The stream calls Hook, adding the event and CGIStream to the closure. Defined events are:

header

Called if the header is complete. Typically it uses cgi_property/2 to extract the collected header and combines these with the request and policies to decide on encoding, transfer-encoding, connection parameters and the complete header (as a Prolog term). Typically it uses cgi_set/2 to associate these with the stream.

send_header

Called if the HTTP header must be sent. This is immediately after setting the transfer encoding to chunked or when the CGI stream is closed. Typically it requests the current header, optionally the content-length and sends the header to the original (client) stream.

close

Called from close/1 on the CGI stream after everything is complete.

The predicates cgi_property/2 and cgi_set/2 can be used to control the stream and store status info. Terms are stored as Prolog records and can thus be transferred between threads.

 cgi_property(+CGIStream, ?Property) is det

Inquire the status of the CGI stream. Defined properties are:

request(-Term)

The original request

header(-Term)

Term is the header term as registered using cgi_set/2

client(-Stream)

Stream is the original output stream used to create this stream.

thread(-ThreadID)

ThreadID is the identifier of the `owning thread'

transfer_encoding(-Tranfer)

One of chunked or none.

connection(-Connection)

One of Keep-Alive or close

content_length(-ContentLength)

Total byte-size of the content. Available in the close handler if the transfer_encoding is none.

header_codes(-Codes)

Codes represents the header collected. Available in the header handler.

state(-State)

One of header, data or discarded

id(-ID)

Request sequence number. This number is guaranteed to be unique.

 cgi_set(+CGIStream, ?Property) is det

Change one of the properies. Supported properties are:

request(+Term)

Associate a request to the stream.

header(+Term)

Register a reply header. This header is normally retrieved from the send_header hook to send the reply header to the client.

connection(-Connection)

One of Keep-Alive or close.

transfer_encoding(-Tranfer)

One of chunked or none. Initially set to none. When switching to chunked from the header hook, it calls the send_header hook and if there is data queed this is send as first chunk. Each subsequent write to the CGI stream emits a chunk.

 cgi_discard(+CGIStream) is det

Discard content produced sofar. It sets the state property to discarded, causing close to omit the writing the data. This must be used for an alternate output (e.g. an error page) if the page generator fails.

 is_cgi_stream(+Stream) is semidet

True if Stream is a CGI stream created using cgi_open/4.

  305:- multifile
  306    http:encoding_filter/3,                 % +Encoding, +In0,  -In
  307    http:current_transfer_encoding/1.       % ?Encoding
  308
  309:- public
  310    http:encoding_filter/3,
  311    http:current_transfer_encoding/1.

 http:encoding_filter(+Encoding, +In0, -In) is semidet

Install a filter to deal with chunked encoded messages. Used by library(http_open).

  318http:encoding_filter(chunked, In0, In) :-
  319    http_chunked_open(In0, In,
  320                      [ close_parent(true)
  321                      ]).

 http:current_transfer_encoding(?Encoding) is semidet

True if Encoding is supported. Used by library(http_open).

  327http:current_transfer_encoding(chunked).

 cgi_statistics(?Term)

Return statistics on the CGI stream subsystem. Currently defined statistics are:

requests(-Integer)

Total number of requests processed

bytes_sent(-Integer)

Total number of bytes sent.

  339cgi_statistics(requests(Requests)) :-
  340    cgi_statistics_(Requests, _).
  341cgi_statistics(bytes_sent(Bytes)) :-
  342    cgi_statistics_(_, Bytes)