4 Plugin modules for rdf_db

The library(rdf_db) module provides several hooks for extending its functionality. Database updates can be monitored and acted upon through the features described in section 3.3. The predicate rdf_load/2 can be hooked to deal with different formats such as rdfturtle, different input sources (e.g. http) and different strategies for caching results.

4.1 Hooks into the RDF library

The hooks below are used to add new RDF file formats and sources from which to load data to the library. They are used by the modules described below and distributed with the package. Please examine the source-code if you want to add new formats or locations.

library(library(semweb/turtle))

Load files in the Turtle format. See section ??.

library(library(semweb/rdf_zlib_plugin))

Load gzip compressed files transparently. See section 4.2.

library(library(semweb/rdf_http_plugin))

Load RDF documents from HTTP servers. See section 4.3.

library(library(http/http_ssl_plugin))

May be combined with library(library(semweb/rdf_http_plugin)) to load RDF from HTTPS servers.

library(library(semweb/rdf_persistency))

Provide persistent backup of the triple store.

library(library(semweb/rdf_cache))

Provide caching RDF sources using fast load/safe files to speedup restarting an application.

rdf_db:rdf_open_hook(+Input, -Stream, -Format)

Open an input. Input is one of file(+Name), stream(+Stream) or url(Protocol, URL). If this hook succeeds, the RDF will be read from Stream using rdf_load_stream/3. Otherwise the default open functionality for file and stream are used.

rdf_db:rdf_load_stream(+Format, +Stream, +Options)

Actually load the RDF from Stream into the RDF database. Format describes the format and is produced either by rdf_input_info/3 or rdf_file_type/2.

rdf_db:rdf_input_info(+Input, -Modified, -Format)

Gather information on Input. Modified is the last modification time of the source as a POSIX time-stamp (see time_file/2). Format is the RDF format of the file. See rdf_file_type/2 for details. It is allowed to leave the output variables unbound. Ultimately the default modified time is `0' and the format is assumed to be xml.

rdf_db:rdf_file_type(?Extension, ?Format)

True if Format is the default RDF file format for files with the given extension. Extension is lowercase and without a '.'. E.g. owl. Format is either a built-in format (xml or triples) or a format understood by the rdf_load_stream/3 hook.

rdf_db:url_protocol(?Protocol)

True if Protocol is a URL protocol recognised by rdf_load/2.

4.2 library(semweb/rdf_zlib_plugin): Reading compressed RDF

This module uses the library(zlib) library to load compressed files on the fly. The extension of the file must be .gz. The file format is deduced by the extension after stripping the .gz extension. E.g. rdf_load('file.rdf.gz').

4.3 library(semweb/rdf_http_plugin): Reading RDF from a HTTP server

This module allows for rdf_load('http://...'). It exploits the library library(http/http_open.pl). The format of the URL is determined from the mime-type returned by the server if this is one of text/rdf+xml, application/x-turtle or application/turtle. As RDF mime-types are not yet widely supported, the plugin uses the extension of the URL if the claimed mime-type is not one of the above. In addition, it recognises text/html and application/xhtml+xml, scanning the XML content for embedded RDF.

4.4 library(semweb/rdf_cache): Cache RDF triples

The library library(semweb/rdf_cache) defines the caching strategy for triples sources. When using large RDF sources, caching triples greatly speedup loading RDF documents. The cache library implements two caching strategies that are controlled by rdf_set_cache_options/1.

Local caching This approach applies to files only. Triples are cached in a sub-directory of the directory holding the source. This directory is called .cache (_cache on Windows). If the cache option create_local_directory is true, a cache directory is created if posible.

Global caching This approach applies to all sources, except for unnamed streams. Triples are cached in directory defined by the cache option global_directory.

When loading an RDF file, the system scans the configured cache files unless cache(false) is specified as option to rdf_load/2 or caching is disabled. If caching is enabled but no cache exists, the system will try to create a cache file. First it will try to do this locally. On failure it will try to configured global cache.

rdf_set_cache_options(+Options)

Change the cache policy. Provided options are:

• enabled(Boolean) If true, caching is enabled.
• local_directory(Name). Plain name of local directory. Default .cache (_cache on Windows).
• create_local_directory(Bool) If true, try to create local cache directories
• global_directory(Dir) Writeable directory for storing cached parsed files.
• create_global_directory(Bool) If true, try to create the global cache directory.

[semidet]rdf_cache_file(+URL, +ReadWrite, -File)

File is the cache file for URL. If ReadWrite is read, it returns the name of an existing file. If write it returns where a new cache file can be overwritten or created.

4.5 library(semweb/rdf_litindex): Indexing words in literals

The library library(semweb/rdf_litindex.pl) exploits the primitives of section 4.5.1 and the NLP package to provide indexing on words inside literal constants. It also allows for fuzzy matching using stemming and `sounds-like' based on the double metaphone algorithm of the NLP package.

rdf_find_literals(+Spec, -ListOfLiterals)

Find literals (without type or language specification) that satisfy Spec. The required indices are created as needed and kept up-to-date using hooks registered with rdf_monitor/2. Numerical indexing is currently limited to integers in the range ±2^30 (±2^62 on 64-bit platforms). Spec is defined as:

and(Spec1, Spec2)

Intersection of both specifications.

or(Spec1, Spec2)

Union of both specifications.

not(Spec)

Negation of Spec. After translation of the full specification to Disjunctive Normal Form (DNF), negations are only allowed inside a conjunction with at least one positive literal.

case(Word)

Matches all literals containing the word Word, doing the match case insensitive and after removing diacritics.

stem(Like)

Matches all literals containing at least one word that has the same stem as Like using the Porter stem algorithm. See NLP package for details.

sounds(Like)

Matches all literals containing at least one word that `sounds like' Like using the double metaphone algorithm. See NLP package for details.

prefix(Prefix)

Matches all literals containing at least one word that starts with Prefix, discarding diacritics and case.

between(Low, High)

Matches all literals containing an integer token in the range Low..High, including the boundaries.

ge(Low)

Matches all literals containing an integer token with value Low or higher.

le(High)

Matches all literals containing an integer token with value High or lower.

Token

Matches all literals containing the given token. See tokenize_atom/2 of the NLP package for details.

rdf_token_expansions(+Spec, -Expansions)

Uses the same database as rdf_find_literals/2 to find possible expansions of Spec, i.e. which words `sound like', `have prefix', etc. Spec is a compound expression as in rdf_find_literals/2. Expansions is unified to a list of terms sounds(Like, Words), stem(Like, Words) or prefix(Prefix, Words). On compound expressions, only combinations that provide literals are returned. Below is an example after loading the ULAN^{2Unified List of Artist Names from the Getty Foundation.} database and showing all words that sounds like `rembrandt' and appear together in a literal with the word `Rijn'. Finding this result from the 228,710 literals contained in ULAN requires 0.54 milliseconds (AMD 1600+).

?- rdf_token_expansions(and('Rijn', sounds(rembrandt)), L).

L = [sounds(rembrandt, ['Rambrandt', 'Reimbrant', 'Rembradt',
                        'Rembrand', 'Rembrandt', 'Rembrandtsz',
                        'Rembrant', 'Rembrants', 'Rijmbrand'])]

Here is another example, illustrating handling of diacritics:

?- rdf_token_expansions(case(cafe), L).

L = [case(cafe, [cafe, caf\'e])]

rdf_tokenize_literal(+Literal, -Tokens)

Tokenize a literal, returning a list of atoms and integers in the range -1073741824 ... 1073741823. As tokenization is in general domain and task-dependent this predicate first calls the hook rdf_litindex:tokenization(Literal, -Tokens). On failure it calls tokenize_atom/2 from the NLP package and deletes the following: atoms of length 1, floats, integers that are out of range and the english words and, an, or, of, on, in, this and the. Deletion first calls the hook rdf_litindex:exclude_from_index(token, X). This hook is called as follows:

no_index_token(X) :-
        exclude_from_index(token, X), !.
no_index_token(X) :-
        ...

4.5.1 Literal maps: Creating additional indices on literals

`Literal maps' provide a relation between literal values, intended to create additional indexes on literals. The current implementation can only deal with integers and atoms (string literals). A literal map maintains an ordered set of keys. The ordering uses the same rules as described in section 4.5. Each key is associated with an ordered set of values. Literal map objects can be shared between threads, using a locking strategy that allows for multiple concurrent readers.

Typically, this module is used together with rdf_monitor/2 on the channals new_literal and old_literal to maintain an index of words that appear in a literal. Further abstraction using Porter stemming or Metaphone can be used to create additional search indices. These can map either directly to the literal values, or indirectly to the plain word-map. The SWI-Prolog NLP package provides complimentary building blocks, such as a tokenizer, Porter stem and Double Metaphone.

rdf_new_literal_map(-Map)

Create a new literal map, returning an opaque handle.

rdf_destroy_literal_map(+Map)

Destroy a literal map. After this call, further use of the Map handle is illegal. Additional synchronisation is needed if maps that are shared between threads are destroyed to guarantee the handle is no longer used. In some scenarios rdf_reset_literal_map/1 provides a safe alternative.

rdf_reset_literal_map(+Map)

Delete all content from the literal map.

rdf_insert_literal_map(+Map, +Key, +Value)

Add a relation between Key and Value to the map. If this relation already exists no action is performed.

rdf_insert_literal_map(+Map, +Key, +Value, -KeyCount)

As rdf_insert_literal_map/3. In addition, if Key is a new key in Map, unify KeyCount with the number of keys in Map. This serves two purposes. Derived maps, such as the stem and metaphone maps need to know about new keys and it avoids additional foreign calls for doing the progress in rdf_litindex.pl.

rdf_delete_literal_map(+Map, +Key)

Delete Key and all associated values from the map. Succeeds always.

rdf_delete_literal_map(+Map, +Key, +Value)

Delete the association between Key and Value from the map. Succeeds always.

[det]rdf_find_literal_map(+Map, +KeyList, -ValueList)

Unify ValueList with an ordered set of values associated to all keys from KeyList. Each key in KeyList is either an atom, an integer or a term not(Key). If not-terms are provided, there must be at least one positive keywords. The negations are tested after establishing the positive matches.

rdf_keys_in_literal_map(+Map, +Spec, -Answer)

Realises various queries on the key-set:

all

Unify Answer with an ordered list of all keys.

key(+Key)

Succeeds if Key is a key in the map and unify Answer with the number of values associated with the key. This provides a fast test of existence without fetching the possibly large associated value set as with rdf_find_literal_map/3.

prefix(+Prefix)

Unify Answer with an ordered set of all keys that have the given prefix. Prefix must be an atom. This call is intended for auto-completion in user interfaces.

ge(+Min)

Unify Answer with all keys that are larger or equal to the integer Min.

le(+Max)

Unify Answer with all keys that are smaller or equal to the integer Max.

between(+Min, +Max)

Unify Answer with all keys between Min and Max (including).

rdf_statistics_literal_map(+Map, +Key(-Arg...))

Query some statistics of the map. Provides keys are:

size(-Keys, -Relations)

Unify Keys with the total key-count of the index and Relation with the total Key-Value count.

4.6 library(semweb/rdf_persistency): Providing persistent storage

The library(semweb/rdf_persistency) provides reliable persistent storage for the RDF data. The store uses a directory with files for each source (see rdf_source/1) present in the database. Each source is represented by two files, one in binary format (see rdf_save_db/2) representing the base state and one represented as Prolog terms representing the changes made since the base state. The latter is called the journal.

rdf_attach_db(+Directory, +Options)

Attach Directory as the persistent database. If Directory does not exist it is created. Otherwise all sources defined in the directory are loaded into the RDF database. Loading a source means loading the base state (if any) and replaying the journal (if any). The current implementation does not synchronise triples that are in the store before attaching a database. They are not removed from the database, nor added to the presistent store. Different merging options may be supported through the Options argument later. Currently defined options are:

concurrency(+PosInt)

Number of threads used to reload databased and journals from the files in Directory. Default is the number of physical CPUs determined by the Prolog flag cpu_count or 1 (one) on systems where this number is unknown. See also concurrent/3.

max_open_journals(+PosInt)

The library maintains a pool of open journal files. This option specifies the size of this pool. The default is 10. Raising the option can make sense if many writes occur on many different named graphs. The value can be lowered for scenarios where write operations are very infrequent.

silent(Boolean)

If true, supress loading messages from rdf_attach_db/2.

log_nested_transactions(Boolean)

If true, nested log transactions are added to the journal information. By default (false), no log-term is added for nested transactions.

The database is locked against concurrent access using a file lock in Directory. An attempt to attach to a locked database raises a permission_error exception. The error context contains a term rdf_locked(Args), where args is a list containing time(Stamp) and pid(PID). The error can be caught by the application. Otherwise it prints:

ERROR: No permission to lock rdf_db `/home/jan/src/pl/packages/semweb/DB'
ERROR: locked at Wed Jun 27 15:37:35 2007 by process id 1748

rdf_detach_db

Detaches the persistent store. No triples are removed from the RDF triple store.

rdf_current_db(-Directory)

Unify Directory with the current database directory. Fails if no persistent database is attached.

rdf_persistency(+DB, +Bool)

Change presistency of named database (4th argument of rdf/4). By default all databases are presistent. Using false, the journal and snapshot for the database are deleted and further changes to triples associated with DB are not recorded. If Bool is true a snapshot is created for the current state and further modifications are monitored. Switching persistency does not affect the triples in the in-memory RDF database.

rdf_flush_journals(+Options)

Flush dirty journals. With the option min_size(KB) only journals larger than KB Kbytes are merged with the base state. Flushing a journal takes the following steps, ensuring a stable state can be recovered at any moment.

1. Save the current database in a new file using the extension .new.
2. On success, delete the journal
3. On success, atomically move the .new file over the base state.

Note that journals are not merged automatically for two reasons. First of all, some applications may decide never to merge as the journal contains a complete changelog of the database. Second, merging large databases can be slow and the application may wish to schedule such actions at quiet times or scheduled maintenance periods.

4.6.1 Enriching the journals

The above predicates suffice for most applications. The predicates in this section provide access to the journal files and the base state files and are intented to provide additional services, such as reasoning about the journals, loaded files, etc.^{3A library library(rdf_history) is under development exploiting these features supporting wiki style editing of RDF.}

Using rdf_transaction(Goal, log(Message)), we can add additional records to enrich the journal of affected databases with Term and some additional bookkeeping information. Such a transaction adds a term begin(Id, Nest, Time, Message) before the change operations on each affected database and end(Id, Nest, Affected) after the change operations. Here is an example call and content of the journal file mydb.jrn. A full explanation of the terms that appear in the journal is in the description of rdf_journal_file/2.

?- rdf_transaction(rdf_assert(s,p,o,mydb), log(by(jan))).

start([time(1183540570)]).
begin(1, 0, 1183540570.36, by(jan)).
assert(s, p, o).
end(1, 0, []).
end([time(1183540578)]).

Using rdf_transaction(Goal, log(Message, DB)), where DB is an atom denoting a (possibly empty) named graph, the system guarantees that a non-empty transaction will leave a possibly empty transaction record in DB. This feature assumes named graphs are named after the user making the changes. If a user action does not affect the user's graph, such as deleting a triple from another graph, we still find record of all actions performed by some user in the journal of that user.

rdf_journal_file(?DB, ?JournalFile)

True if File is the absolute file name of an existing named graph DB. A journal file contains a sequence of Prolog terms of the following format.^{4Future versions of this library may use an XML based language neutral format.}

start(Attributes)

Journal has been opened. Currently Attributes contains a term time(Stamp).

end(Attributes)

Journal was closed. Currently Attributes contains a term time(Stamp).

assert(Subject, Predicate, Object)

A triple {Subject, Predicate, Object} was added to the database.

assert(Subject, Predicate, Object, Line)

A triple {Subject, Predicate, Object} was added to the database with given Line context.

retract(Subject, Predicate, Object)

A triple {Subject, Predicate, Object} was deleted from the database. Note that an rdf_retractall/3 call can retract multiple triples. Each of them have a record in the journal. This allows for `undo'.

retract(Subject, Predicate, Object, Line)

Same as above, for a triple with associated line info.

update(Subject, Predicate, Object, Action)

See rdf_update/4.

begin(Id, Nest, Time, Message)

Added before the changes in each database affected by a transaction with transaction identifier log(Message). Id is an integer counting the logged transactions to this database. Numbers are increasing and designed for binary search within the journal file. Nest is the nesting level, where `0' is a toplevel transaction. Time is a time-stamp, currently using float notation with two fractional digits. Message is the term provided by the user as argument of the log(Message) transaction.

end(Id, Nest, Others)

Added after the changes in each database affected by a transaction with transaction identifier log(Message). Id and Nest match the begin-term. Others gives a list of other databases affected by this transaction and the Id of these records. The terms in this list have the format DB:Id.

rdf_db_to_file(?DB, ?FileBase)

Convert between DB (see rdf_source/1) and file base-file used for storing information on this database. The full file is located in the directory described by rdf_current_db/1 and has the extension .trp for the base state and .jrn for the journal.