highlight.pl -- Highlight token server

This module provides the Prolog part of server-assisted highlighting for SWISH. It is implemented by managing a shadow copy of the client editor on the server. On request, the server computes a list of semantic tokens.

To be done: - Use websockets

codemirror_change(+Request)[private]

Handle changes to the codemirror instances. These are sent to us using a POST request. The request a POSTed JSON object containing:

uuid: string holding the editor's UUID
change: the change object, which holds:
- from: Start position as {line:Line, ch:Ch}
- to: End position
- removed: list(atom) of removed text
- text: list(atom) of inserted text
- origin: what caused this change event
- next: optional next change event.

Reply is JSON and either 200 with true or 409 indicating that the editor is not known.

apply_change(+TB, -Changed, +Changes) is det[private]

Note that the argument order is like this to allow for maplist.

Arguments:

Changed - is left unbound if there are no changes or unified to true if something has changed.

throws

- cm(outofsync) if an inconsistent delete is observed.

create_editor(+UUID, -Editor, +Change) is det[private]

Create a new editor for source UUID from Change. The editor is created in a locked state and must be released using release_editor/1 before it can be publically used.

current_highlight_state(?UUID, -State) is nondet

Return info on the current highlighter

uuid_like(+UUID) is semidet[private]

Do some sanity checking on the UUID because we use it as a temporary module name and thus we must be quite sure it will not conflict with anything.

destroy_editor(+UUID)[private]

Destroy source admin UUID: the shadow text (a memory file), the XREF data and the module used for cross-referencing. The editor must be acquired using fetch_editor/2 before it can be destroyed.

gc_editors[private]

Garbage collect all editors that have not been accessed for 60 minutes.

To be done: - Normally, deleting a highlight state can be done aggressively as it will be recreated on demand. But, coloring a query passes the UUIDs of related sources and as yet there is no way to restore this. We could fix that by replying to the query colouring with the UUIDs for which we do not have sources, after which the client retry the query-color request with all relevant sources.

fetch_editor(+UUID, -MemFile) is semidet[private]

Fetch existing editor for source UUID. Update the last access time. After success, the editor is locked and must be released using release_editor/1.

check_unlocked(+Reason)[private]

Verify that all editors locked by this thread are unlocked again.

update_access(+UUID)[private]

Update the registered last access. We only update if the time is behind for more than a minute.

prolog:xref_open_source(+UUID, -Stream)[multifile]

Open a source. As we cannot open the same source twice we must lock it. As of 7.3.32 this can be done through the prolog:xref_close_source/2 hook. In older versions we get no callback on the close, so we must leave the editor unlocked.

codemirror_leave(+Request)[private]

POST handler that deals with destruction of our mirror associated with an editor, as well as the associated cross-reference information.

mark_changed(+MemFile, ?Changed) is det[private]

Mark that our cross-reference data might be obsolete

xref(+UUID) is det[private]

xref_source_id(+Editor, -SourceID) is det[private]

SourceID is the xref source identifier for Editor. As we are using UUIDs we just use the editor.

xref_state_module(+UUID, -Module) is semidet[private]

True if we must run the cross-referencing in Module. We use a temporary module based on the UUID of the source.

codemirror_tokens(+Request)[private]

HTTP POST handler that returns an array of tokens for the given editor.

json_source_id(+Input, -SourceID)[private]

Translate the Input, which is either a string or a list of strings into an atom or list of atoms. Older versions of SWI-Prolog only accept a single atom source id.

shadow_editor(+Data, -MemoryFile) is det[private]

Get our shadow editor:

If we have one, it is updated from either the text or the changes.
If we have none, but there is a text property, create one from the text.
If there is a role property, create an empty one.

This predicate fails if the server thinks we have an editor with state that must be reused, but this is not true (for example because we have been restarted).

throws: - cm(existence_error) if the target editor did not exist; - cm(out_of_sync) if the changes do not apply due to an internal error or a lost message.

show_mirror(+Role) is det[private]

server_tokens(+Role) is det[private]

These predicates help debugging the server side. show_mirror/0 displays the text the server thinks is in the client editor. The predicate server_tokens/1 dumps the token list.

Arguments:

Role - is one of source or query, expressing the role of the editor in the SWISH UI.

server_tokens(+TextBuffer, -Tokens) is det[private]

Arguments:

Tokens

- is a nested list of Prolog JSON terms. Each group represents the tokens found in a single toplevel term.

group_by_term(+Tokens, -Nested) is det[private]

Group the tokens by input term. This simplifies incremental updates of the token list at the client sides as well as re-syncronizing. This predicate relies on the fullstop token that is emitted at the end of each input term.

json_token(+TB, -Start, -JSON) is nondet[private]

Extract the stored terms.

To be done: - We could consider to collect the attributes in the colour_item/4 callback and maintain a global variable instead of using assert/retract. Most likely that would be faster. Need to profile to check the bottleneck.

style(+StyleIn) is semidet[private]

style(+StyleIn, -SWISHType:atomOrPair, -Attributes:list)[private]

Declare that we map StyleIn as generated by library(prolog_colour) into a token of type SWISHType, providing additional context information based on Attributes. Elements of Attributes are terms of the form Name(Value) or the atom text. The latter is mapped to text(String), where String contains the text that matches the token character range.

The resulting JSON token object has a property type, containing the SWISHType and the properties defined by Attributes.

Additional translations can be defined by adding rules for the multifile predicate style/3. The base type, which refers to the type generated by the SWISH tokenizer must be specified by adding an attribute base(BaseType). For example, if the colour system classifies an atom as refering to a database column, library(prolog_colour) may emit db_column(Name) and the following rule should ensure consistent mapping:

swish_highlight:style(db_column(Name),
                      db_column, [text, base(atom)]).

goal_arity(+Goal, -Arity) is det[private]

Get the arity of a goal safely in SWI7

swish_config:config(-Name, -Styles) is nondet[multifile]

Provides the object config.swish.style, a JSON object that maps style properties of user-defined extensions of library(prolog_colour). This info is used by the server-side colour engine to populate the CodeMirror styles.

To be done: - Provide summary information

x11_color(+Name, -R, -G, -B)[private]

True if RGB is the color for the named X11 color.

css(?Context, ?Selector, -Style) is nondet[multifile]

Multifile hook to define additional style to apply in a specific context. Currently defined contexts are:

hover: Used for CodeMirror hover extension.

Arguments:

`Selector`	- is a CSS selector, which is refined by `Context`
`Style`	- is a list of Name(Value) terms.

token_info(+Request)[private]

HTTP handler that provides information about a token.

token_info(+Token:dict)// is det[multifile]

Generate HTML, providing details about Token. Token is a dict, providing the enriched token as defined by style/3. This multifile non-terminal can be hooked to provide details for user defined style extensions.

token_predicate_module(+Token, -Module) is semidet[private]

Try to extract the module from the token.

predicate_info(+PI, -Info:list(dict)) is det[private]

Info is a list of dicts providing details about predicates that match PI. Fields in dict are:

module:Atom: Module of the predicate
name:Atom: Name of the predicate
arity:Integer: Arity of the predicate
summary:Text: Summary text extracted from the system manual or PlDoc
iso:Boolean: Presend and true if the predicate is an ISO predicate

predicate_info(?PI, -Key, -Value) is nondet[private]

Find information about predicates from the system, manual and PlDoc. First, we deal with ISO predicates that cannot be redefined and are documented in the manual. Next, we deal with predicates that are documented in the manual.

bug: - : Handling predicates documented in the manual is buggy because their definition may be overruled by the user. We probably must include the file into the equation.

show_mirror(+Role) is det[private]

server_tokens(+Role) is det[private]

These predicates help debugging the server side. show_mirror/0 displays the text the server thinks is in the client editor. The predicate server_tokens/1 dumps the token list.

Arguments:

Role - is one of source or query, expressing the role of the editor in the SWISH UI.

style(+StyleIn) is semidet[multifile]

style(+StyleIn, -SWISHType:atomOrPair, -Attributes:list)[multifile]

The resulting JSON token object has a property type, containing the SWISHType and the properties defined by Attributes.

swish_highlight:style(db_column(Name),
                      db_column, [text, base(atom)]).