API Documentation

This documentation covers version 5 of the Flying Sphinx HTTP API - this is the version that the latest releases of the Ruby and NodeJS clients use.

Version 3 is also documented - at this point in time only the Python library still uses this, and if you've built your own client, it's highly recommended using version 5, as it's faster, smarter, and just generally better.

Please note: if you don't understand Sphinx, this is not the place to start. This is just an API for managing Sphinx when hosted on Flying Sphinx.

Overview

The v5 API takes a different approach compared to previous versions. Instead of many endpoints for each of the available actions, there is instead one key action. All actions are asynchronous, and updates are received through a WebSocket connection, rather than polling.

Also, configuration files are uploaded in separate requests (via unique pre-signed credentials), and then the path to the uploaded file is sent to the API as part of registering the action.

URIs and Content

The API is accessible over HTTPS through https://flying-sphinx.com/api/my/v5/, and always responds with JSON. The default response is a JSON object with the key status and the value OK. Calls that return additional information are noted as required below.

Authentication and Headers

All requests are required to have a HMAC-signed header which both confirms your identity and the authenticity of your request.

The payload for the signature is each of the following items in this specified order, concatenated together with \n separators:

  • The HTTP method, upper-cased (e.g. POST or GET)
  • The value of the Content-Type header
  • The value of the Content-Digest header
  • The value of the Date header
  • The request path (e.g. /api/my/v5/perform)

Each of the above headers must also be provided. The Content-Digest header must be the MD5 digest of the request's body, represented in hexadecimal characters. If there is no body (in the case of GET requests), then this can be blank.

The HMAC signature must use SHA256, using your FLYING_SPHINX_API_KEY value for encryption, and be represented in base64 characters. It must be added to the Authorization header as follows (with the appropriate identifier and signature substituted in):

Authorization: Thebes FLYING_SPHINX_IDENTIFIER:SIGNATURE

It's also preferred that you provide another header which specifies the version and language of the library used to talk to the API. For example, the Python client uses 0.2.0+python, and the Node.js one uses 1.0.0+js. If the code you're writing isn't versioned, then some information identifying you or your project would be nice.

X-Flying-Sphinx-Version: 0.7.0+ruby

Registering Actions

All actions to manage the configuration, running state and indexing are managed through a single endpoint:

POST /api/my/v5/perform

Parameters:

To be sent through in the request body as form values (i.e. where the content type is application/x-www-form-urlencoded).

  • action: The type of action to register. Available options are listed below.
  • path: Optional string of the path where a configuration file has been uploaded. e.g. ac7c855ebb718c6e5/2019-07-14/cd0cf3b1-36bf-46cb-a801-07aa2c9a6bf9.tar.gz. Each path value is supplied via the presignature endpoint below, and is unique.
  • indices: Optional string which lists the names of indexes to process when performing the index action. This should be comma separated. If it is not supplied, all indices will be processed. eg. article_core,article_delta.
  • core_index: Optional string which is the name of the core index for delta documents to be merged into. Only used by the merge action.
  • delta_index: Optional string which is the name of the index that delta documents are sourced from. Only used by the merge action.
  • filters: Optional JSON object of filters to determine which delta documents are included. Only used by the merge action.

Actions:

  • clear: Removes all stored index and binlog files.
  • configure: updates the configuration files. This action requires the path parameter.
  • index: Processes the specified SQL-backed indices (or all of them, if none are specified via the indices parameter).
  • merge: Merges documents from one index (as named in the delta_index parameter) into another (named in the core_index paramter. If only some documents are to be merged, the appropriate attribute filters can be given using the filters parameter.
  • rebuild: Restarts the daemon, clears the index files, and re-processes all indices. If the path parameter is supplied, configuration files are overwritten between stopping and starting the daemon.
  • reset: Restarts the daemon, and clears the index files. If the path parameter is supplied, configuration files are overwritten between stopping and starting the daemon.
  • restart: Restarts the Sphinx daemon.
  • rotate: Rotates index files.
  • start: Starts the Sphinx daemon.
  • stop: Stops the Sphinx daemon.

Response:

Returned as JSON.

  • status: a string indicating whether the action was registered. If everything was fine, it will be OK. Other known states include BLOCKED (when attempting single-index processing on a plan that doesn't allow it), and UNCONFIGURED when trying to process indices when there's no configuration or no SQL-backed indices available.
  • id: The unique integer representing the action. This is useful later to determine whether an action is complete via WebSocket messages.

Requesting pre-signature credentials for configuration uploads

Retrieve a unique set of credentials for uploading a gzipped tar archive containing the Sphinx configuration files.

GET /api/my/v5/presignature
No parameters are required for this request.

Response:

Returned as JSON.

  • status: a string indicating whether the action was registered. If everything was fine, it will be OK.
  • path: The unique path which can be passed in when registering actions after this file is uploaded. e.g. ac7c855ebb718c6e5/2019-07-14/cd0cf3b1-36bf-46cb-a801-07aa2c9a6bf9.tar.gz.
  • url: The URL to POST the configuration archive to.
  • fields: The form parameters to include when uploading the configuration archive.

Uploading a configuration archive

Once you've received pre-signature details, you should send a POST request to the given URL, with the given fields and the gzipped tar archive contents as the file parameter.

After that upload is finished, you can then send the provided path value to an action that involves configuration (such as configure, rebuild, and reset).

Archive:

All files are optional.

  • sphinx/raw.conf: Your Sphinx configuration file. Flying Sphinx will rewrite the provided configuration to set paths appropriately, so don't feel you need to get carried away with exactly where things are located. Index and source names are maintained.
  • sphinx/version.txt: The preferred Sphinx/Manticore version. Expected values are covered in the documentation. e.g. 2.2.11.
  • sphinx/engine.txt: The preferred search library. Defaults to sphinx, but can also be manticore.
  • sphinx/extra.txt: A string list of setting files, separated by semi-colons, and prefixed by the relevant setting. e.g. wordforms/words.txt;stopwords/stops.txt.

Additionally, any setting files should also be added, as per their key in sphinx/extra.txt. So, if that file contained wordforms/words.txt;stopwords/stops.txt, you would add two files: wordforms/words.txt, which contained the wordforms settings, and stopwords/stops.txt, containing the stopwords settings.

These settings must start with the setting name and a slash, and then the name of the file follows.

Action results via websockets

The results of registered actions are communicated via websockets using Pusher's channel system. You either need to use a third-party client for your language of choice, or build your own.

To connect to their server, you will want to use the App Key a8518107ea8a18fe5559. You do not need an App ID or Secret. Once connected, subscribe to a channel named after your identifier (FLYING_SPHINX_IDENTIFIER).

Once you've subscribed to this channel, you can then register your chosen action. The messages you need to pay attention will have one of the two event types: completion and failure. Both will come with a data parameter, which is a JSON-encoded object.

The data object has just one key:

Once you've received either a completion or failure event for your action, you can disconnect from the websocket server.

Searching is done by connecting to Sphinx directly - and the address and port are provided through the FLYING_SPHINX_HOST and FLYING_SPHINX_PORT environment variables. You can also access this information through an API call:

GET /api/my/v5/app

Response

  • server: The server to direct search queries to.
  • port: The port to connect through to talk to Sphinx.