Documentation

Flying Sphinx isn't much more than a wrapper over Sphinx (and if you're using Ruby and ActiveRecord, Thinking Sphinx), which automates the connection to a managed Sphinx server for both searching and indexing. So for the most part, there's not too much different to a normal Sphinx workflow.

If you're not familiar with Sphinx, then it's highly recommended you read through the Sphinx documentation and have it set up and working in your local development environment.

Regions

Flying Sphinx is available in both US and European regions.

Languages

Flying Sphinx currently has client libraries for Ruby/Rails, Python and Node.js. If you wish to use a different language, you'll need to talk to the Flying Sphinx API directly.

If you are using Ruby and Thinking Sphinx, then it's highly recommended you read through the documentation for that library if you're not already familiar with it:

Installation

First up, let Heroku know you want to use the Flying Sphinx add-on:

heroku addons:add flying_sphinx:wooden

You can find a list of plans on Heroku's website.

Ruby and ActiveRecord/Rails 3.2 or newer

You'll need the Flying Sphinx gem as part of your Rails application. If you're using Bundler, it's as simple as adding it to your Gemfile and bundling as per normal:

gem 'thinking-sphinx', '~> 4.0'
gem 'flying-sphinx',   '~> 2.0'

If you're using MRI 1.8, you'll also need the openssl-nonblock gem:

gem 'openssl-nonblock', '0.2.1'

Ruby and ActiveRecord/Rails 2

A few things to note here:

  • Only Rails 2.3.6 or later is supported.
  • The openssl-nonblock gem is required if you're using MRI 1.8.

And if you're not using Bundler, add the gem to both your config/environment.rb file:

config.gem 'thinking-sphinx',
  :version => '1.5.0'
config.gem 'flying-sphinx',
  :version => '1.2.0'
config.gem 'openssl-nonblock'

… and your .gems file:

thinking-sphinx --version 1.5.0
flying-sphinx --version 1.2.0
openssl-nonblock --version 0.2.1

Lastly, if you're using Rails 2.3 you need to add this line to the end of your Rakefile:

# Not required for Rails 3+
require 'flying_sphinx/tasks'

In my examples here, you'll note that I'm always referencing the Flying Sphinx gem after the Thinking Sphinx gem. I recommend you do the same, otherwise you may end up with some dependency confusion.

You must use Thinking Sphinx 2.1.0 or later (for Rails 3+), or 1.5.0 or later (for Rails 2.3). Older versions of Thinking Sphinx will not work with Flying Sphinx.

Python

The flyingsphinx package is available via pip:

$ pip install flyingsphinx

Node/Javascript

The flying-sphinx package is available via npm:

$ npm install flying-sphinx

Advanced Heroku Database Configurations with Ruby

Thinking Sphinx (and thus, Flying Sphinx) uses the connection attributes at the time the index is defined. Unless you're doing something particularly creative, this occurs when the model is loaded, and is referencing the default database credentials (inserted into config/database.yml from DATABASE_URL).

If you want to do something not quite so standard, please contact Flying Sphinx support to talk through what approaches could work better for your setup.

Back to the top

The heroku-20 stack

Heroku's current default stack is heroku-20, which comes with client libraries for MySQL v8. However, the default (and stable) Sphinx version is v2.2.11, and MySQL clients that communicate with this must be compiled using the v5.7 client libraries.

To include these old libraries, the following steps are required:

  • Add the apt buildstack ahead of your standard buildpack:
$ heroku buildpacks:add --index 1 heroku-community/apt
  • Create an Aptfile in the root of your project, and add the MySQL 5.7 client tools (as a URL for the deb file) to it. Please note that the latest available (5.7) release is best - at the time of writing, the latest is 5.7.35, but you may want to verify via MySQL's website (filter to Ubuntu, aim for 18.04 on AMD 64bit, DEB Package).
# in Aptfile:
https://cdn.mysql.com/Downloads/MySQL-5.7/libmysqlclient-dev_5.7.35-1ubuntu18.04_amd64.deb
  • Ensure that the mysql2 gem compiles using the Apt-provided library (via BUNDLE_BUILD__MYSQL2) and that, when the library loads, it knows how to find libstdc++ (via LD_PRELOAD). By default LD_PRELOAD doesn't have an existing value, but your app may be different, so do check before running this command.
$ heroku config:set \
  LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 \
  BUNDLE_BUILD__MYSQL2=--with-mysql-config
  • Update your app to use the heroku-20 stack if you haven't already:
$ heroku stack:set heroku-20
  • Deploy this Aptfile to your app, and it should work. If the gems are cached from a previous build (for example, if you had already changed the stack and deployed other changes), you will need to clear your build cache prior to deploying, to ensure the mysql2 gem is recompiled.

Back to the top

Configuration

Unless you're using Ruby (see below), you'll need to write a Sphinx configuration by hand, and then upload it to Flying Sphinx. That last step can be done easily enough using the command line tool:

$ heroku run flying-sphinx configure /path/to/sphinx.conf

If you want to run this command on your own machine instead, please note that you'll need the FLYING_SPHINX_IDENTIFIER and FLYING_SPHINX_API_KEY environment variables set with the values your Heroku application has set.

Additional Configuration Files

If you're using additional configuration files for wordforms, stopwords, or exceptions, please refer to the extended documentation for either Python or Node.js.

Configuration with Rails or other Ruby frameworks

The flying-sphinx gem can generate configuration via Thinking Sphinx - so, don't provide a file path:

$ heroku run rake ts:configure

If you're using a version of Thinking Sphinx prior to v4, you'll need to use the fs-specific rake task fs:configure.

Additional configuration files are handled by the gem automatically.

Sphinx Versions

The recommended versions are as follows:

  • Sphinx: either v3.1.1 or v2.2.11.
  • Manticore: 3.5.4

Existing customers may be using older versions of Sphinx or Manticore. These include:

  • Sphinx: 3.0.3, 2.2.9, 2.2.3, 2.1.8, 2.1.1, 2.1.0-dev, 2.0.9, 2.0.8, 2.0.6, 2.0.5, 2.0.4 and 1.10-beta
  • Manticore: 2.6.3, 2.7.5, 2.8.2

If you are using Thinking Sphinx and Ruby, you can choose which version of Sphinx you'd like to use through the version setting in your config/thinking_sphinx.yml file.

For NodeJS, you can configure the version via a custom script of your own, using code like the following:

var flyingSphinx = require('flying-sphinx');
var configuration = flyingSphinx.configuration();

// first argument can one of two options:
// * 'configure' - updates the configuration only.
// * 'rebuild'   - stops the daemon, updates the configuration, re-indexes, and
//                 starts the daemon up again.
configuration.process('configure', function(configurer) {
  // Upload Sphinx configuration
  configurer.addConfiguration('indexer { }');

  // Can be 'sphinx' or 'manticore'
  configurer.addEngine('sphinx');

  // For Sphinx, you should use 2.2.11 or newer.
  // For Manticore, you should use 2.7.5 or newer.
  // The supported versions are listed here:
  // http://support.flying-sphinx.com/kb/configuration/setting-your-sphinx-version
  configurer.addVersion('2.2.11')

  // For setting files, specify the setting, the name of the file, and
  // the contents of the file.
  configurer.addSettingFile('wordforms', 'wordforms.txt', 'file contents');
});

If you're using Thinking Sphinx v1/v2, and you've not specified a version in config/sphinx.yml, you'll see this warning (but everything will work fine), and it'll default to 2.2.3:

Sphinx cannot be found on your system. You may need to configure the
following settings in your config/sphinx.yml file:
  * bin_path
  * searchd_binary_name
  * indexer_binary_name

For more information, read the documentation:
http://pat.github.com/ts/en/advanced_config.html

The reason for this is that Heroku doesn't know what version of Sphinx you're using - but we can tell Thinking Sphinx (and Flying Sphinx), and that hides this message. Add (or edit) your config/thinking_sphinx.yml file to include the version setting for your production environment:

production:
  version: '2.2.11'

Back to the top

Processing Sphinx Indices

To tell Flying Sphinx to process your Sphinx indices, it's just a single call to the command line:

$ heroku run rake ts:index

If you're using a version of Thinking Sphinx prior to v4, you'll need to use the fs-specific rake task fs:index.

Indexing is something you'll want to do at a regular interval. Heroku's Scheduler Add-on will do the trick nicely - just use flying-sphinx index as the task.

Real-time indices

All of the standard Thinking Sphinx rake tasks will operate the same with Flying Sphinx as they do locally.

If you're using a version of Thinking Sphinx prior to v4, you'll need to use the fs-specific rake task fs:regenerate instead of ts:rebuild.

With Node or Python

Instead of rake, you can use the Flying Sphinx CLI:

$ heroku run flying-sphinx index

Back to the top

Controlling the Daemon

Starting and stopping the Sphinx daemon is done through the command line with two simple commands:

$ heroku run rake ts:start
$ heroku run rake ts:stop

You can also use the restart command to stop and then start the daemon, and the rebuild command to stop Sphinx, process the indices, and start Sphinx up again:

$ heroku run rake ts:restart
$ heroku run rake ts:rebuild

If you're using a version of Thinking Sphinx prior to v4, you'll need to use the fs-specific rake tasks instead (e.g. fs:start, fs:stop).

With Node or Python

Instead of rake, you can use the Flying Sphinx CLI:

$ heroku run flying-sphinx start
$ heroku run flying-sphinx stop
$ heroku run flying-sphinx restart
$ heroku run flying-sphinx rebuild

Back to the top

Searching

Once you've got some indexed data and have started the daemon, you can then send search queries to Sphinx. The Sphinx server and port are available through the environment variables FLYING_SPHINX_HOST and FLYING_SPHINX_PORT. Use them through whichever Sphinx client you prefer in your language.

If you're using Ruby and Thinking Sphinx, then the flying-sphinx gem manages all of that for you in the background, so just run your search calls as you normally would.

Back to the top

Delta Indexing (Ruby Only)

If real-time indices aren't a viable solution for you, then an alternative is to have delta indices covering recent changes. This can be done by using Sidekiq with the ts-sidekiq-deltas gem, Delayed Job with the ts-delayed-deltas gem, or Resque with the ts-resque-deltas gem. Configuration is covered in each of those gems, and does not require any alteration for the Flying Sphinx environment.

Don't forget to add a column called delta to your model as well — just like with the standard delta approaches.

Once you've got this deployed, then rebuild your Sphinx setup:

$ heroku run rake ts:rebuild

And from that point, the rest is taken care of, provided you have a worker managing your Sidekiq, Delayed Job or Resque jobs and processing the appropriate queue (which defaults to ts_delta).

Back to the top

Upgrading and Downgrading

You can upgrade and downgrade plans just by informing Heroku you want to use a different plan:

$ heroku addons:upgrade flying_sphinx:granite
$ heroku addons:downgrade flying_sphinx:ceramic

Flying Sphinx will migrate your data between plans and update your app accordingly - depending on how much data you have, this could take several minutes. The owner of the app will be emailed once the plan change is complete.

Back to the top