Scripting

FlexSearch Scripts are snippets of code that execute a certain logic at a specific moment in the FlexSearch event pipeline. Scripts are written in F# programming language. There are two separate pipelines where a script can be added.

  • Pre-index scripts - Just before a document is added or modified in an index. This script lets you manipulate the data that is being indexed.

  • Pre-search scripts - just before a search request is sent to Lucene. This script helps you control the data that is being sent for searching.

Pre-index script

Pre-index scripts are F# methods that execute a piece of code just before indexing occurs. They are perfect when you want to populate or update some fields in a document and persist that change. Compared to pre-search scripts, in pre-index scripts the new values of the document fields will be stored in the index and visible each time you search for it.

Indexing request receivedPlatformDocument Version control checkUserModePreIndex script executionPlatformDocument ValidationOther Platform logic exectionReport operation status back to the user

Pre-index scripts are declared in the same file as the pre-search scripts are, namely scripts.fsx in the index configuration folder.

Just like any other script, pre-index scripts are loaded when FlexSearch server starts or when an index is reloaded.

Usage

Pre-index scripts are used when you want to modify the data that will end up in the index during indexing time (when adding or updating documents).

The name of the F# method that holds the pre-index script logic is preIndex.

The signature of the method is:

val preIndex : Document -> unit

e.g.
let preIndex (document : Document) = ()

This translates to a method that takes a Document as a parameter and returns nothing (void).

The preIndex method needs to be placed in the script.fsx file located in the index configuration folder. For example, if you have an index named contact, the script.fsx file would need to be in:

/conf/indices/contact/script.fsx

The way the pre-index scripts are designed to work is:

  • A Document is received for indexing. Its fields are either empty or prepopulated with some data.

  • The preIndex function runs against the received Document. This function will either modify, populate or empty some fields in the document.

  • The modified Document is then passed to Lucene to index, store the data and make it available for searching.

Example

Let’s then imagine we want to populate the gender field from the employee index according to the title field - if it’s Mr., then it’s Male, otherwise Female. Here is part of the definition of the employee index:

{
  "indexName": "employee",
  "fields": [
    {
      "allowSort": false,
      "fieldName": "gender",
      "fieldType": "Text",
      "indexAnalyzer": "standard",
      "searchAnalyzer": "standard",
      "similarity": "TFIDF"
    },{
      "allowSort": false,
      "fieldName": "title",
      "fieldType": "Text",
      "indexAnalyzer": "standard",
      "searchAnalyzer": "standard",
      "similarity": "TFIDF"
    },
    ...
  ],
  ...  
}

We would then write the following code in the scripts.fsx file.

module Script

open FlexSearch.Api.Model
open Helpers
open System

let preIndex (document : Document) = 
    // Get the title field value from the document and make it lowercase
    match document.Get("title").ToLower() with
    // If it's `Mr.` then set the gender to Male
    | "mr." -> document.Set("gender", "Male")   
    // If it's empty then don't set the gender
    | "" -> ()
    // Anything else is Female                                  
    | _ -> document.Set("gender", "Female")      

The new piece of functionality will be loaded the next time FlexSearch is restarted or when the employee index is reloaded (closed then opened back).

Search execution pipeline

Let’s examine the search execution pipeline from scripting perspective. There are various places where a user can insert custom logic to modify the search behaviour. The first logical places Pre-search script. This can be used to perform operations like:

  • Modify the input search data. This is helpful in cases where a user wants to enforce certain results. For example when a user searches for a specific input you may only want to return a specific result. In these cases you can investigate the incoming data and set the right parameters so that the platform always returns the right result. This stage can also be used to set default values for fields in case no data is provided. Another example would be to normalize an incoming telephone number to a certain format before submitting the search…

  • Modify the search query parameters.

  • Update the search profile name based on some dynamic condition.

Search request receivedUserModePreSearch script executionPlatformPlatform logic exectionApply search time filters like distinctBy and cutOffUserModePost search script executionResult compilationResult is returned back to the caller

Pre-search script

A FlexSearch pre-search script is a snippet of F# code that gets executed before the search is run.

The source code of the script is loaded from the script.fsx file from the index configuration folder. A PreSearch script is a method within the script.fsx file that has a name starting with preSearch (e.g. preSearchTest).

Search scripts are loaded automatically on startup or when an index is opened. To add a new script you need to either restart the system or reload the index (close, then open it back again).

Usage

Pre-search scripts are stored in a script.fsx file located in the index configuration folder. For example, if you have an index named contact, you would write the script.fsx file at:

/conf/indices/contact/script.fsx

Pre-search script methods have the following naming convention:

preSearch<script_name>

e.g. preSearchTest

And the following signature:

val preSearch<script_name> : SearchQuery -> unit

e.g. 
let preSearchTest (query: SearchQuery) = ()

The above signature means that we should have a function that takes a SearchQuery and returns nothing (void).

Having a FlexSearch.Api.Model.SearchQuery as a parameter gives you access to get or even set any of its properties. For example you can modify the columns to retrieve, or execute some code conditionally based on the query name, or modify the query string, etc.

Probably the most powerful feature is the access to the Variables (the @variable_name pieces from the query string) property from the Search Query. This means you can modify the values that get passed into the query string.

Example

Let’s say you want to bring all the employees that have been in the company for more than 10 years. We assume that we have an employee index with the following fields:

  • yearJoined
  • name

You can initially write the following query on the employee index:

gt(yearJoined, '2006')

And it will work just fine. But next year you’ll realize you have to change the query from 2006 to 2007. So here comes the variable to the rescue:

gt(yearJoined, @tenYearsAgo)

In this case you would add an entry in the SearchQuery.Variables dictionary for:

searchQuery.Variables.Add("tenyearsago", DateTime.Now.AddYears(-10).Year)

And then you would submit this search query to FlexSearch. This would work as well, but maybe it would be easier if you would just pass the current year and then substract 10 from it before submitting the search. You could reuse this piece of functionality for other queries as well. You can do this using a pre-search script!

You modify the search query string like so:

gt(yearJoined, @year)

And you create a new file in the Conf/Indices/employee folder called script.fsx in which you add the following code:

module Script

open FlexSearch.Api.Model
open Helpers
open System

let preSearchTenYearsAgo (query : SearchQuery) =
    // Get the variable called "year". Take care of upper vs lower case.
    let kvp = query.Variables
              |> Seq.find (fun kv -> kv.Key.ToLower() = "year")

    // Modify / populate its value
    kvp.Value <- DateTime.Now.AddYears(-10).Year

Lastly, before submitting the SearchQuery you just need to specify that you want to use the newly created pre-search query:

searchQuery.PreSearchScript = "TenYearsAgo";

Post-search script

Post-search script is still in development stage and we will add relevant documentation once the feature is finalized.