Version 1.12.0 of refinery offers many exciting features such as extended filtering, new embedding providers as well as handy attribute types. With many of these new features, we strive to make refinery even more suitable for Retrieval Augmented Generation use cases. Let's dive deeper into the new features!

Qdrant filtering options

We now offer filtering options for embeddings, which allows you to set conditions for retrieving or searching for records that have embeddings. These additional filters are especially useful when it is not possible to express all the features for a search in the actual embeddings.

With these filters, it's easier to access the correct information when you retrieve documents via similarity search. An example: Imagine you have a collection of documents about car insurance claims, which have the categories "car and car", "car and bike" and so on. If you are interested in only retrieving data about incidents with cars and bikes, you can directly filter them out using the filtering options.

You can set these filters during the embedding creation and use them in the data browser and via API calls in gates. Of course it is also possible to edit these filters afterwards.

Currently, the filters can only be set for all attributes containing integers, floats and categories. You can read more about the filter option on Qdrants Documentation.

Azure OpenAI Embeddings

The new update extends the list of possible providers for creating embeddings. You now have the ability to create embeddings using the Azure OpenAI embedding service. Functionally, this service is the same as the standard OpenAI embeddings, as the service on Azure also uses OpenAI's text-embedding-ada-002 embedding model.

The parameters are set during creation of the Azures OpenAI Studio service or when deploying the embedding service.

Using this service via Microsoft Azure has, however, two distinct differences. The service on Azure:

  • Can be hosted in Europe, making the usage more attractive from a GDPR standpoint, as the data is not required to leave the EU for processing.
  • You have a bit more control over factors such as the maximum throughput, limits when setting up the model.

For more information regarding limits and service quotas, please refer to the official documentation.

So in total, there are now the following embedding methods available:

  • HuggingFace
  • Cohere Embeddings
  • OpenAI
  • Microsoft Azure OpenAI
  • Python functions (for bag-of-words or TF-IDF)

New attribute type: Embedding lists

Embedding lists are a new type of attribute in refinery which we have included as part of our focus on use-cases using Retrieval Augmented Generation (RAG). The main idea behind embedding lists is fairly simple: You take a text and chunk it up into multiple, smaller pieces. The reason for this is that with smaller chunks, it is easier to find relevant texts via similarity search, which is crucial for RAG. All the items in the embedding list still point to the original text, meaning that you can still provide LLMs with the full context of a record while also improving the accuracy for similarity search tasks.

A created embedding list will look like this in the data browser:

Keep in mind that by default, the embedding lists are hidden in the application to avoid showing you data redundantly. To change this, you can always set the embedding list attribute to "Do not hide" on the settings page.

By default, the embedding list attribute will provide you with a small code snippet that will split attributes based on sentences using SpaCy:

def sentence_split(record):
    # e.g. use SpaCy sentences to create a list
    return [r.text for r in record["str_attribute"].sents]

Of course, the programmatic approach to this attribute offers you the ability to split and chunk texts as you please. For example, you may split texts using new line characters "\n" or based on RegEx.

You can now edit records in refinery

It is now possible to edit individual records directly inside of refinery. To do this, you can head over to the data browser and click on the editing icon of any records. This will open a new editing view. After you have edited one or more records, click on "Synchronize with DB" to make your changes persistent. After the changes have been made, refinery will automatically re-embed the records on the managed version. Editing records requires the project to be gates ready, which can be checked and set on the settings page. The new editing page uses the available records from your data browser session, which means that you can filter your records prior to entering the editing mode.

Be aware that with old projects in the open-source version (v1.11.0 and older), you'll need to re-calculate all of the embeddings. When embeddings are created, some information is stored in pickle files. With older projects on the open-source version, these files haven't been created yet. New projects in the open-source version do not experience this behavior and will function like on the managed application without the need to re-calculate all of the embeddings after editing records.

Additionally, when updating the newest open-source version, you should remove the qdrant-storage folder after updating, as this folder is not being used anymore. Not removing the folder won't cause any issues, you would just have an old and a new qdrant-storage folder.

Minor changes and bug fixes

  • Fixed error that line breaks were not visible when filtering in the data browser Issue #267
  • Fixed issue with faulty height of heuristic cards Issue #270
  • It is now possible to import lookup lists in the attribute calculation Issue #271
  • Fixed error in the knowledge module where multiple line breaks caused issues Issue #268
  • The embedding modals should now behave as expected Issue #263
  • Resolved issue where name of snapshot could be set multiple times Issue #261