Queuing system & bricks integrator

This release focuses on reducing CPU spikes by introducing a queue for your tasks. Furthermore, our tech stack is changing (we are moving from angular to react), and started with an updated admin dashboard to test our current approach without changing anything on your end.

One more thing: Some of you might have already noticed it but around refinery version 1.9.1 we introduced the new and improved bricks platform. Since refinery features a dynamic integrator we needed to update that too.

Task Queue

Did you ever try to copy all your files from an internal drive to an external one? It takes some time, right? Now did you ever try to do this folder specific? Unsurprisingly this takes more time. The reason (or better one of the reasons) is multitasking. Trying to do everything simultaneously will run into issues - or at least result in longer waiting times.

As you probably know refinery has some pretty computational heavy tasks like embedding creation or zero shot heuristics. To optimize the overall experience and prevent system failure we introduced a global queueing system. Without going too much into detail it’s a multi FIFO queue that prioritizes short-running tasks without completely ignoring the longer-running ones.

Currently included in the queueing system are heuristic and attribute calculation, embedding creation, and tokenization.

Bricks Integrator for 1.1.0

Though already part of refinery 1.9.1 we haven’t mentioned the integrator update in a changelog yet. Long story short: It mirrors the new features introduced for bricks 1.1.0.

In practice, this means that the integration part of the integrator hasn’t changed much. It’s still the same streamlined experience where the code is prepared for your specific project without the hassle of changing the python code by hand. Sure, a few more fields in the overview, maybe a new slider for a variable in the integrator, and a new approach to variable detection are implemented.

Now where the changes really come into play is the search page. A new GDPR badge to indicate compliance at a glance. Groups to help you find what you are looking for and the search bar filters them even dynamically.

Minor Changes

  • Rework of admin dashboard in react for a cleaner tech stack
  • Updated the welcome screen to reduce irritation regarding doc/application links
  • Deleted users are now listed as "unknown user" in the project overview
  • Modal for Embedding creation now displays a message if no TEXT attributes exist in the project
  • Added two libraries to the execution environments holidays & sumy
  • Reworked the logic for automatically creating running_id
  • Temporary removed data tracking from the application
  • Fixed a bug for label imports with numeric values
  • Fixed a bug for tokenization tasks on deleted projects
  • Fixed a bug for record export for a slice