Join us in building a fintech company that provides fast and easy access to credit for small and medium sized businesses — like a bank, but without the white collars. You’ll work on software wiring million of euros, every day, to our customers.
We’re looking for both junior and experienced software developers.
A crucial job of the Data Ops team within Floryn is to create and maintain clear and coherent reporting. We do that with a Business Intelligence (BI) tool. Given the large number of dashboards and visualizations in a BI tool, you might think migrating from one tool to another would be an almost impossible task, but here’s how we managed to migrate from Looker to Metabase with time to spare.
The first challenge we ran into was logic. In the realm of analytics, logic is the set of business rules used to transform and manipulate data to represent something useful. Using logic, you can model data. For example, you could transform (model) raw funnel data into a structured dataset based on your company’s definition of conversion (logic).
Most BI tools offer their users the ability to model their analytics within the tool itself. Although this is the easiest option, it can quickly evolve into a problem. Firstly, modeling your analytics in a BI tool creates a dependency on that specific BI tool. This means that when you do decide to move away from that BI tool, you will have to replicate that logic elsewhere. Additionally, transforming data within your BI tool will impact performance.
To solve these problems, we decided to implement dbt a couple of years ago. dbt is a tool that enables data analysts and analytics engineers to transform data in a cloud analytics warehouse. Essentially, this tool allows you to model your data outside of your BI tool, which increases performance and reduces dependency.
Unfortunately, however, not all logic was modeled in dbt. This meant that the highest priority during the migration was to make sure that dbt and Looker had the same output data. For roughly a month, we followed the following process:
Identify which logic was only modeled in Looker and not in dbt yet. Replicate the logic in dbt. Validate that Looker and dbt had the same output.
Once all logic was replicated, we moved on with the next part of the migration.
The next challenge we faced was determining which dashboards to recreate. Fortunately, Looker provided us with the tools to identify which dashboards and visualizations are being actively queried. This was a powerful tool for us, as we could see which dashboards were being used and which weren’t.
After identifying all the dashboards which needed to be migrated, we were a bit overwhelmed by the total number of dashboards which needed to be recreated in Metabase. To tackle this problem, we created a timeline. This timeline specified when each team within Floryn (e.g. sales, marketing) would have their dashboards migrated, and therefore acted as a guideline for us. Additionally, it allowed us to communicate clearly with stakeholders, and avoid any unnecessary chaos as to when dashboards would be migrated.
A big risk associated with this migration was that Metabase wouldn’t be used by our colleagues as long as our old BI tool was still up and running – why would you switch to a tool you don’t know if the tool you do know is still functioning and working well? We wanted to avoid switching everyone over in the final week of our migration, as it would likely lead to a flood of questions, implementation issues and perhaps external reporting breaking.
To combat this problem, we also used our timeline to structurally disable our colleague’s access to Looker, the old BI tool. For this we used a two step approach. Firstly, once a dashboard was recreated in Metabase we would delete it in Looker. If a user then wanted to use the dashboard, they had to do so in Metabase. This left us with the problem that users could potentially still create new dashboards in Looker.
So, once all the dashboards belonging to a team were migrated, we disabled that team’s access to Looker. This meant that users could no longer use Looker, and were now fully dependent on Metabase for all their analytics. If this led to no problems in their day-to-day operations, the team had successfully been migrated from Metabase to Looker!
We repeated this process for all of the teams in Floryn, and successfully migrated all of our dashboards and visualizations into Metabase with a week to spare.
To conclude, migrating to a new BI tool proved to be quite the challenge. By making use of various tools, a timeline, teamwork and some common sense we managed to transform an almost impossible task into a routine job.
Floryn is a fast growing Dutch fintech, we provide loans to companies with the best customer experience and service, completely online. We use our own bespoke credit models built on banking data, supported by AI & Machine Learning.