Your business intelligence domain runs on data. When your data is spread across different platforms and warehouses, it impedes operational efficiency, adds to your total cost of ownership (TCO), and limits your flexibility.
Migrate to a single DataOps-driven data platform to reduce your amount of technology vendors and increase the efficiency of business intelligence (BI) solutions. Such BI solutions include predictive analytics, data visualization, and self-service data delivery and acquisition.
Unify disparate data management
Takeoff Technologies is a provider of eGrocery solutions — including hardware, cloud orchestration, industrial AI, data-driven analytics, partnership, and support — to streamline retail automation.
Takeoff primarily functions as a micro fulfillment center to lower the last-mile and assembly costs for grocers. It provides end-to-end eGrocery solutions to its customers, including user interface, assortment and content management, fulfillment, spoke routing, customer pickup and delivery, and replenishment.
To unify its landscape infrastructure and reduce the number of technology vendors involved, Takeoff needed to migrate from a Snowflake-based data warehouse to Google Cloud.
Build a DataOps-driven platform
As part of Takeoff’s migration, the team needed to build a brand-new DataOps-driven data platform on Google Cloud based on site reliability engineering (SRE) best practices. Simultaneously, preserving the same functional requirements of the Snowflake data warehouse.
Successful completion required the team to:
- Migrate all the historical data from Snowflake into Google BigQuery
- Design and implement a new data lakehouse solution with data ingestion and processing logic using the latest Google Cloud best practices
- Enable real-time analytics for data consumers
- Create clear SLAs for data consumers and implement a set of monitoring services to provide SLIs
- Establish data governance for company-wide data assets
- Share expertise within the data lake, BI, and Google-driven infrastructure development
Customize a new Google Cloud environment
SoftServe incorporated a group of team members into Takeoff’s data lake and BI teams, where they worked together in alignment with Takeoff's processes governance. Starting with a discovery phase, SoftServe facilitated an end-to-end migration flow from Snowflake to Google Cloud.
The team implemented data ingestion and processing logic into a new Google Cloud environment using the latest best practices. Monitoring services and metrics were put in place to control the new platform’s stability, guaranteeing service level objectives (SLOs).
The embedded team’s approach provided transparency and continuous knowledge sharing as the SoftServe team was included in operational communications. This deepened the team’s understanding of Takeoff and allowed them to become consultants on cost saving and other solutions. Additionally, as the Takeoff team underwent staffing changes during the project, SoftServe’s team mentored and trained new members of the engineering team to own the project after its completion. Training involved creating precise documentation and providing pair programming and demo sessions for new engineers so they could maintain the cloud solution after migration.
Optimize processes and reduce TCO
The project decreased Takeoff’s TCO by switching to the latest Google Cloud tools and reducing the number of tech vendors. The deployment process is fully automated in GitHub Workflows to optimize processes.
All historical data were successfully migrated from Snowflake into Google BigQuery. To further streamline workflows, the data lakehouse is controlled by an infrastructure-as-code approach using Terraform, and an on-call process based on the five-level severities Opsgenie alerts was established.
David Chang, Director of Engineering, Big Data Domain, describes the team’s success after giving the SoftServe team a 10/10 NPS score:
“SoftServe’s project leaders did an amazing job understanding our needs and staffing a highly qualified team. They managed the project incredibly well from beginning to end, always making sure to surface questions and concerns in a timely manner to de-risk unexpected issues in our migration. The team designed an elegant data architecture solution that both met our business needs and allowed for maximum scale with minimal effort. The engineers all integrated incredibly well into our data lake and business intelligence teams, executing reliably against work and doing so with long-term maintainability and industry best practices in mind. This truly was an A+ team whom I could count on to achieve my desired business goals. Given sufficient funding/budget, I would not hesitate to rehire this team for future projects.”
Let’s talk about how SoftServe can customize your data management to reduce your TCO and increase operational efficiency.