Enhancing Numerical Awareness: Our Engineering Journey
Written on
Chapter 1: Introduction to Numerical Analysis
In today’s service-driven landscape, a solid grasp of numerical analysis is crucial. Last year, we laid the groundwork for comprehensive numerical analysis, and I am excited to share our efforts in this endeavor!
One of our company’s guiding principles is to "Develop plans with a numerical focus to enhance execution capabilities." Without a solid numerical foundation, our decision-making process can become unclear.
By presenting data as concrete evidence, we can make fact-based decisions that align with our envisioned service value. Our aim is to ensure that everyone in the organization understands this approach and can implement it effectively.
In this piece, I will outline the steps our engineers took to fulfill the goal of "Crafting plans that prioritize numbers and bolster execution."
Section 1.1: Introducing Redash
To foster a culture of numerical awareness, we initially adopted Redash. This open-source BI/dashboard tool allows us to pull data from various sources, including MySQL, BigQuery, and Treasure Data, offering features like query scheduling, execution caching, and dashboard creation.
Redash's Adoption Journey
We chose Redash for its ability to handle heavy queries with ease, streamlining data extraction. Previously, our sample department store operated on a two-device model: a Master and a Slave setup, where the marketing team accessed data via Microsoft Access linked to the Slave. Initially, access was limited, but as our services expanded, the increased demand caused the Slave to fail.
Section 1.2: Data Extraction Platform Post-Redash
Following the Slave's failure, our system development team sought to minimize direct connections from Access. This is not to suggest that Access is ineffective; rather, it is a valuable tool in its own right.
With Redash in place, queries are cached, significantly speeding up data retrieval for subsequent requests. Additionally, once a query is created, it can be reused by other users, eliminating redundant query creation and enhancing overall efficiency. Consequently, Redash has gained significant popularity within our organization.
After implementing Redash, we took further action by establishing a Slave dedicated solely to Redash. The previous Slave failure underscored the need for a separate database for analysis, ensuring that service operations remained unaffected.
The introduction of an independent Slave configuration allows for active data extraction without disrupting service functions. We can now efficiently acquire analytical data related to users and sales, as well as integrate external Google Analytics data through Redash.
Benefits of Implementing Redash
Let’s summarize the key advantages that emerged from our Redash integration:
Caching Execution Results:
Queries that have been executed once are stored, allowing subsequent users to access the same results without re-executing the query against the database. This significantly reduces the load on our database by preventing duplicate queries.
Query Sharing:
Once a query is created, it can be saved and shared within Redash. This feature encourages collaboration, as team members can reuse existing queries rather than duplicating efforts. Redash also supports the embedding of variables in queries, allowing for dynamic input forms that streamline query creation.
Dedicated Redash Database:
With a dedicated database for Redash, queries can be run without impacting service operations, bringing us closer to becoming a data-driven organization!
Chapter 2: New Challenges Post-Redash Implementation
Despite the many advantages of Redash, we faced new challenges, including:
Dependence on Redash:
As Redash became integral to marketing and operational tasks, any downtime leads to panic within departments reliant on its functionality. With the whole organization depending on Redash, its stable operation has become a priority for the System Development Department.
Increased Query Creation Workload:
Although Redash simplifies query reuse, engineers still face increased demands to create new queries for minor variations. The ratio of development/operation tasks to query creation shifted from 8:2 to 6:4, limiting engineers' ability to focus on their core responsibilities.
Implementing SQL Training
To counteract this trend, we initiated SQL training sessions aimed at non-engineers to alleviate the burden of query creation on our engineering team.
#### Overview of the SQL Course
This hands-on course covers everything from basic data retrieval using Select statements to crafting queries with Redash.
Outcomes of the SQL Course
Feedback from participants has been positive, and it is gratifying to see many attendees successfully creating queries using Redash. This trend gives hope that engineers can reclaim some of their time.
Installing Real-Time Data Displays
Recognizing the need for greater awareness of numerical data, the System Development Department installed displays to showcase real-time data.
#### Rationale Behind the Display Installation
While Redash enabled numerical monitoring, engagement was limited to a few individuals. To foster a culture of data awareness, we believed that visible displays of current service metrics would encourage more informed planning.
Impact of the Display Installation
The displays showcase the real-time Redash dashboard and Google Analytics data. This initiative has sparked discussions about performance, leading to insights such as "Why are my sales spiking on weekdays?" and "What strategies can drive growth?" I am optimistic that this will lead to more effective decision-making in the future!
Conclusion
What are your thoughts on our journey? We shared how our engineers have become more conscious of numerical data, a practice that I hope benefits our readers.
We are a company that prioritizes numerical insight in our planning processes. Our engineers play a vital role in establishing a platform that empowers the right individuals to access the data they need, whenever they need it. As long as we continue to operate, the necessity for data analysis and extraction will endure.
In the first video, mathematician Joe Rogan explores the complexities of measuring consciousness. This thought-provoking discussion delves into how our understanding of numbers and consciousness intertwine.
The second video features Sadhguru addressing the phenomenon of seeing repeating numbers. He discusses whether these occurrences hold more significance than mere coincidence, prompting deeper reflection on our perception of patterns in data.