Unifying a Information Stack and Leveraging Self-Serve Analytics with Atlan
The Energetic Metadata Pioneers sequence options Atlan clients who’ve accomplished a radical analysis of the Energetic Metadata Administration market. Paying ahead what you’ve realized to the following knowledge chief is the true spirit of the Atlan neighborhood! So that they’re right here to share their hard-earned perspective on an evolving market, what makes up their fashionable knowledge stack, modern use instances for metadata, and extra.
On this installment of the sequence, we meet Daniel Ferguson, Information and Analytics Director at PHMG, an audio branding firm that helps over 36,000 purchasers throughout 56 international locations sound their greatest. Daniel shares how PHMG remodeled their knowledge stack from fragmented to unified, and the way Atlan has been a vital piece in by monitoring lineage, managing stories, and easing staff onboarding.
This interview has been edited for brevity and readability.
Might you inform us a bit about your self, your background, and what drew you to Information & Analytics?
I was a DJ after which labored as a sound engineer constructing recording studios. Throughout my time as a sound engineer, I discovered myself within the technical and analytical facet of issues. After beginning a household, I wished a change. After promoting my recording studio, my mom, who managed a council workplace, provided me a job and I began out within the name heart unit, dealing with calls. It shortly grew to become obvious that I may do extra than simply calls, so I moved to the database staff.
I began learning for a level in Economics and Mathematical Science at The Open College whereas working on the council. Utilizing the abilities learnt on my diploma I began to construct logistic regression fashions to focus on contacts within the name heart I had beforehand labored in. I proposed that, with only one particular person, I may obtain the identical outcomes as your entire staff. My work generated 300% extra outcomes than the staff’s mixed efforts by optimizing knowledge assortment, addressing lacking data, and cherry choosing the perfect contacts. After that I used to be hooked on the facility of information & analytics.
I then constructed an organization offering knowledge providers to different Native Authorities. Close to the tip of my diploma, a consultancy in Scotland, Aquila Insights, provided me a place. They labored with purchasers like Sony, Workplace Depot, and RBS, which gave me early publicity to the info occupation. From there, I superior within the discipline and ultimately joined PHMG. My journey into knowledge was considerably unintentional, however it introduced me to the place I’m at present.
Would you thoughts describing PHMG?
We specialise in audio branding. Consider logos like Netflix or Disney Plus, via sound alone, these manufacturers are immediately recognizable as business leaders in leisure and streaming even when their visible logos usually are not in sight.
We additionally transcend conventional audio branding by creating customized music tailor-made to every group. We’ll take Atlan for instance: What’s Atlan about? What do you symbolize? What’s the kind of rhythm that it needs to convey?
This connection between music and id is what attracted me to the corporate. We’ve been extremely profitable, working in 56 international locations with 36,000 purchasers.
Might you describe your knowledge stack, and the way it got here collectively?
After I acquired right here, we have been utilizing SQL servers with Excel spreadsheets. There have been restricted to no interactive stories, and each knowledge request needed to be raised to the info staff.
There was a must modernize the knowledge flowing into the corporate and implement the suitable expertise to attain this effectively and reliably. I centered on discovering expertise options that might streamline operations and cut back the necessity for added engineers.
I used to be actually cautious with expertise choice, avoiding options for the sake of it, and never constructing from scratch. Whereas Azure Material affords a complete answer, for instance, it’s nonetheless new and that comes with further dangers, however is one thing I’m retaining my eye on. It’s essential to decide on the perfect instruments for the job and guarantee they work effectively collectively. Investing in a seamless course of with these instruments lets you begin sturdy and display worth shortly, with room to evolve as you scale.
In my board proposal, I highlighted two important instruments: Atlan and ThoughtSpot. I defined that whereas we may handle with out them, they’d make a big distinction. I wished governance to turn out to be embedded in our processes, and that as a substitute of assigning knowledge stewards with out clear route, we supplied actionable stories and comprehensible knowledge. With correctly organized knowledge, governance turns into simple, and Atlan streamlines this course of.
I chosen Snowflake for its robustness and cheap pricing, and Fivetran for its dependable pipeline efficiency, which successfully handles our knowledge integration wants.
I carried out PowerBI for government stories, and ThoughtSpot for our self-serve knowledge wants. I’m an enormous fan of ThoughtSpot, as a result of it permits customers to regulate their very own stories, decreasing the necessity for fixed modifications from the info staff.
For orchestration, I exploit Airflow to handle pipelines, and DBT with GitLab for our code repository and CI/CD processes.
Why was Atlan match? Did something stand out throughout your analysis course of?
In my earlier group, I attempted utilizing open-source with DataHub, however its upkeep and growth required important funding. Atlan stood out as a result of it’s plug-and-play, mechanically constructing out miners that reveal beforehand unknown insights. It identifies and explains scripts we weren’t conscious of, saving time and decreasing technical debt from having to manually assessment in depth code.
Atlan lets us observe and monitor what we’ve constructed, together with knowledge lineage and belongings. It’s invaluable for reviewing stories with no need to ask for code particulars—simply navigate via Atlan to see the report’s historical past. New staff members may also perceive report building via Atlan.
For me, Atlan was a key piece of the puzzle.
I researched Collibra, Alation, and Atlan extensively, and Atlan was the clear alternative. It felt designed for medium-sized enterprises and required minimal engineering effort. Given our state of affairs, it was essential to combine it from the beginning, relatively than as an afterthought. This allowed us to be taught and develop Atlan alongside our current programs, relatively than making an attempt to power it into our pre-built setup.
I all the time make it some extent to satisfy with management groups at occasions to gauge their perspective and willpower, and I don’t know of another gamers which might be doing it in addition to Atlan. I used to be genuinely impressed by Atlan’s management staff — not solely their ardour for the product but additionally their dedication to addressing my challenges and enhancing our state of affairs.
How are you planning to harness Atlan to reinforce your knowledge stack? What thrilling use instances and targets do you take note of?
We’ve invested in an information vault mannequin for our knowledge warehouse, which feeds into an operational knowledge retailer, what I name the info mart. All our stories and metrics are constructed from this knowledge mart. In Atlan, we outline assemble every thing, so as soon as a metric is outlined, we will write the SQL to extract it from the mart.
We then create curated tables for consumer providers and gross sales organizations, enabling them to self-serve by way of ThoughtSpot. For detailed insights into the development and rationale of those metrics, we retailer that data in Atlan, which turns into our catalog.
As new individuals come on board, I be certain that there’s no want for a handover. By default, we doc our processes as we go and construct programs that depart clear breadcrumbs for others to comply with. Atlan performs a vital function on this. We direct new staff members to Atlan to assist them perceive how every thing is constructed and what it’s constructed from. Atlan doesn’t simply spill out the code, it highlights the important thing objects, their utilization, and their significance.
One other main venture entails making a complete glossary inside Atlan, serving as our single supply of reality. This surroundings permits enterprise customers to entry all company metrics and think about stories from Salesforce, PowerBI, and ThoughtSpot, all linked round key KPIs.
We’re additionally presently refining our knowledge lineage and mannequin descriptions. As we create new knowledge fashions, we replace descriptions incrementally relatively than in bulk. This ongoing effort helps be certain that our knowledge fashions are well-documented and simply comprehensible.
Do you’ve any recommendation to share along with your friends who’re beginning out in managing and organizing their knowledge belongings successfully?
Companies all the time discuss being knowledge pushed, however they don’t discuss in regards to the belongings that really drive the info. We wish data to circulation in our group, however data can’t circulation if it’s not organized persistently. And for me, instruments like Atlan are making it considerably simpler for us to prepare and talk what knowledge issues.
Don’t get me unsuitable, Atlan isn’t a silver bullet. It received’t repair poor group inside your knowledge warehouse. Nonetheless, it does present a centralized place to outline and assess your processes, serving to you establish which of them are efficient and which of them want enchancment.
Atlan helped us decide the place to start out by figuring out our most important tables and specializing in what was vital. For example, we discovered one desk vital for every thing we constructed, permitting us to prioritize it. We then assessed our stories and found that some we thought have been vital have been related solely to particular stories, not the broader context.
As we get delicate knowledge, we will additionally instantly flag it. If we get audited, we will merely pull up Atlan and say, “Hey, that is what we have now. That is how we handle our knowledge. That is what our knowledge belongings are.” So, for these dedicated to being data-driven, they should take care of their knowledge belongings and perceive what their knowledge belongings are.
Picture by Adi Goldstein on Unsplash