With regards to telemetry information – which means the logs, metrics, traces, and different info engineers use to observe purposes, handle efficiency, and troubleshoot outages– extra is often higher. However there is a large caveat: For those who fail to handle telemetry information successfully, the info can shortly create extra issues than it solves, resulting in issues like larger storage prices, issue discovering the fitting info when responding to failures, and the danger of exposing delicate info to unauthorized entry.
That is why telemetry pipelines have turn into a vital ingredient in fashionable software observability and efficiency administration methods and a important useful resource when deploying instruments like Safety Data and Occasion Administration (SIEM) platforms. Telemetry pipelines enable companies to gather, course of, route, and retailer telemetry information effectively and at scale.
That, at the least, is a high-level overview of what telemetry pipelines do and their significance from each a technical and a enterprise perspective. For a deeper dive, preserve studying as we draw on our collective expertise in designing, implementing, and managing telemetry pipelines to elucidate why they’re so invaluable and what to search for when constructing a telemetry pipeline tailor-made to your group’s wants.
What are telemetry pipelines, and why do they matter?
Telemetry pipelines are a sort of resolution that collects, processes, and routes telemetry information – which implies logs, metrics, traces, and another sort of info that gives visibility into software efficiency.
Telemetry information has been vital for many years, given the central position it performs in permitting organizations to observe software efficiency, detect issues, and troubleshoot them in ways in which decrease the affect on customers. However till comparatively lately, few organizations had deliberate methods in place for managing that information. As a substitute, they relied on advert hoc approaches for gathering information from the locations the place it originates and shifting it to the instruments they relied on to investigate it.
That strategy labored properly sufficient most often when the quantity of logs, metrics, and traces {that a} enterprise managed was comparatively low. However in as we speak’s world of distributed software program architectures, the quantity of telemetry information that the standard group should cope with has exploded. As a substitute of getting to gather only one set of logs information and metrics for every software, as you’ll have finished within the age of monolithic apps, it is common as we speak for a single software to encompass a dozen or extra microservices, every of which generates its personal logs and metrics. Couple that with the truth that as we speak’s apps usually run on distributed infrastructure that will encompass lots of or hundreds of particular person servers, and it is simple to see why there may be a lot extra telemetry information to handle as we speak.
To deal with that quantity effectively, most organizations want greater than an advert hoc resolution for gathering, processing, and routing information. They require a purpose-built resolution that systematically pulls information from the varied sources the place it originates, normalizes and transforms it as essential, and delivers it to the locations the place it is analyzed or saved. They want, in different phrases, a telemetry pipeline.
The advantages of telemetry pipelines
By bringing consistency and order to telemetry information administration, telemetry pipelines produce a variety of enterprise advantages. A very powerful embrace:
-
Value discount: Pipelines can assist scale back the price of processing and storing telemetry information by means of capabilities like deduplication (which removes redundant information, thereby decreasing storage volumes and prices) and information minimization (which reduces the quantity of knowledge ingested into analytics instruments, leading to decrease prices to function instruments which are priced primarily based on whole information ingested).
-
Knowledge privateness and safety: Telemetry information might comprise delicate info, akin to personally identifiable info (PII) saved in log information. By offering capabilities like encryption for information in movement, telemetry pipelines assist defend delicate information and meet compliance obligations.
-
Enhanced software efficiency: The quicker and extra reliably you may transfer telemetry information from its fatherland to the place the place you analyze it, the higher you may be at detecting and fixing software program efficiency points earlier than they result in failure.
-
Centralized management and visibility: A telemetry pipeline provides you a consolidated view of your entire telemetry information. This implies you may all the time know which information sources can be found, the way you’re utilizing them, and what you may do to make your telemetry course of much more environment friendly.
-
Operational flexibility: When you construct a telemetry pipeline, you may simply swap information sources and locations out and in as required. This implies you may join and deploy apps or analytics instruments at will with out having to implement customized telemetry administration processes for each.
-
Freedom from lock-in: Alongside related traces, telemetry pipelines assist be sure that companies can simply migrate to completely different analytics or software efficiency administration tooling with out being tied into a selected vendor’s stack because of the problem of getting to replace complicated telemetry information administration processes.
Telemetry pipeline options: Fundamental vs. superior
To ship the advantages we simply mentioned, each telemetry pipeline resolution worthy of the identify ought to present a core set of options, together with:
-
Assortment, which means the power to tug information from the disparate locations the place it originates.
-
Processing, which transforms information in varied methods in order that it’s ideally fitted to use by analytics or software efficiency administration instruments.
-
Routing, or the supply of processed information to the varied instruments that a corporation makes use of to investigate or interpret it. Routing may also ship information to long-term storage repositories if the group must retain the info.
These, nonetheless, are the naked minimal options that telemetry pipelines should help. To realize optimum pipeline effectivity and suppleness, organizations ought to search out a number of key extra capabilities.
The OpenTelemetry Mission
OpenTelemetry (or OTel for brief) is an open framework (ruled by the CNCF) that gives a standardized strategy to gathering, processing, and transmitting telemetry information.
OTel has turn into a just about common customary, with its instruments experiencing greater than 30 million downloads every month, which signifies that so long as your pipeline helps OTel, you can use the pipeline to attach virtually any information supply to any information analytics or administration device.
The Worth of an Open Framework
Supporting OTel ensures {that a} telemetry pipeline will work with any OTel-compatible information supply or device. Nonetheless, to maximise the pliability of your pipeline and decrease the danger of vendor lock-in, you may take openness a step additional by constructing a pipeline that features minimal proprietary elements.
When your pipeline software program is open – which means it is primarily based on transparently standardized elements – you do not have to fret about changing into locked into your pipeline software program itself or beholden to a selected vendor to help the device you rely upon to work with telemetry information.
Processing previews
Processing previews will let you predict how any information processing routines that you’ve got configured inside a pipeline will change your information. That is vital since you do not wish to uncover after the truth that you processed information in a means that made it unusable or that launched errors or formatting issues. With previews, you may take a extra proactive strategy to information administration and get forward of knowledge points in actual time.
Knowledge summaries
Along with viewing information because it flows by means of particular person pipelines, the power to summarize all information inside your pipeline helps you monitor how a lot information you are processing and what you are doing with it. In flip, these insights will let you determine long-term tendencies involving your telemetry information. They’ll additionally assist to trace telemetry prices and discover alternatives to streamline information workflows.
Learn how to construct a telemetry pipeline
As soon as you’ve got determined what you want your telemetry pipeline to do and located software program that does it, it’s essential to implement the pipeline itself – which is usually a difficult activity, given the complexity of recent pipelines and telemetry information.
We can’t stroll by means of each step of the implementation course of right here for the reason that specifics fluctuate relying on which telemetry pipeline software program you are utilizing. What we want to point out, nonetheless, are implementation and operational challenges that groups generally overlook, akin to:
-
Agent migration: You could have already got software program monitoring brokers in place which are gathering information from purposes or providers. Reasonably than reconfiguring these brokers, you may ideally be capable of migrate them into your pipeline, which saves time and reduces the hassle required to implement a brand new pipeline.
-
Pipeline observability: You will want a technique to monitor and observe your pipeline itself to detect potential efficiency points or errors.
-
Pipeline scalability: It is a protected wager that the quantity of telemetry information that companies should cope with will solely enhance within the coming years. For that cause, it is vital to make sure that your pipeline can scale as much as accommodate ever-larger volumes of knowledge, in addition to a rise within the variety of information sources and instruments it helps.
Conclusion: Remodeling software efficiency administration and SIEM with telemetry pipelines
The underside line: For a lot of companies, conventional approaches to managing telemetry information not suffice. They’re too gradual, pricey, and difficult to scale.
The answer is to take a deliberate, constant strategy to telemetry information processing and supply by constructing a pipeline to make sure that each information supply reaches its meant vacation spot and is able to help its meant use case. Once you do that, you’ve got set your group up for long-term success within the period of more and more massive and complicated telemetry information units.
