18/04/2023
Observability Engineer
Observability Engineer, Enterprise Engineering
About this team
We are looking for a motivated engineer to become a core member of the Observability team in Enterprise Engineering, guiding the enterprise organization to improve the practice of observability. We are a consultative enablement team providing guidance and support to product engineering teams for the development of high-quality and resilient software systems through the use of monitoring tools and best practices.
As an Engineer for Observability, your team owns the management of our monitoring tools and the best practices for using those tools to provide total visibility into our systems; our two primary tools are Splunk and Datadog. This role requires someone who can help to get most value from our investment, implement and refine governance practices, as well as handling hands on admin tasks support across a disparate organization.
As a successful candidate for this role, you will support our lead engineers to administer our tools, enable our end users, be a custodian for best practices and standards, and work with vendor TAMs, to resolve issues and learn about new capabilities and opportunities.
A day in the life
Tool administration: roles and capabilities, users, API keys, Apps, HEC tokens, indexes, etc
Support users with Getting Data In (GDI), dashboard & alert creation
Tool Governance and Best Practices documentation, training colleagues for our 2 primary tools: Splunk & Datadog Support data retention policies
Build and maintain Splunk Cloud components (Universal Forwarders, Heavy Forwarders, HEC, Add-ons, etc)
Understand our end users needs to ensure our platforms meet their needs
Collaborate with cross-functional teams to troubleshoot and resolve monitoring related issues
Managing Splunk ingestion/SVC usage, and communicating chargeback data
Managing tool agent licensing, data ingestion and resource tagging
Maintain/update administration dashboards for Datadog/Splunk platform health
Qualifications
College degree in computer science/engineering or related field
Minimum 2+ years of experience with Splunk in one of the following areas: IT Operations, compliance, dev ops, network security, and system security, supporting security event management tools (SIEMs) Minimum 1+ year experience working with Datadog.
Strong understanding of Splunk “SPL”, search and dashboard optimization Knowledge of:
PII / CCPA / GDPR rules
Enterprise Single Sign-On
Docker and Kubernetes
Atlassian Suite tools
Experience with Linux
A track record delivering quality results on complex cross-functional projects Analytical and problem-solving capabilities
Strong verbal and written communication skills. Must be able to communicate with a wide variety of audiences, both business and technical.
Bonus
Experience with other monitoring tools such as CloudWatch, New Relic, SignalFX, Thousand Eyes, etc.
Knowledge of ML/ ITSI
Knowledge of OpenTelemetry, experience with OpenTelemetry API/SDK
Knowledge and implementation experience with Splunk Connect for Kubernetes (otel-sck)
Interpersonal Must Haves
Acknowledges the presence of choice in every moment and takes personal responsibility for their life
Possesses an entrepreneurial spirit and continuously innovates to achieve great results
Communicates with honesty and kindness, and creates the space for others to do the same
Leads with courage, knowing the possibility of greatness is bigger than the fear of failure
Fosters connection by putting people first and building trusting relationships
Integrates fun and joy as a way of being and working, aka doesn’t take themselves too seriously
