Staff Engineer, Observability

We are seeking a Staff engineer to design, build, and operate the internal and external Observability stack for the MongoDB platform. Tens of thousands of customers depend on our Observability stack to monitor their database clusters and to generate actionable alerts to safeguard critical workloads. This is an opportunity to join a team that is responsible for all Observability systems that support metrics, metric visualization, logs, traces, and alerts for MongoDB. We are looking for engineers with high standards, and experience in setting direction and technical leadership for large engineering teams in designing and operating complex distributed systems, with strict SLO on security, durability, availability and performance. As MongoDB Atlas and its supporting infrastructure continue to experience rapid growth, the demand for high-cardinality observability data for internal and external use cases means we need to continually innovate and scale our systems to the next level. For example, MongoDB Observability systems need to handle 10’s of billions of metrics time series, all whilst processing petabytes of logs, traces, and events. Our stack includes VictoriaMetrics, Splunk, Flink, WarpStream/Kafka, Java, Golang Fluentbit. In addition to owning critical components of our observability infrastructure, as a Staff engineer on the team, you’ll also work closely with other SWE, Product and SRE teams to promote and implement best practices in instrumenting and monitoring their services. This is a highly collaborative role, and you will get to own some of the most relied upon internal infrastructure at Mongo. Our team champions a strong culture of inclusivity, diversity, and collaboration. If you want to be a deeply technical leader on a collaborative team that applies low-level systems expertise to build the foundational infrastructure of a popular database, join us! Let’s build a faster, more reliable, and exceptionally observable database system together. We are looking to speak to candidates who are based in Dublin for our hybrid working model. Candidate Profile • Solving complex problems at scale, and to high standards, excites you • Minimum 10 years of experience in designing, programming, debugging, and tuning distributed and/or highly concurrent C/C++/Java/Rust mission critical software systems. • Experience running latency sensitive, high throughput systems • Strong systems fundamentals, including multi-threaded programming, performance profiling, and expert-level programming • Familiarity with database internals or building core components for data processing systems • (Nice to have) Indexing or database performance tuning experience • Familiarity with observability ecosystem and best practices • Excellent verbal and written technical communication skills and a strong desire to collaborate with colleagues and mentor engineers • Excellent time and project management skills including the ability to make realistic assessments of project cost and complexity • Has a good understanding of information security management Responsibilities • Help build and architect systems for the mission-critical observability platform, leading the architecture and implementation of components that drive performance, scalability, cost-efficiency, and resiliency • Design and implement observability improvements that enable MongoDB engineers and customers to quickly and accurately diagnose the root cause of production issues. • Handle production customer escalations from Technical Support team and coach teammates to do the same • Write production-ready database code, improve the existing code, and mentor their team to write higher quality code • Own areas of code that the Observability Team maintains, ensuring it achieves a high standard for quality (including security, durability, availability, and performance) and maintainability • Diagnose test failures, identify bugs in existing code, fix them, and prevent bugs from being introduced in new code • Investigate the performance impact of code changes that may cause software performance regressions • Interview candidates for advanced software engineering positions • Develop and maintain expertise on cutting edge database, and observability developments from industry and academia • Lead development and project management of some of the larger projects across the company • Collaborate with stakeholders and engineering teams across the company to jointly work on large initiatives • Advise Product Management on technical product direction, engineering complexity and inter-project dependencies • Collaborate with Product Management and Engineering leadership to define product roadmaps • Champion and improve unit and integration tests to demonstrate correctness • Work with customers and support engineers to fix issues and become part of our on-call rotation Success Measures • In the first month, you will have understood the high level architecture of MongoDB observability, and fixed a few bugs • In three months, you will have contributed to the development of a project slated for the next major release of MongoDB, and diagnosed and fixed a few customer or testing-reported issues • In six months, you will have taken on code review responsibilities and are involved in reviewing the design for new features • In twelve months, you will be leading the development of new features and initiatives, helping to mentor engineers on the team, and using your experience to drive the long-term technical roadmap of the Observability Team About MongoDB MongoDB is built for change, empowering our customers and our people to innovate at the speed of the market. We have redefined the data platform for the AI era, enabling builders to create, transform, and disrupt industries with software. MongoDB’s unified data platform, the most widely available, globally distributed data platform on the market, helps organizations modernize legacy workloads, e