Lead Site Reliability Engineer – Observability

Job Title: Lead Site Reliability Engineer – Observability
Location: Remote, United States
Work Arrangement: Fully remote within the United States
Employment Type: Contract (12 months)
Experience: 7+ years

About the Lead Site Reliability Engineer Role

Join an Observability team responsible for designing, building and operating enterprise platforms for logging, metrics, tracing and alerting across large-scale cloud infrastructure. You will lead initiatives that improve reliability, scalability and operational excellence.

Key Responsibilities

• Design, deploy and operate enterprise observability platforms.




• Build and maintain Splunk Enterprise/Splunk Cloud infrastructure, including Indexers, Search Head Clusters, Heavy Forwarders and Deployment Servers.




• Deploy and operate large-scale Elasticsearch clusters for log analytics and search.




• Design, deploy and support distributed tracing platforms using Grafana Tempo and OpenTelemetry.




• Build and maintain end-to-end tracing pipelines, instrumentation standards and trace retention strategies.




• Scale Prometheus, Grafana, Kafka, Tempo and OpenTelemetry-based monitoring solutions.




• Develop dashboards, alerts, analytics and trace visualizations using Splunk SPL, Grafana, Kibana and Tempo.




• Automate infrastructure using Terraform and configuration management tools.

Required Qualifications

• 7+ years in Site Reliability Engineering, Platform Engineering or DevOps.




• Hands-on experience administering Splunk Enterprise or Splunk Cloud.




• Strong knowledge of Splunk SPL.




• Experience with Elasticsearch/ELK, Prometheus, Grafana, Grafana Tempo, distributed tracing, OpenTelemetry and Kafka.




• Experience implementing metrics, logs and traces as part of a modern observability strategy.




• Experience with Terraform and Infrastructure as Code.




• Programming experience in Python, Go, Ruby or Bash.

Preferred Qualifications

• Splunk certification.




• Experience with Kubernetes, AWS/Azure/GCP, Ansible, Consul, CI/CD pipelines and service mesh technologies.




• Experience supporting FedRAMP or regulated environments.

Technology Stack

Splunk Enterprise, Splunk Cloud, Elasticsearch, ELK, Kibana, Prometheus, Grafana, Grafana Tempo, OpenTelemetry, Distributed Tracing, Kafka, Terraform, Kubernetes, Docker, Linux, Python, Go, Ruby, Bash, AWS, Ansible and Consul.

FedRAMP and IL-5 Environment Notice

The successful applicant may perform work in FedRAMP High or IL-5 environments.

Ready to Apply?

Send your updated resume to jobs@rishiwrites.com with the subject line “Lead SRE Observability, US Remote”.

Looking for more opportunities? Check out other positions.

Want guidance on your resume or career path? Let’s discuss your next career move.

Frequently Asked Questions About the Lead Site Reliability Engineer Role

1. What does a Lead Site Reliability Engineer do in this observability role?

You will design, deploy and operate enterprise platforms for logging, metrics, tracing and alerting across large-scale cloud infrastructure. You will also lead initiatives that improve reliability, scalability and operational excellence.

2. How much experience is required?

Candidates need 7+ years in Site Reliability Engineering, Platform Engineering or DevOps.

3. Is this role remote?

Yes. This is a fully remote position for candidates based in the United States.

4. Is this a full-time or contract position?

This is a 12-month contract position.

5. How important is Splunk experience?

It is central to the role. You should have hands-on experience administering Splunk Enterprise or Splunk Cloud and strong knowledge of Splunk SPL.

6. Which Splunk components will I work with?

You will build and maintain Splunk infrastructure, including Indexers, Search Head Clusters, Heavy Forwarders and Deployment Servers.

7. What Elasticsearch work is involved?

You will deploy and operate large-scale Elasticsearch clusters for log analytics and search.

8. What distributed tracing tools are used?

The role focuses on Grafana Tempo and OpenTelemetry. You will build end-to-end tracing pipelines, instrumentation standards and trace retention strategies.

9. Which monitoring tools will I scale?

You will scale Prometheus, Grafana, Kafka, Tempo and OpenTelemetry-based monitoring solutions.

10. Will I build dashboards and alerts?

Yes. You will develop dashboards, alerts, analytics and trace visualizations using Splunk SPL, Grafana, Kibana and Tempo.

11. Is Infrastructure as Code experience required?

Yes. Experience with Terraform and Infrastructure as Code is required, and you will automate infrastructure using Terraform and configuration management tools.

12. Which programming languages are expected?

Programming experience in Python, Go, Ruby or Bash is required.

13. What preferred qualifications will strengthen my application?

A Splunk certification, experience with Kubernetes, AWS/Azure/GCP, Ansible, Consul, CI/CD pipelines and service mesh technologies, and experience supporting FedRAMP or regulated environments.

14. Will this role involve FedRAMP or IL-5 environments?

Yes. The successful applicant may perform work in FedRAMP High or IL-5 environments.

15. How do I apply?

Email your updated resume to jobs@rishiwrites.com with the subject line “Lead SRE Observability, US Remote”.