Scientific Networking Research Associate

Caltech is a world-renowned science and engineering institute that marshals some of the world's brightest minds and most innovative tools to address fundamental scientific questions. We thrive on finding and cultivating talented people who are passionate about what they do. Join us and be a part of the diverse Caltech community.

The Compact Muon Solenoid (CMS) is one of four experiments at CERN's Large Hadron Collider (LHC) in Geneva, Switzerland. The CMS collaboration, numbering over 3,000 physicists and engineers, makes use of a worldwide computing grid for the processing and analysis of the data taken by the experiment. The Caltech group is one of the leading US institutes in CMS, with an excellent track record and reputation both in particle physics research and in state-of-the-art research and development for scientific computing and networking. Caltech hosts one of seven US-based "Tier2" High Performance computing facilities in CMS, which are part of this grid. The Caltech HEP group also manages testbed clusters as part of testing new technologies and network/computing R&D projects. The successful candidate will join the Caltech High Energy Physics group and the worldwide CMS community, and will work on networking R&D projects in association with Caltech HEP, Fermilab, ESNet, Internet2, the CENIC regional network, CERN and campus network engineers in state of the art network and distributed system developments to evaluate and implement new networking technologies to ensure fast and reliable data transfers to/from the computing clusters and new data access methods. The successful candidate will participate in state-of-the-art developments, working with other CMS sites and the global network community, aimed at making the experiment's data distribution and analysis more efficient through the use of advanced network monitoring, caching and high throughput data transport tools. The candidate also will participate in the ongoing development of a next generation system that interfaces intelligent networks to the data management systems of CMS and the other LHC experiments, in order to meet the experiments' rising needs while also accommodating the network traffic of other programs and the worldwide academic and research community. As part of a small but skilled team, he/she will also assist in maintaining and advancing Caltech HEP's Software Defined Networking (SDN) Testbed computing facility dedicated to network R&D developments. We are looking for an eager self-motivated person who can quickly learn to deal with both hardware and software cluster issues as well as grid issues as needed, who will enjoy working with a dedicated team of scientists and engineers engaged in frontier physics research, along with cutting edge technologies and
systems needed to support this program.

• Develops and operates the scientific networking using as a test bed the Caltech Tier-2 facility and the associated R&D USCMS computing ecosystems, storage and network clusters at Caltech Fermilab and beyond. System management duties for the SDN computing clusters at Caltech include:
  • Hardware installation, OS installation, software installation and updates, monitoring, security set up, and troubleshooting.
  • Maintaining SDN Kubernetes cluster with storage at Caltech.
  • Implement and test networking protocols, automation tools and simulation environments.
  • OS installation, software installation and updates, monitoring, security set up, and troubleshooting.
  • Participate (usually by videoconference) in biweekly CMS computing meetings where the status of the Caltech CMS Tier2 computing resources are discussed.
  • Work with members of the HEP group and keep abreast of emerging computing, networking, storage technologies and evaluate their suitability for the computing clusters.
  • Actively participate in the design and implementation of an integrated infrastructure supporting research and development of new networking techniques.
  • Participate in regular technical and strategy meetings in association with these developments.
  • Work with the Open Science Grid (OSG) and local campus security teams to implement reasonable security safeguards and respond to any security threats.
  • Collaborating with ESNET on deploying SENSE/Rucio Testbed on US-CMS Tier1/Tie2 sites

• Bachelor of Science degree physical sciences or engineering, computer science or equivalent with related work experience.
• Strong aptitude and capacity to work with scientists and engineers engaged in frontier physics research.
• Solid understanding of TCP/IP, OSI layers and networking protocols and networking fundamentals.
• Knowledge and hands on experience in computer networking, including switches and routers, routing and switching protocols, including BGP and OSPF., network interfaces and distributed system and network.
• Analyze network performance, identify bottlenecks, and develop solutions for scalability and resilience which includes working knowledge of tools like iperf3, tcpdump, Wireshark, netcat and perfSONAR.
• Knowledge and hands on experience with open-source projects. Experience in Unix/Linux operating systems and hands-on experience with operating, monitoring and managing large scale computing, networking and storage infrastructures.
• Sound knowledge on GitHub/GitLab.
• Excellent oral and written communication skills including developing, writing, and reviewing technicaldocuments.
• Excellent team player, ability to work independently and as part of a team.
• Ability to work on multiple tasks simultaneously and set priorities.
• Ability to work across institutions and communicate effectively with both operational and research teams
• Some travel in-state and nationally may be required.

• Master's degree with at least 2 years of work experience.
• Operational experience of production network operations.Ability to learn any network operating system including but not limited to Dell, Cisco, P4, and Sonic, and to adapt to new network tools, methods and devices as needed.
• Experience with automation tools such as Foreman, Puppet and Ansible.
• Knowledge on virtualization such as network, storage.
• Solid knowledge and hands-on experience on software-defined networking (SDN).
• Knowledge and experience with container networking as Kubernetes and Docker.
• Hands-on experience with network monitoring systems (e.g., Grafana, Prometheus), packet analysis, and performance tuning.
• Proficiency in shell scripting, Python.
• Familiarity with R&D networking environments and wide-area performance challenges.

• Resume.