AI Infrastructure Operator

Inovi Inc.  - Richmond, VA 23219

$32.50 an hour

401(k), Health insurance, Retirement plan, 401(k) matching, Paid time off, Vision insurance, Health savings account, Dental insurance, Life insurance

Full-time : Weekends as needed, Night shift, Rotating shift, Holidays

Full job description

Location: Richmond, VA — On-site (this is not a remote or hybrid role)

Type: Full-time, exempt

Reports to: Director of Infrastructure / VP of Operations

Direct reports: none

Work Authorization: Due to client contractual obligations, all applicants must be a US Citizen or Permanent Resident

Position Summary

Inovi's AI Infrastructure Operations Center (AIOC) is the always-on nerve center for our AI compute infrastructure. Every GPU cluster, switch, router, and rack in our fleet is monitored, maintained, and restored from this room. As a Tier 2 AI Infrastructure Operator, you're the person who takes an alert past triage and actually resolves it — hands on the console, hands on the CLI, hands on the ticket until the customer is back to healthy.

This is a hands-on operations role on a rotating 24/7 schedule. AI training workloads don't stop overnight, and neither does the AIOC floor.

Essential Job Function and Key ResponsibilitiesTeam Leadership

• Monitoring & incident response
• Own alerts from the monitoring stack through resolution — triage severity, diagnose root cause, remediate, and document

Bare metal fleet management

• Perform firmware and BMC/IPMI updates, out-of-band management, and remote console recovery
• Investigate thermal, power, and hardware fault conditions; coordinate with data center and vendor RMA processes
• Direct on-site smart hands through component replacement and physical remediation, and validate the fix before returning the node to service

Provisioning & deprovisioning

• Provision servers end to end — imaging, OS install, network configuration, validation, burn-in, and handoff to the customer
• Configure and deploy switches and routers, including VLANs, port configs, LACP/bonding, and uplink turn-up
• Execute clean deprovisioning: secure data wipe, decommissioning records, inventory and DCIM updates, asset reclaim
• Keep asset, IP, and rack elevation records accurate — the fleet is only as reliable as the documentation

Network operations

• Troubleshoot L1–L3 issues: link flaps, optics and transceiver faults, routing and switching problems, packet loss, latency
• Support high-performance fabrics used for AI workloads (InfiniBand and/or RoCE/Ethernet)
• Verify connectivity, BGP sessions, and circuit health; coordinate with carriers and transit providers on outages

Customer support & general IT

• Handle customer tickets with professionalism and clarity, translating technical detail into plain language
• Respond to access, connectivity, and configuration requests within SLA
• Support internal IT needs: user accounts and access, endpoint issues, internal tooling, and general helpdesk coverage
• Contribute to knowledge base articles, runbooks, and shift handoff notes

What you bring

Required

• 3+ years in an operations center, data center, MSP, or infrastructure operations role
• Solid Linux administration skills (RHEL/Ubuntu): CLI fluency, log analysis, networking tools, systemd, storage basics
• Working knowledge of networking fundamentals — TCP/IP, subnetting, VLANs, routing, DNS, DHCP — and comfort in switch and router CLIs
• Experience with ticketing and monitoring platforms (e.g., Jira/ServiceNow, Zabbix/Prometheus/Grafana, or equivalents)
• Ability to work a rotating 24/7 schedule including nights, weekends, and holidays
• Clear written and verbal communication under pressure

Preferred

• Exposure to AI/HPC infrastructure: NVIDIA GPUs, InfiniBand, GPU health checks and diagnostics, NCCL testing
• Scripting for automation (Bash, Python) and familiarity with Ansible, Terraform, or similar
• IPMI/Redfish, PXE/netboot, and bare metal provisioning tooling (MAAS, Foreman, Tinkerbell, or comparable)
• Certifications: CompTIA Server+/Network+, CCNA, RHCSA, or NVIDIA data center credentials
• Experience in a customer-facing hosting, colocation, or cloud provider environment

What success looks like

In 90 days you're running full shifts independently, resolving Tier 2 incidents without escalation, and provisioning servers end to end.

In a year you've become a trusted escalation point for the AIOC, improved runbooks and automation that cut repeat incidents, and helped raise the technical bar across the operator team.

Why Inovi

You'll work on the physical infrastructure behind real AI workloads — not abstractions, but actual racks, actual fabric, actual scale. The AIOC is where problems surface first, which makes it the fastest place in the company to learn how all of it fits together.

Virginia is the center of gravity for global data center infrastructure, and Inovi's AIOC sits right in it. You'll build a skill set that's in short supply and high demand — and you'll build it on real hardware, not in a simulator.

• Serve as the escalation point for Tier 1 operators; take handoff on incidents that need deeper hardware or network work
• Drive incidents to resolution against defined SLAs, providing clear status updates to customers and internal stakeholders throughout
• Execute runbooks under pressure, and flag or rewrite the ones that don't hold up in the real world

Inovi is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, protected veteran status, or any other characteristic protected by law. Inovi will use e-Verify to confirm work authorization.

Pay: $32.50 per hour

Benefits:

• 401(k)
• 401(k) matching
• Dental insurance
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Retirement plan
• Vision insurance

Experience:

• Linux: 1 year (Required)
• Nvidia GPU troubleshooting: 1 year (Preferred)