Behind the Scenes of Building a Government-Grade Product
Verkada partners with many public sector agencies to modernize their physical security and has built our platform with leading security standards and compliance features. However, meeting the stringent requirements of the federal government required us to take our infrastructure to the next level. Our engineering team rearchitected our platform to meet the highest standards of federal security, which resulted in the deployment of a custom version of Verkada Command to AWS GovCloud. Along the way, we built custom tooling for automating reliability and security analysis.
Below, we’ll share our engineering journey in building a platform tailored for the federal government, and explain the technical decisions we made that allowed us to begin our FedRAMP (Federal Risk and Authorization Management Program) audit.
Vehicle Search page on Verkada Command platform for our Gov product.
Architecting for FedRAMP in AWS GovCloud
To achieve FedRAMP Authorized status, we redesigned our system’s operation to meet three architectural pillars:
Restricted Data Residency and Personnel: All data must remain within U.S. borders and be managed exclusively by U.S. persons.
Stringent Security Controls: All data must be protected by FIPS 140 validated cryptography while in transit and at rest. Every layer of our platform, from physical data centers to encryption protocols, had to be federally validated.
Strict Isolation: The cloud environment must be isolated from our commercial product, with its own dedicated infrastructure.
Our commercial platform already ran on AWS, so we chose to build our federal offering in AWS GovCloud (US), giving us a single compliant cloud provider for both products.
Verkada GovCloud: Building an Isolated Environment
Ensuring isolation while still maintaining operational efficiency was our first challenge. Fortunately, Verkada operates on a multi-shard architecture, running self-contained, individually scalable production instances in different locations. We run shards in the U.S., the EU, and Australia, each with dedicated resources including databases, compute, and storage, and more than 100 microservices running on AWS.
High-level user login flow for each production shard.
While this architecture provided the perfect foundation for a new AWS GovCloud-based shard, which we call the Verkada GovCloud, running in the government cloud demanded a level of isolation beyond anything we had implemented before. Our team achieved this with full physical and logical separation:
Distinct Domains: We created command.verkadagov.com, a separate domain that uses DNSSEC to prevent cache poisoning attacks.
Dedicated AWS Accounts: Distinct AWS accounts for compute, storage, and networking, each with its own VPCs and IAM policies, enforce isolation.
Separate Development Lifecycle: Dedicated CI/CD pipelines prevent code and data from crossing boundaries, and only authorized U.S. personnel can make production changes on the Verkada GovCloud. To enforce this, we use a separate code repository and built an internal deployment tool called vgovdeploy, which is designed to ensure changes are tracked by source control and prevent any tampering with the deployment logic.
Isolated Tooling: Because many common cloud services (e.g., GitHub, Temporal) aren’t FedRAMP-compliant, we took on the operational responsibility for management and security, building or self-hosting our own tools. Examples include self-hosting Apache Airflow for our data pipelines, and self-hosting Temporal to manage application workflows, and building “vIaC”.
vIaC: From Isolation to Abstraction
Meeting these requirements for isolation introduced new challenges for our engineering team, as using different tools and building Verkada GovCloud-specific features can be highly time-consuming. To enable our engineers to move more quickly while still putting out a high-quality, Gov-compliant product, we have begun to abstract away the specifics of each shard.
The development of “vIaC”, our own "Infrastructure as Code" workflow management system based on OpenTofu, is a prime example of this new direction. With more than 1,400 workspaces across complex platform use cases, “vIaC” enables automated, reliable, and low-toil rollouts of critical infrastructure changes. For instance, the system could automatically detect a single change to our Redis configuration and apply a safe rollout strategy to all dependent shards, commercial and government alike. This approach not only reduces operational burden but also improves reliability, a true win-win.
Argus Eye: Internal Tool for Rapid Synthetic Monitoring
Reliability also means ensuring that our government customers get the same consistent performance as our commercial customers. To guarantee that all of our shards are 99.99% available, we built “Argus Eye”.
“Argus Eye” is a synthetic monitoring platform that runs critical user journey tests against all of our shards, including Verkada GovCloud, ensuring the end-user experience, from login to camera streaming, is the same across shards.
Uptime for critical workflows for production shards on “Argus Eye”.
“Argus Eye” performs tests every minute that simulate web user actions, such as login and camera home page load.
Architectural diagram for Argus Eye.
The test cases themselves are written using the open-source test automation framework Playwright, which provides APIs for simulating user actions. A Test Runner Controller manages the execution of each test workflow, and for scalability, these workflows run in parallel on a Kubernetes compute cluster, allowing us to easily add more tests in the future. The test results are processed by AWS Lambda before being stored on Aurora, a PostgreSQL-compatible relational database. When a failure is detected, the system is designed to immediately alert on-call engineers via Slack and Opsgenie, enabling fast response times while still meeting the isolation and compliance requirements of a government cloud.
Engineering for Impact
Custom solutions such as our internally-developed “Argus Eye” and “vIaC” tools enabled us to not only meet regulatory requirements and achieve FedRAMP Moderate, but also deliver the best possible experience. We are proud that Verkada GovCloud is successfully meeting high-security government standards to protect people and places for federal agencies, and that it now supports over hundreds of government customers and counting.
Ready to build systems that protect people and places alongside smart and passionate engineers? Explore our open engineering roles.
