An on-premises AI factory needs a governed path from data intake to model evaluation and production serving, backed by compute, storage, networking and an operating team. Size those components against a representative workload before ordering hardware. GPU count alone cannot tell you whether the system will meet its latency, recovery or data-control requirements.
This reference design is for planning a customer-controlled deployment. The bill of materials identifies the decisions a supplier quote must resolve; it is not a tested hardware configuration or a claim about capacity.
What should the first workload establish?
Choose a use case with a measurable output. For a document assistant, specify the documents it may retrieve, the permitted users, expected request patterns and what a useful answer looks like. For a simulation pipeline, specify the input assets, job duration target and artifacts that must be retained.
Measure realistic input sizes. Long documents, large images and concurrent requests can change memory and throughput requirements substantially. A demonstration using short prompts cannot size a system that will process lengthy technical manuals.
Agree on failure behaviour too. A service that can queue work overnight needs a different design from one supporting an interactive workflow. Decide whether a failed request should retry, return an explicit error or use an approved fallback. A fallback outside the intended environment is a data-flow decision.
What does the architecture look like?
The following arrangement separates shared platform controls from workload-specific components. Names identify possible components, not a requirement to install every product shown.
Trace a request through the gateway, serving layer and retrieval store. Then trace an administrator changing a model version. These paths require different permissions, and both need records that an operator can inspect.
Kubernetes guidance on multi-tenancy explains why sharing a cluster involves more than creating namespaces. Isolation choices must account for network access, resource contention and the trust relationship between workloads.
Which components belong in the bill of materials?
Request a quote that separates hardware, software subscriptions, support, installation and ongoing services. An attractive server price can conceal a facility upgrade or an unsupported software combination.
Do not select an accelerator solely because its memory can hold the model weights. Serving also consumes memory for runtime state and concurrent requests. Measure the intended model configuration with representative inputs before treating a memory estimate as a purchase recommendation.
How should GPU scheduling be introduced?
The NVIDIA GPU Operator manages software components needed to expose NVIDIA GPUs in Kubernetes. Its supported combinations and installation requirements should guide the platform build.
Begin with separate workload classes for interactive serving and batch work. Decide how priorities, quotas and maintenance affect each class. A batch evaluation should not unexpectedly consume resources reserved for a production endpoint.
This small configuration is an illustrative namespace quota, assuming the NVIDIA extended resource has already been configured. It is not a deployment manifest or a sizing recommendation:
The example caps requested GPU resources in that namespace. It does not establish network isolation, authorise model downloads, create storage or make the workload safe to run. Those controls belong in the surrounding platform design.
What data controls are easy to overlook?
Keep raw inputs, approved datasets, model artifacts and generated outputs distinguishable. A researcher should be able to identify which source version produced an evaluation result without depending on a personal notebook.
Apply access controls at the actual data and service entry points. A gateway policy helps only for traffic that passes through it. Direct database credentials, maintenance interfaces and background jobs need their own controls.
HyperLake's architecture uses customer environments and modular data and AI components, with scoped access and policy checks at integrated access points. A deployment review should identify those points explicitly and document any remaining paths. Do not infer universal enforcement from the presence of a policy engine.
For physical AI workloads, add provenance for sensor recordings, simulation assets and model releases. NVIDIA OSMO, Omniverse and Isaac components may be relevant, subject to licensing and validated integration. A simulation result is not proof of safe behaviour in a physical environment.
How do you plan for disconnected operation?
Write down every external dependency: package repositories, container registries, model downloads, licence checks, identity services, telemetry and support tooling. Determine which can be mirrored or replaced and which require an approved connection.
A restricted network and an air-gapped deployment are different commitments. If the intended system must continue without external connectivity, demonstrate that condition during acceptance. Include certificate renewal, patch delivery and incident response in the exercise.
Create a controlled promotion path for new artifacts. Record origin, integrity checks, review status and the environment in which the artifact was evaluated. That process is especially useful when a model upgrade arrives faster than the infrastructure team's normal release cycle.
What should acceptance cover?
Run the chosen workload with representative data and concurrent demand. Record useful output quality alongside latency, throughput and resource use. A fast endpoint that gives unsuitable answers has not met the workload requirement.
Then examine degradation: a node failure, an unavailable retrieval store, a full disk and a rejected permission request. Confirm that errors are visible and that recovery does not depend on undocumented operator knowledge.
Finally, restore the assets needed to reproduce a release. Backing up model weights is insufficient if the tokenizer, configuration, retrieval index or source permissions are missing. Give the operations team the runbooks and escalation contacts before calling the platform ready.
Frequently asked questions
How many GPUs does an AI factory need?
There is no universal count. Size against the selected models, input lengths, concurrency, training or inference work, and availability needs. Use measurements before committing to a hardware quantity.
Is Kubernetes mandatory?
No. It is a common orchestration option and fits this reference design. A smaller deployment may use a simpler arrangement if it can meet the same operational, security and recovery requirements.
Can the platform run without internet access?
Potentially, but every runtime and maintenance dependency must support that arrangement. Confirm model distribution, licensing, updates, identity and support processes for the selected components.
Does on-premises deployment prove sovereignty?
No. It establishes a location choice. Operator access, keys, external dependencies, contractual control and the ability to maintain or exit the system still need assessment.
