September 11, 2026
12 min read
By Kokil Thapa | Last reviewed: September 2026
Your database outgrew local SSD capacity, but RDMA NICs are not in the budget. NVMe over TCP Basics matter because they let you attach remote NVMe namespaces over ordinary Ethernet—the same switches and firewalls you already run for TCP/IP fundamentals in DevOps. NVMe over Fabrics (NVMe-oF) keeps the NVMe command set end to end. TCP is simply the transport. In practice, that means lower latency than iSCSI on many workloads without InfiniBand hardware. This guide covers the wire model, Linux host and target setup, and the decisions I weigh when sizing storage for production apps.
Quick answer: NVMe over TCP Basics boil down to running the NVMe command protocol across a standard TCP connection—usually port 4420—between a Linux initiator (nvme-tcp) and a target exposing NVMe namespaces, giving block storage over existing IP networks without RDMA.
What is NVMe over TCP and why does it matter for modern storage?
NVMe was built for PCIe-attached flash. NVMe over Fabrics extends that same queue-based command model across a network. NVMe/TCP wraps those commands in TCP segments. The initiator still sees a normal /dev/nvme* block device.
That design choice matters for teams running Ubuntu 22/24 servers with MySQL, PostgreSQL, or Redis. You get shared storage for Kubernetes workers, backup targets, or HA database pairs. You do not need a dedicated Fibre Channel fabric or RDMA-capable NICs on every node.
On client projects where I handle both application code and Linux system administration, NVMe/TCP often sits between “local disk only” and “full SAN budget.” A 10 GbE link with jumbo frames and tuned TCP buffers can serve many Laravel or WordPress stacks cleanly. Latency-sensitive OLTP may still need local NVMe or RDMA—but TCP closes a real gap.
The NVM Express organisation publishes the NVMe-oF specification that defines TCP binding, discovery, and authentication hooks. Linux mainline has shipped initiator and in-kernel target support for several years. Vendor arrays from Pure, NetApp, and others also expose NVMe/TCP front ends.
Core terms you will see in documentation
- Host / initiator: the machine that connects and mounts remote namespaces.
- Target / subsystem: the appliance or Linux box exporting namespaces.
- NQN: NVMe Qualified Name—a unique ID like an iSCSI IQN.
- Namespace: the logical unit your OS formats and mounts.
- Discovery: a well-known service that lists available subsystems on a network.
If you manage infrastructure for platforms like high-traffic booking systems, shared NVMe/TCP storage can simplify live migration and snapshot workflows. The block device behaves like local flash from the application's point of view.
How does NVMe over TCP transport work on the wire?
Understanding the wire model prevents misconfigured MTUs and mystery latency spikes. NVMe/TCP does not tunnel SCSI commands. It serialises native NVMe command capsules inside TCP payloads.
Each connection negotiates header digest and data digest options during the initial exchange. Most production setups enable data digest (CRC) for integrity. Header digest is optional and adds CPU cost.
PDU types in a live session
- ICREQ / ICRESP: initial connection request and response set protocol parameters.
- H2C / C2H term: host-to-controller and controller-to-host command capsules carry NVMe commands.
- R2T: ready-to-transfer tells the host where to send write data.
- DATA: carries payload for read/write operations.
- H2C / C2H data: inline data PDUs for small transfers.
TCP handles retransmits and ordering. That is the main trade-off versus RDMA, which bypasses much of the kernel networking stack. For many web and API workloads, the simplicity wins.
Default IANA ports are 4420 for I/O and 8009 for discovery. Document these in firewall change tickets alongside SSH and database ports. I log JSON discovery output through a quick pass in the JSON formatter tool when debugging automation scripts.
The official NVM Express specifications define capsule formats and discovery log pages. The Linux kernel documents driver behaviour in its NVMe/TCP documentation.
How do you enable NVMe over TCP on Linux hosts and targets?
Lab setup on Ubuntu 22.04 or 24.04 takes under an hour with two VMs. Production targets need multipath, monitoring, and documented recovery steps before you point a database at remote volumes.
Initiator (host) setup
Load the kernel module and confirm it registered:
sudo modprobe nvme-tcp
lsmod | grep nvme_tcp
sudo apt install nvme-cli
Discover subsystems on a target IP:
sudo nvme discover -t tcp -a 192.168.10.50 -s 4420
Connect using the SubNQN from discovery output:
sudo nvme connect -t tcp -n nqn.2024-01.com.example:disk1 \
-a 192.168.10.50 -s 4420
Verify the new device and partition it like local NVMe:
lsblk | grep nvme
sudo nvme list
sudo mkfs.ext4 /dev/nvme1n1
sudo mount /dev/nvme1n1 /mnt/nvme-remote
Persist connections across reboots with /etc/nvme/hostnqn and a systemd unit, or use nvme connect-all with a JSON config under /etc/nvme/. I treat fstab entries on remote NVMe the same way I treat iSCSI: verify _netdev and network-online targets.
Target setup with kernel nvmet
On the storage node, enable the target framework:
sudo modprobe nvmet
sudo modprobe nvme-tcp
sudo apt install nvme-cli nvmetcli
Create a subsystem, namespace, and TCP port via configfs:
sudo nvmetcli
> cd /subsystems
> create nqn.2024-01.com.example:disk1
> cd nqn.2024-01.com.example:disk1/namespaces/1
> set device=/dev/nvme0n1
> set enable=1
> cd /ports/1
> set addr traddr=192.168.10.50
> set addr trtype=tcp
> set addr trsvcid=4420
> cd subsystems
> create nqn.2024-01.com.example:disk1
> saveconfig /etc/nvmet/config.json
> exit
For performance testing, SPDK's userspace NVMe/TCP target can push higher IOPS on dedicated cores. That path suits Linux performance tuning exercises more than a quick lab.
Kubernetes clusters consuming NVMe/TCP volumes typically use CSI drivers from the array vendor or open-source projects. Read the Kubernetes deployment guide first if you are attaching remote volumes to worker nodes. Node loss must not orphan mounts.
NVMe over TCP vs iSCSI vs NVMe-oF RDMA: which should you choose?
Protocol religion helps nobody. Match transport to latency needs, existing hardware, and ops skill on the team.
| Criterion | NVMe/TCP | iSCSI | NVMe-oF RDMA |
|---|---|---|---|
| Network hardware | Standard Ethernet NIC | Standard Ethernet NIC | RDMA-capable NIC (RoCEv2 or IB) |
| Typical latency | Low–medium | Medium–high | Lowest |
| Protocol overhead | NVMe native over TCP | SCSI over TCP | NVMe native over RDMA |
| Linux maturity | Mainline host + target | Very mature (open-iscsi) | Mature with tuned fabrics |
| Firewall friendliness | TCP 4420, 8009 | TCP 3260 | UDP/TCP plus PFC/DSCP tuning |
| Best fit | Modern all-flash over IP | Legacy apps, mixed estates | HFT, low-latency OLTP |
Choose NVMe/TCP when you want NVMe semantics without RDMA switch configuration. Stay on iSCSI when legacy Windows or old SAN tooling cannot migrate yet. Pick RDMA when microseconds matter and you can invest in lossless Ethernet or InfiniBand.
For Nepal-based businesses weighing cloud vs on-prem, NVMe/TCP on a pair of Ubuntu servers can beat managed block storage on cost. Expect roughly Rs 15,000–40,000/month (~USD 110–295) for colocated hardware versus higher cloud IOPS charges. Run the numbers against your FinOps baseline before committing.
Databases like MongoDB or PostgreSQL benefit from predictable latency. Pair remote NVMe with MongoDB administration best practices—place the oplog and data on separate namespaces when the target allows it.
What are common NVMe over TCP deployment mistakes in production?
Most failures I troubleshoot are network and config issues, not NVMe protocol bugs. The symptoms look like slow queries or hung mounts after reboot.
Network and tuning errors
- MTU mismatch: jumbo frames on one hop only cause silent fragmentation or black holes.
- Undersized TCP buffers: high-throughput links need tuned
net.core.rmem_maxandwmem_max. - Single path: one cable or one switch port is a SPoF—use multipath or bonded links.
- CPU starvation: software digest and encryption eat cores; size targets accordingly.
Apply the same discipline you use for SELinux hardening. NVMe/TCP ports must stay allowed in firewall policies after policy reloads.
Security and operations gaps
NVMe/TCP in Linux does not replace TLS for data-at-rest on the wire in all setups. Run it on isolated storage VLANs. Use CHAP or TLS where your target supports in-band authentication per the NVMe-oF spec.
Snapshot and backup teams must know namespaces are block devices. File-level backup agents need mount awareness. For enterprise application stacks, document which namespaces hold database data versus static assets.
Monitoring should track path state, latency, and queue depth. Tools like nvme list-subsys, iostat -x, and array-side dashboards belong in the same runbook as database slow-query alerts. Include checks in your testing and optimization checklist before major releases.
I've seen reboot ordering bite teams that skip network-online.target. The app starts, MySQL opens tables on a missing device, and systemd marks the unit started anyway. Fix the unit dependencies—not the database config.
Hosting providers offering only iSCSI today may add NVMe/TCP soon. When evaluating domain and hosting options, ask whether block storage supports NVMe-oF and which ports they expose.
TLS for management planes overlaps with topics in PKI and certificate management. Keep storage management off the public internet even if I/O VLANs are private.
GPU-heavy workloads rarely share NVMe/TCP paths with inference nodes, but mixed clusters appear in CUDA container setups. Isolate storage traffic with VLAN QoS so training jobs do not contend with database I/O.
After cutover, validate page-speed and app latency through speed optimization monitoring. Remote flash should not regress Time to First Byte on your Laravel or WordPress front ends.
Long-term, plan support and maintenance windows for kernel upgrades. nvme-tcp driver changes land in point releases. Test disconnect/reconnect on staging before production kernel bumps.
Infrastructure projects like SRP Infrastructure Development Nepal show why documented storage architecture matters. NVMe/TCP fits the same story: clear diagrams, named owners, tested recovery.
If you are new to Kokil's work, the about page outlines the full-stack and Linux ops background behind these recommendations.
Key Takeaways
- NVMe over TCP exports remote flash as standard
/dev/nvme*devices over IP port 4420 without SCSI translation. - Use
nvme discoverthennvme connect -t tcpon Linux initiators; configure targets with nvmetcli or vendor arrays. - Pick NVMe/TCP over iSCSI for modern NVMe arrays on plain Ethernet; pick RDMA when sub-100 µs latency is mandatory.
- Fix MTU, TCP buffer, multipath, and boot-order issues before pointing production databases at remote namespaces.
- Isolate storage traffic on dedicated VLANs, enable data digest on untrusted paths, and monitor path health continuously.
- Document NQNs, ports, and failover steps in the same runbook as database backup and restore procedures.
People Also Ask
What port does NVMe over TCP use?
NVMe/TCP uses TCP port 4420 for I/O connections by default. Discovery services typically listen on port 8009. Open both between initiator and target subnets, and restrict them to storage VLANs rather than the public internet.
Does Linux support NVMe over TCP as both initiator and target?
Yes. Mainline Linux includes the nvme-tcp host driver and the nvmet in-kernel target framework. Install nvme-cli on initiators and nvmetcli for configfs-based target management on Ubuntu and most current distributions.
Is NVMe over TCP faster than iSCSI?
On equivalent networks, NVMe/TCP usually delivers lower latency and higher IOPS than iSCSI because it avoids SCSI command translation. Exact gains depend on workload, digest settings, NIC offload, and target implementation. Benchmark with fio on your own hardware before migrating production volumes.
Can Kubernetes use NVMe over TCP volumes?
Yes, through CSI drivers provided by storage vendors or community projects. Worker nodes connect namespaces with the standard NVMe/TCP initiator stack, then the CSI plugin publishes PersistentVolumes. Ensure multipath and node reboot ordering are configured before scheduling stateful pods.
Build storage that your applications can trust
NVMe over TCP Basics are approachable once you treat remote namespaces like local block devices with extra network dependencies. Start in a lab, benchmark with realistic I/O patterns, then roll out dual paths and monitoring before production cutover. If you want help designing Linux storage, HA database layout, or full-stack deployment on top of NVMe/TCP, contact us to discuss your infrastructure goals.
Frequently Asked Questions
NVMe over TCP runs the native NVMe command protocol across a standard TCP connection, usually on port 4420, between a Linux initiator and a target exposing remote NVMe namespaces. The host sees normal /dev/nvme* block devices without SCSI translation or RDMA hardware.
NVMe over TCP uses TCP port 4420 for I/O by default. Discovery services typically listen on port 8009. Document both in firewall change tickets and restrict them to storage VLANs, not the public internet.
On equivalent networks, NVMe over TCP usually delivers lower latency and higher IOPS than iSCSI because it avoids SCSI command translation. Exact gains depend on workload, digest settings, NIC offload, and target implementation.
NVMe over TCP serialises native NVMe command capsules inside TCP payloads rather than tunneling SCSI. Connections negotiate header and data digest options during the initial ICREQ/ICRESP exchange. PDU types include H2C and C2H command capsules, R2T for writes, and DATA payloads. TCP handles retransmits and ordering, which adds overhead versus RDMA but keeps standard Ethernet viable for many web and API workloads.
Yes. Mainline Linux ships the nvme-tcp host driver and the nvmet in-kernel target framework. On Ubuntu 22.04 or 24.04, load nvme-tcp on initiators, install nvme-cli for discovery and connect, and use nvmetcli with configfs on targets. Vendor arrays from Pure, NetApp, and others also expose NVMe/TCP front ends if you prefer appliance targets over kernel nvmet.
Load the nvme-tcp module, install nvme-cli, then run nvme discover -t tcp against the target IP on port 4420. Connect with nvme connect -t tcp using the SubNQN from discovery output. Verify with lsblk and nvme list, format the namespace, and mount it. For production, persist connections via /etc/nvme/hostnqn, a systemd unit, or nvme connect-all with JSON under /etc/nvme/, and use _netdev plus network-online targets like you would for iSCSI.
On the storage node, load nvmet and nvme-tcp, install nvme-cli and nvmetcli, then create a subsystem NQN, bind a namespace to a local block device such as /dev/nvme0n1, and expose it on a TCP port with trtype=tcp and trsvcid=4420. Save the config to /etc/nvmet/config.json. For higher IOPS lab testing, SPDK's userspace NVMe/TCP target can push more throughput on dedicated cores, though kernel nvmet suits most production labs and small deployments.
Choose NVMe over TCP when you want NVMe semantics on standard Ethernet without RDMA switch tuning. Stay on iSCSI for legacy Windows estates or SAN tooling that cannot migrate yet. Pick NVMe-oF RDMA when sub-100 microsecond latency is mandatory and you can invest in RDMA-capable NICs plus lossless Ethernet or InfiniBand. NVMe/TCP sits between local disk only and full SAN budget for teams running MySQL, PostgreSQL, or Redis on Ubuntu servers.
For Nepal-based businesses weighing cloud versus on-prem, NVMe/TCP on a pair of Ubuntu servers can beat managed block storage on cost. Expect roughly Rs 15,000 to 40,000 per month, about USD 110 to 295, for colocated hardware versus higher cloud IOPS charges. Run the numbers against your FinOps baseline before committing, because savings depend on IOPS profile, redundancy, and ops labour you must cover yourself.
Most failures are network and config issues, not protocol bugs. Watch for MTU mismatch when jumbo frames are enabled on only one hop, undersized TCP buffers on high-throughput links, single-path cabling or switch ports without multipath, and CPU starvation from software digest or encryption on undersized targets. Reboot ordering without network-online.target can leave MySQL or other apps starting before remote devices appear. Fix unit dependencies and validate disconnect or reconnect on staging before kernel upgrades.
NVMe over TCP in Linux does not replace wire encryption in all setups, so run it on isolated storage VLANs rather than exposing ports 4420 and 8009 broadly. Enable data digest for integrity on untrusted paths; header digest is optional and costs CPU. Use CHAP or TLS where your target supports in-band authentication per the NVMe-oF spec. Keep storage management off the public internet even when I/O VLANs are private, and document namespace ownership alongside database backup runbooks.
Yes, through CSI drivers from storage vendors or community projects. Worker nodes connect namespaces with the standard Linux NVMe/TCP initiator stack, then the CSI plugin publishes PersistentVolumes. Before scheduling stateful pods, configure multipath and node reboot ordering so node loss does not orphan mounts. Read the Kubernetes deployment guide for your chosen driver first, because path health and discovery behaviour vary between array vendors and open-source targets.
NQN stands for NVMe Qualified Name, a unique identifier for subsystems similar to an iSCSI IQN. Discovery lists available subsystems with their NQNs, and the initiator connects using that SubNQN plus target address and port. Document NQNs alongside ports and failover steps in the same runbook as database backup and restore procedures so on-call engineers can reconnect quickly after storage or network changes.
Use multipath or bonded links whenever a single cable or switch port would otherwise be a single point of failure. Production layouts should include dual initiators, redundant network paths, and paired storage nodes before pointing databases at remote namespaces. Monitoring should track path state, latency, and queue depth using tools like nvme list-subsys and iostat -x alongside array dashboards, matching the same discipline you apply to database slow-query alerts.
A 10 GbE link with jumbo frames and tuned TCP buffers can serve many Laravel or WordPress stacks cleanly, but jumbo frames must be consistent end to end or you risk silent fragmentation or black holes. Raise net.core.rmem_max and wmem_max for high-throughput links. Isolate storage traffic with VLAN QoS so other workloads do not contend with database I/O. Benchmark with realistic fio patterns on your own hardware before production cutover because latency-sensitive OLTP may still need local NVMe or RDMA.
0 Comments
Leave a comment
Your email is not published. Comments appear once they have been read. Sign in to have your details filled in.
