Linux Logging with journald and rsyslog

By Kokil Thapa | Last reviewed: September 2026

When a Laravel queue worker dies at 2 a.m., you need answers in minutes, not after a support ticket. Linux logging with journald and rsyslog is how most Ubuntu and Rocky Linux servers capture that story today. systemd-journald collects structured, indexed logs from services, the kernel, and applications. rsyslog turns those events into plain-text files or remote syslog streams that fit backup scripts, SIEM tools, and old-school grep workflows. On the Linux system administration work I do for client sites, both daemons run side by side—and understanding how they interact saves real debugging time.

What is the difference between journald and rsyslog on Linux?

journald is systemd’s logging daemon. It writes to a binary journal under /var/log/journal/ or a volatile runtime journal. Each entry carries priority, unit name, PID, and structured fields you can filter instantly. rsyslog is the classic syslog implementation. It reads inputs—from files, sockets, or journal forwarding—and writes traditional log files under /var/log/ or ships them to a remote collector.

They are complementary, not interchangeable. journald excels at “what failed in the last five minutes on this box?” rsyslog excels at “store Apache access logs for ninety days and forward auth events to a central server.” A common mistake is disabling rsyslog because journalctl feels modern, then losing log files that legacy monitoring still expects.

On Ubuntu 22.04 and 24.04 servers I maintain, both daemons start by default. Debian-family images often ship rsyslog preinstalled. RHEL, Rocky Linux, and AlmaLinux follow the same pattern. Treat them as a pipeline: capture first with journald, route second with rsyslog. That matches how systemd manages services on Linux while still honouring decades of syslog conventions.

How do you query logs with journalctl on a Linux server?

journalctl is the front door to journald. It reads the journal from disk or memory and filters by time, unit, priority, and boot session. Start with the basics before you write complex queries.

Essential journalctl commands

# Follow all logs live (like tail -f)
journalctl -f

# Logs for one systemd unit since today
journalctl -u nginx.service --since today

# Only errors and worse (priority 0–3)
journalctl -p err..emerg

# Logs from the previous boot
journalctl -b -1

# JSON output for scripting
journalctl -u php8.3-fpm.service -o json-pretty --since "1 hour ago"

When PHP-FPM throws a segfault after deploy, I reach for journalctl -u php8.3-fpm.service -n 100 --no-pager before I touch application logs. The unit filter cuts noise from unrelated services. Priority filtering surfaces kernel OOM kills that never reach Laravel’s storage/logs directory.

Structured fields worth knowing

• _SYSTEMD_UNIT — ties a line to nginx, mysql, or your queue worker
• _PID and _COMM — identify the exact process
• MESSAGE — the human-readable text
• _BOOT_ID — separates reboot cycles cleanly
• SYSLOG_IDENTIFIER — appears when journald forwards legacy syslog names

For application teams that already follow structured logging best practices, journald’s field model feels familiar. You can grep JSON output or pipe it into a JSON formatter when building one-off reports. The journal is not a replacement for application-level logs—it is the system layer beneath them.

How do you configure rsyslog to receive and route journald logs?

By default, journald forwards many messages to the syslog socket when ForwardToSyslog=yes is set. rsyslog listens on /dev/log and applies rules from /etc/rsyslog.conf and drop-in files under /etc/rsyslog.d/. Your job is to confirm forwarding, define file targets, and optionally ship logs off-host.

Step 1: Verify journald forwarding

Edit /etc/systemd/journald.conf. Ensure these settings are active (uncomment if needed):

[Journal]
Storage=persistent
SystemMaxUse=500M
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no
Compress=yes

Then reload journald:

sudo systemctl restart systemd-journald

Storage=persistent writes under /var/log/journal/ so logs survive reboot. Cap disk use with SystemMaxUse—unbounded journals have filled disks on small VPS instances I have seen. Pair that cap with log rotation and disk space management on Linux policies so text files do not grow unchecked either.

Step 2: Create rsyslog routing rules

On Ubuntu, create /etc/rsyslog.d/50-default.conf fragments or edit the main config. Example: separate Laravel queue worker noise from mail logs.

# /etc/rsyslog.d/30-app.conf
# Local file for auth events
auth,authpriv.*                 /var/log/auth.log

# Capture cron separately
cron.*                           /var/log/cron.log

# Forward all kernel messages at warning+ to remote collector
kern.warning                      @@logcollector.example.com:514

# Discard debug chatter from one noisy identifier
:syslogtag, contains, "noisy-daemon" ~

Validate syntax before restart:

sudo rsyslogd -N1
sudo systemctl restart rsyslog

The @@ prefix means TCP. A single @ uses UDP—fine for lab setups, risky for production. For TLS, use rsyslog’s gnutls module and point to port 6514. Official reference: the rsyslog documentation covers templates, property-based filters, and omprog actions in detail.

Step 3: Read imjournal when forwarding is disabled

Some hardening guides set ForwardToSyslog=no and load rsyslog’s imjournal input module instead. That pulls directly from the journal without the socket hop. Both approaches work. Pick one—duplicating inputs creates double-written lines and confuses auditors.

How do you forward Linux logs to a remote syslog or ELK stack?

Single-server grep stops scaling around ten hosts. At that point you forward rsyslog output to a collector or ship files with Filebeat. I have wired sister sites on shared EC2—similar to the Deployer pipeline behind Notary Kathmandu—to a central syslog receiver for auth and kernel alerts.

1. Define a forwarding rule in /etc/rsyslog.d/90-forward.conf targeting your collector IP and port.
2. Open the firewall only from app servers to the collector—see iptables vs nftables on Linux for rule patterns.
3. On the collector, enable imtcp and write per-host directories or pipe to Elasticsearch.
4. Test with logger -p local0.notice "remote forward test" and confirm arrival.
5. Add monitoring on the collector disk—silent log loss is worse than a loud outage.

# /etc/rsyslog.d/90-forward.conf on each app server
*.*  @@10.0.1.50:514

# Queue for brief network blips
$ActionQueueType LinkedList
$ActionQueueFileName fwdq
$ActionResumeRetryCount -1

For full-text search dashboards, pair forwarded syslog with centralized logging with the ELK stack or a managed observability product. journald remains your first stop on the origin host. Central storage is for correlation across machines and retention policies measured in months, not minutes.

Container hosts add another wrinkle. Docker’s json-file driver writes container stdout to disk on the node. Those lines often hit journald when systemd manages Docker. Read Docker logging drivers before you assume every layer lands in the same file path.

How do you debug production failures using journald and rsyslog together?

Production debugging is a sequence, not a single command. I follow the same path on Laravel stacks, WooCommerce servers, and plain LAMP boxes.

Workflow for a typical outage

1. Check service state: systemctl status nginx php8.3-fpm mysql.
2. Pull recent unit logs: journalctl -u nginx.service --since "30 min ago".
3. Scan priority: journalctl -p warning --since "30 min ago" for kernel and auth clues.
4. Cross-check text logs: sudo tail -n 200 /var/log/syslog and domain-specific files.
5. Correlate timestamps with app logs in storage/logs or Monolog handlers.
6. If disk pressure suspected, run journalctl --disk-usage and review df -h.

On a legal-tech portal I maintain, a failed queue job once traced to AppArmor denying a write—not a PHP bug. journald showed the denial in kernel messages. rsyslog had copied the same line to /var/log/syslog. Application logs alone would have sent us in circles for hours.

Permission problems follow a similar pattern. If rsyslog cannot write a file after rotation, messages silently buffer or drop. Confirm ownership with guidance from Linux file permissions and ACLs explained. rsyslog typically runs as syslog or root depending on distro defaults.

journald retention and vacuum

Persistent journals grow until limits bite. Inspect and trim:

journalctl --disk-usage
sudo journalctl --vacuum-time=14d
sudo journalctl --vacuum-size=400M

Set permanent caps in journald.conf rather than manual vacuum cron jobs. Scheduled tasks still belong in cron or systemd timers—see cron jobs explained for timing patterns. Align journal retention with your rsyslog logrotate schedules so you do not delete the only copy of an event.

rsyslog rate limiting and repeated lines

Runaway loops can fill disks faster than application traffic. rsyslog supports rate limiting via the immark module and legacy $SystemLogRateLimitInterval settings on some distros. journald has RateLimitIntervalSec and RateLimitBurst in journald.conf. Tune these on chatty services after you fix the root cause—not as a permanent bandage.

For ongoing visibility, hook metrics into Linux server monitoring with Netdata and alerts. Alert on disk use, rsyslog restarts, and sudden spikes in journalctl -p err counts. Combine that with support and maintenance runbooks so on-call steps are documented before the next deploy.

What are common mistakes when mixing journald and rsyslog?

Most logging pain is configuration drift, not missing software. These issues recur across client servers.

• Double forwarding paths — ForwardToSyslog and imjournal both enabled produces duplicate lines and inflated storage.
• No persistent journal on servers — Default volatile storage loses history after reboot; debugging “what happened before the crash” becomes impossible.
• Assuming application logs replace syslog — PHP exceptions do not capture OOM kills, SSH brute-force attempts, or failed package upgrades.
• UDP forwarding over the public internet — Logs are plaintext; use TCP/TLS inside VPN or private networks.
• Ignoring timezone alignment — journalctl defaults to local time; remote collectors may use UTC. Match offsets when correlating.
• Never testing rsyslog config — A syntax error on restart can stop all file writes until someone notices missing logs days later.

Official systemd guidance lives in the systemd-journald.service manual. Read it when you tune rate limits, forward targets, or split journals per user with SplitMode=.

If you run performance work alongside logging changes, review Linux performance tuning basics and diagnose high CPU and memory usage so you do not mistake log I/O pressure for application regressions. Heavy synchronous writes to slow disks show up in both journald and rsyslog latency.

For Laravel and PHP-FPM stacks specifically, remember that stderr from systemd units lands in journald automatically when you use StandardError=journal in the unit file. That is cleaner than redirecting to temp files. Pair it with testing and optimization passes after deploy so new services actually emit the fields you expect.

Backup scripts deserve the same attention as application code. Log-based restore audits fail when auth.log rotated off-disk. Coordinate retention with automate database backups on Linux policies so compliance windows overlap. A thirty-day DB backup policy paired with seven-day syslog retention is a gap auditors notice.

When interviewing DevOps candidates, I still ask journalctl basics alongside Linux interview questions for DevOps topics. Knowing both layers separates someone who restarted nginx from someone who traced a failure across kernel, syslog, and application timestamps.

Key Takeaways

• Run journald for indexed local queries and rsyslog for text files, rotation, and remote forwarding—they solve different problems.
• Set ForwardToSyslog=yes and Storage=persistent in journald.conf, then validate with logger and journalctl.
• Use journalctl -u unit -p err --since for fast triage; confirm the same event in /var/log/syslog before you close an incident.
• Cap journal size with SystemMaxUse and schedule vacuum or align limits with logrotate so disks never fill silently.
• Forward with TCP or TLS to private collectors; add queues on rsyslog forward actions to survive brief network loss.
• Correlate journald timestamps, rsyslog files, and application logs—production root cause often lives in the system layer, not the framework.

People Also Ask

Does Ubuntu use journald or rsyslog by default?

Modern Ubuntu uses both. systemd-journald captures service and kernel output by default. rsyslog is installed on server images and receives forwarded journal entries when ForwardToSyslog=yes is active. You get journalctl for live debugging and traditional files under /var/log/ for scripts and remote shipping.

Can I disable rsyslog and rely only on journald?

Yes, but weigh the trade-offs carefully. journald supports remote forwarding via ForwardToNetwork, yet most enterprise collectors expect classic syslog over TCP. Disabling rsyslog breaks tools that tail /var/log/auth.log or depend on legacy parsing. For single-server setups with adequate journal retention, journal-only can work.

Where are journald logs stored on disk?

With persistent storage enabled, binary journals live under /var/log/journal/ in a machine-id subdirectory. Volatile mode uses /run/log/journal/ and clears on reboot. Check current usage with journalctl --disk-usage and adjust SystemMaxUse in /etc/systemd/journald.conf.

How do I convert journald logs to plain text files?

Either forward from journald to rsyslog and let rsyslog write files, or export on demand with journalctl -u service.name -o short-iso > export.log. For ongoing plain-text archives, rsyslog routing rules are the standard approach on production Linux servers.

Build a logging stack you can trust under pressure

Linux logging with journald and rsyslog is not legacy versus modern—it is two layers of the same observability story. journald gives you speed and structure on the box where the failure happened. rsyslog gives you durable text, rotation, and a path to centralized analysis. Configure both deliberately, test forwarding after every major OS upgrade, and document which unit names map to your application services.

If you want help hardening syslog forwarding, disk limits, and monitoring on production Ubuntu servers, review our Linux system administration services or contact us with your current journald.conf and rsyslog snippets. Bring a recent incident timestamp—we can show you exactly where the missing line should have appeared.