Devanagari Unicode Handling in PHP

By Kokil Thapa | Last reviewed: August 2026

Broken characters, truncated words, and failed searches are the immediate symptoms when Devanagari Unicode handling in PHP is misconfigured. For developers building legal-tech portals, government systems, or eCommerce sites for the Nepal market, treating Nepali text as standard ASCII is a critical failure point that breaks both functionality and user trust. Correct implementation requires aligning your PHP string functions, database collation, and HTTP headers to fully support multi-byte UTF-8 sequences from end to end.

If you are building a Laravel application for Nepali users, many of these configurations are abstracted by the framework, but understanding the underlying mechanics remains essential for debugging edge cases in validation, searching, and PDF generation. The difference between a working system and a broken one often lies in a single missing configuration flag or an incorrect function call.

Why does Devanagari Unicode handling in PHP break with standard functions?

The root cause of most Devanagari issues in PHP is the mismatch between byte length and character count. Standard PHP string functions like strlen(), substr(), and strpos() operate on bytes, not characters. A single Devanagari syllable (akshar) can consist of multiple Unicode code points—a base consonant, a vowel sign, a virama, and a nukta—which may occupy 3 to 12 bytes in UTF-8 encoding.

When you use substr($nepaliString, 0, 10) on Devanagari text, PHP counts 10 bytes, which might land in the middle of a conjunct consonant or vowel sign. The result is a malformed sequence that renders as a replacement character () or disappears entirely. This is particularly destructive in legal-tech applications where truncating a case title or party name incorrectly could have professional consequences.

In my experience maintaining Nepal-focused legal portals, the most common production bugs stem from legacy code using strlen() for validation. A field validated as "max 100 characters" using byte-length actually allows only ~33 Devanagari characters, causing silent data loss or unexpected validation failures. Always audit older codebases for bare string functions when adding Nepali language support.

How do you configure MySQL and PHP for correct Devanagari storage?

Correct Devanagari Unicode handling in PHP extends beyond application code into the persistence layer. The database must store, sort, and search Devanagari text correctly, which requires specific collation settings that older tutorials frequently get wrong.

Database Collation Requirements

You must use utf8mb4 charset, never the legacy utf8 alias. MySQL's utf8 is a proprietary 3-byte encoding that cannot store 4-byte Unicode characters, including some Devanagari Vedic extensions and emoji. For new projects in 2026, use utf8mb4_unicode_ci or utf8mb4_0900_ai_ci (MySQL 8.0+) for linguistically accurate sorting of Nepali text.

-- Correct table creation for Devanagari content
CREATE TABLE legal_documents (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    title VARCHAR(500) NOT NULL,
    body LONGTEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB 
  DEFAULT CHARSET=utf8mb4 
  COLLATE=utf8mb4_unicode_ci;

For existing databases using the wrong charset, migrate carefully. Converting columns from latin1 or utf8 to utf8mb4 requires verifying that no data will be truncated, as utf8mb4 uses up to 4 bytes per character versus 1–3 for the older encodings. Always back up before running ALTER TABLE ... CONVERT TO CHARACTER SET utf8mb4.

PHP Database Connection Configuration

The connection itself must declare UTF-8. Even with correct table collation, if the connection charset defaults to latin1, MySQL will silently transcode incoming Devanagari text, corrupting it before storage. In Laravel 12.x, this is handled automatically when DB_CHARSET=utf8mb4 is set in .env. For raw PDO connections:

$pdo = new PDO(
    'mysql:host=localhost;dbname=legal_db;charset=utf8mb4',
    $username,
    $password,
    [
        PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
        PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci"
    ]
);

I've encountered this issue repeatedly on shared hosting environments in Nepal where the default MySQL client charset is overridden at the server level. Explicitly setting SET NAMES in the init command provides a safety net regardless of server defaults. When troubleshooting garbled Nepali text, always check SHOW VARIABLES LIKE 'character_set%' immediately after connecting to verify the active session charset.

Which mbstring functions replace native PHP string operations for Nepali text?

Every native string function has a multibyte-safe equivalent. Using the wrong one is the single most frequent cause of corrupted Devanagari output in PHP applications. Below is the definitive mapping for production code.

Always pass 'UTF-8' explicitly as the encoding parameter. Relying on mb_internal_encoding() is fragile in shared-hosting or multi-application environments where ini settings may differ. Being explicit prevents subtle bugs when deploying across different server configurations.

Regex Patterns for Devanagari Validation

Validating Nepali input requires Unicode-aware regular expressions. The Devanagari block occupies U+0900 to U+097F. Use the /u modifier to enable UTF-8 mode in PCRE, and reference Unicode blocks with \x{XXXX} syntax:

// Validate pure Devanagari text (Nepali/Hindi/Sanskrit)
$pattern = '/^[\x{0900}-\x{097F}\s\.\,\!\?]+$/u';

if (!preg_match($pattern, $input)) {
    // Contains non-Devanagari characters
}

// Extract Devanagari words from mixed-language text
preg_match_all('/[\x{0900}-\x{097F}]+/u', $mixedText, $matches);
$nepaliWords = $matches[0];

A common mistake is forgetting the /u modifier. Without it, PCRE treats the pattern as a byte sequence, and ranges like \x{0900}-\x{097F} fail silently or match incorrectly. In Laravel Form Requests, apply this in custom validation rules rather than relying on alpha or regex rules that assume Latin scripts.

How do you handle Devanagari sorting, searching, and transliteration?

Sorting Nepali text alphabetically is non-trivial because Devanagari sorting follows phonetic order (vowels first, then consonants grouped by articulation point), not Unicode code-point order. Code-point sorting places vowels and consonants in arbitrary sequence relative to traditional dictionary order.

ICU-Based Sorting with Intl Extension

The PHP intl extension provides locale-aware collation via ICU. This is the only reliable method for sorting Nepali text correctly:

$collator = new Collator('ne_NP');
$words = ['कख', 'अआ', 'गघ', 'चछ'];

$collator->sort($words);
// Result: ['अआ', 'कख', 'गघ', 'चछ'] — correct Nepali alphabetical order

// For case-insensitive comparison in search
$strength = $collator->setStrength(Collator::PRIMARY);
$result = $collator->compare('नमस्ते', 'नमASTE'); // Ignores case/diacritics

Ensure the intl extension is installed and enabled. On Ubuntu 22.04/24.04 servers, install it with sudo apt install php8.4-intl (adjust version as needed). Verify with php -m | grep intl. Many shared hosting providers in Nepal disable this extension by default; confirm availability before relying on it.

Full-Text Search Considerations

MySQL's native FULLTEXT index with utf8mb4 supports Devanagari tokenization starting from MySQL 8.0, but results vary depending on the parser. For production search over Nepali legal documents or product catalogs, I recommend offloading to Meilisearch or Elasticsearch with an Indic language analyzer. These engines understand Devanagari word boundaries and stemming far better than MySQL's built-in parser.

If you must use MySQL full-text search, set the minimum word length appropriately. Devanagari words are often shorter in character count than their English equivalents, and the default innodb_ft_min_token_size = 3 may exclude valid two-character Nepali words. Configure this in your MySQL config and rebuild indexes after changes.

Transliteration Between Scripts

Legal and government systems sometimes require Romanized versions of Nepali names for passports, visas, or international correspondence. The intl extension's Transliterator class handles this reliably:

$transliterator = Transliterator::create('Devanagari-Latin; Any-Lower');
$romanized = $transliterator->transliterate('काठमाडौं');
// Output: "kāṭhmāḍauṁ"

// For ASCII-only output (passport-style)
$asciiTrans = Transliterator::create('Devanagari-Latin; Latin-ASCII');
$ascii = $asciiTrans->transliterate('काठमाडौं');
// Output: "kathmandau"

Note that transliteration rules vary between standards (IAST, ISO 15919, ALA-LC). The ICU library uses a reasonable default, but for official government integrations, verify the expected romanization scheme matches your output. I've had to implement custom transliteration maps for specific Nepal government APIs that use non-standard conventions.

What are the common pitfalls in Devanagari PDF generation and API responses?

Even with perfect database storage and string handling, Devanagari text frequently breaks at the output boundary—PDFs, JSON APIs, and HTML rendering each present distinct challenges.

PDF Generation with Devanagari Fonts

Standard PDF libraries like TCPDF and Dompdf do not ship with Devanagari fonts, and embedding them requires careful subsetting to avoid bloated file sizes. For Laravel applications generating legal documents, I recommend using mpdf which has superior Indic script shaping support:

// composer require mpdf/mpdf
$mpdf = new \Mpdf\Mpdf([
    'fontDir' => storage_path('fonts'),
    'fontdata' => [
        'noto-sans-devanagari' => [
            'R'  => 'NotoSansDevanagari-Regular.ttf',
            'B'  => 'NotoSansDevanagari-Bold.ttf',
        ]
    ],
    'default_font' => 'noto-sans-devanagari'
]);

$mpdf->WriteHTML('<p>यो एउटा कानुनी दस्तावेज हो।</p>');
$mpdf->Output('legal_document.pdf', 'I');

Download Noto Sans Devanagari from Google Fonts and place TTF files in your storage directory. Avoid OTF/CFF fonts; most PHP PDF libraries only support TrueType outlines. Test conjunct rendering thoroughly—some font-library combinations fail to properly shape complex akshars like "क्ष" or "ज्ञ".

JSON API Response Headers

When returning Devanagari text via REST APIs, ensure your response declares UTF-8. Laravel handles this automatically with response()->json(), but if you're streaming responses or using a custom framework, set the header explicitly:

header('Content-Type: application/json; charset=UTF-8');
echo json_encode($data, JSON_UNESCAPED_UNICODE);

The JSON_UNESCAPED_UNICODE flag is critical. Without it, json_encode() escapes all non-ASCII characters as \u09XX sequences. While technically valid JSON, this makes debugging difficult and increases payload size significantly for Devanagari-heavy responses. Modern clients handle unescaped UTF-8 JSON without issue.

For teams working on Laravel API development, remember that API resources and Eloquent serialization handle UTF-8 correctly out of the box in Laravel 12.x. Issues typically arise only when manually constructing JSON responses or interfacing with legacy systems that expect escaped Unicode.

How do you test and validate Devanagari Unicode handling end-to-end?

Testing Devanagari support requires deliberate test cases covering edge conditions that Latin-only tests never exercise. Add these to your automated test suite:

• Conjunct consonants: Test strings containing क्ष, त्र, ज्ञ, श्र to verify shaping survives round-trips through database and API.
• Vowel signs on complex bases: Characters like की, कू, कै combine base + matra; ensure mb_substr doesn't split them.
• Mixed-script strings: "Price: रु ५,००० (five thousand)" combines Devanagari digits, currency symbol, and Latin text.
• Empty and whitespace-only Devanagari: Zero-width joiners and non-breaking spaces used in formatting can create visually empty but non-empty strings.
• Maximum-length boundaries: If a field allows 255 characters, test with exactly 255 Devanagari characters (not bytes) to confirm validation uses mb_strlen.

Store canonical test fixtures in UTF-8 encoded files, never inline in PHP source unless you're certain of your editor's encoding settings. I keep Devanagari test data in separate JSON fixtures loaded during PHPUnit runs to avoid accidental encoding corruption from IDE auto-save behaviors.

For visual regression testing of rendered output, use headless browser screenshots compared against baseline images. Devanagari rendering varies across operating systems and font versions; what looks correct on your macOS development machine may render differently on the Ubuntu production server. Include font installation verification in your deployment checklist—if the server lacks a Devanagari font, PDFs and server-side image generation will produce blank boxes.

When integrating third-party services like SMS gateways or payment processors popular in Nepal (eSewa, Khalti, IME Pay), test their API responses for encoding. Some return Devanagari in UTF-8, others in escaped Unicode, and a few legacy systems still use Preeti or Kantipur font-based encodings that require conversion tables. Never assume external APIs follow modern Unicode standards; validate empirically and build adapters as needed. For deeper integration patterns, see the guide on Laravel payment gateway integration which covers encoding considerations specific to Nepali fintech APIs.

Implementing Reliable Devanagari Unicode Handling in PHP

Correct Devanagari Unicode handling in PHP is not a single configuration change but a discipline applied across every layer of your stack. Use mb_* functions exclusively for string operations, enforce utf8mb4 collation at the database level, leverage the intl extension for sorting and transliteration, embed proper Devanagari fonts for PDF output, and validate with purpose-built test cases covering conjuncts and mixed scripts. Each layer must be verified independently because failures are silent and cumulative.

If you're building or maintaining a Nepal-focused web application and need hands-on help resolving Devanagari encoding issues, auditing your Unicode pipeline, or implementing Nepali language support in an existing system, reach out to discuss your project. I've debugged these exact problems across legal-tech portals, eCommerce platforms, and government-facing applications, and can help you get it right the first time.