Build a Data Lake on S3

By Kokil Thapa | Last reviewed: August 2026

If you need to store structured, semi-structured, and raw logs in one place without managing a proprietary warehouse, the most practical approach is to build a data lake on S3. Unlike traditional databases that force schema-on-write, an S3-based lake decouples storage from compute, letting you ingest cheaply now and apply structure later via Spark, Athena, or Glue. In my experience working on production systems that handle everything from eCommerce transaction logs to legal-tech document metadata, getting the folder hierarchy and lifecycle rules right at the start prevents costly rewrites and massive egress bills down the road.

How do you architect a scalable S3 data lake layout?

The single biggest mistake engineers make when they first migrate from shared hosting or monolithic databases to object storage is treating S3 like a filesystem. It is not. S3 is a key-value store where the "key" happens to look like a path. When you build a data lake on S3, your partitioning strategy dictates query performance and cost more than any other factor.

Hive-style partitioning is non-negotiable

Always use Hive-compatible partitioning (key=value/). This allows query engines like Athena, Trino, and Spark to perform partition pruning automatically. If you query for WHERE date = '2026-08-22', the engine reads only the relevant prefix instead of scanning terabytes of unrelated objects.

# Recommended production layout for multi-source ingestion
s3://my-datalake-prod/
├── raw/                    # Immutable landing zone (bronze)
│   ├── ecommerce-orders/
│   │   └── year=2026/month=08/day=22/
│   │       └── orders_20260822T10.parquet
│   ├── legal-documents/
│   │   └── year=2026/month=08/day=22/
│   │       └── docs_batch_001.json.gz
│   └── application-logs/
│       └── year=2026/month=08/day=22/hour=14/
│           └── app-log-part-000.snappy.parquet
├── curated/                # Transformed business-ready data (silver/gold)
│   ├── daily-revenue/
│   └── client-intake-stats/
└── temp/                   # ETL scratch space (auto-delete after 7 days)

Avoid the small files problem

S3 charges per request and has throughput limits per prefix. Writing thousands of 10KB files per minute will destroy your budget and throttle ingestion. On a real client project involving high-volume application logs, we consolidated writes using Kinesis Firehose buffering (set to 128MB or 900 seconds) before flushing to S3. Target file sizes between 128MB and 1GB for Parquet. If your source produces tiny files, use a compaction job (AWS Glue or Spark) to merge them hourly into the curated layer.

What storage classes and lifecycle rules minimize costs?

When you build a data lake on S3, storage cost optimization is an architectural concern, not an afterthought. S3 Intelligent-Tiering should be your default for the raw zone unless you have predictable access patterns. For most production lakes I've configured, the following lifecycle policy balances accessibility with cost.

Implement lifecycle transitions via Terraform or CloudFormation

Never set lifecycle rules manually in the console for production infrastructure. Infrastructure-as-code ensures your cost controls survive accidental deletions and are reviewable in pull requests.

# Example lifecycle rule for raw zone (Terraform HCL)
resource "aws_s3_bucket_lifecycle_configuration" "datalake_raw" {
  bucket = aws_s3_bucket.datalake_raw.id

  rule {
    id     = "tier-raw-to-glacier"
    status = "Enabled"

    filter {
      prefix = "raw/"
    }

    transition {
      days          = 90
      storage_class = "GLACIER_IR"
    }

    transition {
      days          = 365
      storage_class = "DEEP_ARCHIVE"
    }

    expiration {
      days = 2555  # 7 years for legal compliance
    }
  }

  rule {
    id     = "cleanup-temp"
    status = "Enabled"

    filter {
      prefix = "temp/"
    }

    expiration {
      days = 7
    }
  }
}

For Nepal-based businesses dealing with NPR-denominated transactions and local regulatory requirements, remember that Deep Archive retrieval can take up to 48 hours. If your legal-tech portal needs same-day access to historical case documents for court deadlines, keep those specific prefixes in Glacier Instant Retrieval instead. The price difference (~Rs 0.50 vs ~Rs 0.12 per GB/month) is negligible compared to missing a filing deadline.

How do you secure sensitive data in an S3 data lake?

Security cannot be bolted on after ingestion. When I work on legal-tech portals handling marriage certificates, divorce filings, or notarized documents, every object must be encrypted at rest and access-controlled at the prefix level. Here is the baseline security posture for any production data lake.

• Server-Side Encryption (SSE-S3): Enable AES-256 encryption by default via bucket policy. As of 2026, S3 encrypts all new objects automatically, but explicit bucket policies prevent misconfiguration during cross-account replication.
• Block Public Access: Apply s3:BlockPublicAcls and s3:BlockPublicPolicy at the account level via S3 Block Public Access settings. No data lake bucket should ever have public read ACLs.
• Prefix-Level IAM Policies: Grant ETL jobs access only to their specific source/target prefixes, never *. Use resource tags for dynamic permission boundaries.
• VPC Endpoints: Route all S3 traffic through gateway endpoints to avoid NAT gateway charges and internet exposure. This alone saved one client ~Rs 15,000/month in egress fees.

# Bucket policy enforcing encryption and VPC-only access
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-datalake-prod/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    },
    {
      "Sid": "DenyNonVPCAccess",
      "Effect": "Deny",
      "Principal": "*",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-datalake-prod",
        "arn:aws:s3:::my-datalake-prod/*"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpce": "vpce-0abc123def456789"
        }
      }
    }
  ]
}

How do you integrate AWS Glue Catalog for SQL queries?

Raw files in S3 are useless to analysts without metadata. The AWS Glue Data Catalog acts as a centralized metastore, letting you query S3 data with standard SQL via Athena without loading it into a separate database. This is what makes the phrase "build a data lake on S3" mean something beyond cheap storage.

Register partitions automatically with crawlers or ETL

For production workloads, avoid scheduled crawlers. They are slow, expensive, and often infer schemas incorrectly when data drifts. Instead, update the Glue Catalog directly from your ETL jobs using the boto3 Glue client or Spark's built-in catalog integration.

# Python: Register new partition after ETL write
import boto3
from datetime import datetime

glue = boto3.client('glue', region_name='ap-south-1')

today = datetime.utcnow().strftime('%Y/%m/%d')
partition_values = today.split('/')

glue.batch_create_partition(
    DatabaseName='datalake_curated',
    TableName='daily_revenue',
    PartitionInputList=[{
        'Values': partition_values,
        'StorageDescriptor': {
            'Location': f's3://my-datalake-prod/curated/daily-revenue/year={partition_values[0]}/month={partition_values[1]}/day={partition_values[2]}/',
            'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
            'OutputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat',
            'SerdeInfo': {'SerializationLibrary': 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'},
            'Columns': [
                {'Name': 'order_id', 'Type': 'string'},
                {'Name': 'revenue_npr', 'Type': 'decimal(12,2)'},
                {'Name': 'currency', 'Type': 'string'},
                {'Name': 'created_at', 'Type': 'timestamp'}
            ]
        }
    }]
)

What common mistakes derail S3 data lake projects?

Having debugged several failed data lake initiatives, these anti-patterns appear repeatedly. Avoiding them saves weeks of rework and significant operational overhead.

1. No data quality gates: Dumping malformed JSON into the raw zone without validation corrupts downstream tables. Implement schema enforcement at ingestion (Kinesis Firehose with Lambda transformation, or Glue ETL with schema registry). Reject bad records to a quarantine prefix for investigation.
2. Ignoring GDPR/local privacy laws: Even in Nepal, if you process EU citizen data or plan international expansion, PII must be identifiable and deletable. Use column-level encryption or tokenization for sensitive fields. Never store raw PII in the curated layer without explicit business justification.
3. Over-partitioning: Creating partitions by hour for low-volume sources generates millions of tiny objects and slows catalog operations. Only partition by dimensions that reduce scan size by at least 10x. For sparse data, consider bucketing within partitions instead.
4. Missing monitoring: Without CloudWatch metrics on bucket size, request counts, and 4xx/5xx errors, you won't notice runaway costs until the bill arrives. Set up S3 Storage Lens and billing alarms at 50%, 80%, and 100% of budget threshold.

For teams evaluating whether a full data lake is necessary versus a simpler solution, reviewing database-driven development patterns can clarify when PostgreSQL or MySQL suffices versus when you truly need decoupled storage and compute. Not every analytics requirement justifies S3 complexity.

Build a Data Lake on S3: Next Steps

When you build a data lake on S3 correctly, you gain a foundation that scales from gigabytes to petabytes without architectural changes. Start with the three-tier layout, enforce encryption and VPC access from day one, automate lifecycle transitions, and integrate Glue Catalog early so your data is queryable immediately. Monitor costs aggressively and validate schemas before they pollute downstream analytics. If you're planning a data lake implementation and need hands-on architecture guidance tailored to your workload, reach out to discuss your specific requirements.