DATVERSE
Start a project

Service · data-engineering

Data Engineering & Analytics

Pipelines, warehouses, and dashboards that don't break on Monday morning.

What we do

We build the data infrastructure under your AI features and your reporting: ingestion, transformation, modelling, observability. Snowflake / BigQuery / Postgres + dbt, with tests in CI so the dashboard doesn't show -38% revenue at 9am because someone changed a column.

Why us

Most data pipelines work the day they ship and break two weeks later when an upstream API shape shifts. We build with tests, schema contracts, and dbt freshness checks from day one, so when something breaks you find out before stakeholders do.

  • Snowflake
  • BigQuery
  • PostgreSQL + TimescaleDB
  • dbt
  • Airflow
  • Dagster
  • Fivetran
  • Airbyte
  • Metabase
  • Looker

What you get

  • ETL / ELT pipelines from your operational sources into a warehouse (Postgres, Snowflake, BigQuery) with dbt models on top.
  • Data contracts between producers and consumers so a column rename in the source database doesn’t silently break the CFO’s dashboard.
  • Reverse ETL when the BI workflow needs to push back into Salesforce / HubSpot / a customer-facing app.
  • Streaming pipelines (Kafka / Kinesis / Redpanda + Materialize / Flink) when batch isn’t enough.
  • Observability: dbt freshness, Great Expectations or Soda checks, on-call alerts on broken pipelines.

What we don’t do

  • We don’t replace your BI tool unless we have to. If you’re already on Metabase or Looker, we make it better.
  • We don’t ship pipelines without tests. If your existing setup has none, that’s the first thing we fix.
  • We don’t quote a number without seeing the data. The discovery call is free; the proposal isn’t a guess.

Have a project that fits?

Send a short brief. We'll reply within one business day.

Start a project

04 Selected work

Production systems, not demos.

enterprise · 2025 · a US enterprise SaaS company

Custom LLMs for a domain knowledge base

Fine-tuned and RAG-augmented LLM stack for an internal knowledge base. Answers stayed grounded in source documents instead of hallucinating across the corpus.

Custom LLMs for a domain knowledge base — case study
  • 92% citation-correct answers on a 200-question internal eval client QA team, 30-day window
  • ~30% weekly support ticket deflection by month two client support lead

Context

A mid-sized SaaS company with a sprawling internal knowledge base (Confluence, Notion, Slack archives, dozens of PDFs) wanted a single answer interface for new hires and support engineers. Off-the-shelf “chat with your docs” tools had degraded into hallucinations once the corpus grew past ~5,000 documents.

Problem

The two failure modes the client wanted gone: (1) confidently-wrong answers that referenced documents that didn’t exist, and (2) “I don’t know” responses on questions whose answer was in the corpus but in a format the retriever couldn’t see (tables, screenshots, code).

Approach

The strategy that won: smaller fine-tune for tone + structure, large pretrained model for reasoning, RAG for grounding, with cite-or-refuse as the core rule. If the retriever didn’t surface a confident hit, the model was instructed to say so and link the closest match for a human to follow up. No graceful hallucinations.

Solution

  • Document ingestion pipeline: chunked at semantic boundaries (not fixed token windows), embedded with two complementary models, indexed in Pinecone.
  • A reranker pass on the top-50 retrieval to recover answers buried in tables and code blocks.
  • Lightweight fine-tune on the company’s voice, primarily for first-person plural framing and avoiding marketing language.
  • Citation enforcement: every answer carries a “Sources” section with direct links to the source documents and section headings.
  • Eval harness with a curated 200-question set ran on every prompt or retriever change.

Outcome

Above (in frontmatter). The team kept the eval harness and now uses it to gate every prompt or model change. That’s the part the client found kept coming back to in retrospect.

fintech · 2024 · DFIN (dfinsolutions.com)

DFIN AI Services — financial-document AI for regulatory workflows

Backend AI services for financial-filing ingestion, parsing, summarization, comparison, and embeddings-based retrieval, built for a regulatory-compliance software company.

DFIN AI Services — financial-document AI for regulatory workflows — case study
  • Reduced manual review effort across financial and regulatory document workflows client-reported

Context

DFIN is a financial-compliance and regulatory-software company serving public companies, private companies, investment companies, law firms, and financial advisors, all of which work with complex reporting and compliance documents at scale. The DFIN AI Services engagement focused on document ingestion, parsing, summarization, comparison, and AI-assisted analysis for financial filings.

Problem

Financial and regulatory documents are long, structured-yet-messy, and full of cross-references that humans skim and miss. The compliance teams reading them need accurate summarization and side-by-side comparison without the cost of full manual review on every filing, and without the model inventing content that isn’t in the source filing.

Approach

A Flask-backed service tier with two distinct flows (one for document processing, one for request handling) sitting in front of an LLM summarization and comparison pipeline. Embeddings and vector search underneath, hosted on Azure, with cite-or-refuse posture on extractive answers so the AI surfaces grounded summaries rather than generative paraphrase.

Solution

  • Document upload and ingestion flow with file parsing and content extraction; structured and unstructured document content handled in the same pipeline.
  • Two Flask service flows (a document-processor service and a request-processor service) coordinating the AI work.
  • LLM-based document summarization and document-to-document comparison.
  • Embedding generation and Azure-based storage and retrieval supporting semantic search and retrieval-augmented analysis.
  • Text-chunking and embedding pipeline tuned for the long-document, cross-reference-heavy shape of financial filings.

Outcome

Above. The work combined backend engineering, document processing, cloud integration, and AI services in a regulated financial environment, supporting AI-assisted review of financial filings and compliance documents.

manufacturing · 2024 · a Pakistan-based manufacturer

Computer vision QC for a manufacturing line

Edge-deployed defect-detection model running on a moving production line, replacing a manual visual inspection step that had a 6% miss rate.

Computer vision QC for a manufacturing line — case study
  • 6% → 1.4% defect-miss rate (manual baseline vs. production model after week 4) client QA log, 30-day window
  • ~150ms end-to-end inspection latency at the edge internal benchmark

Context

A manufacturer running a single-shift QC station with a senior inspector, which meant when she was on leave, defective product shipped. Goal: a camera + edge box that flagged the same defect classes she did, live on the line.

Problem

Off-the-shelf industrial CV is expensive and the SKU-set was a moving target. The client also didn’t want a cloud round-trip: the line was in a region with intermittent connectivity, and the rejection signal had to fire within ~150ms of a part hitting the inspection zone.

Approach

Build small. A YOLO-family detector fine-tuned per defect class beat any “general industrial” model the client had trialled, because defect patterns are highly product-specific. Run it on a Jetson box at the inspection station; only sync the labelled imagery and a daily metrics summary to the cloud.

Solution

  • Labelled the first 4,000 frames by hand alongside the inspector, paying attention to the edge cases she flagged that an off-the-shelf model would miss.
  • Fine-tuned a YOLOv8 variant; converted to ONNX for Jetson inference.
  • Built a thin FastAPI service that took camera frames, ran inference, and published rejection signals over MQTT to the line’s existing rejector arm.
  • Daily nightly retrain trigger: when the inspector overrides the model 3+ times in a shift, the new images get added to the training set and the model rebuilds at midnight.

Outcome

Above. The “accept the inspector’s overrides as ground truth” loop is what kept accuracy climbing instead of plateauing.

healthcare · 2024 · On Target Clinical (ontargetconnect.com/clinical)

On Target Clinical — integrated EHR, billing, and payroll for behavioral health

Multi-module behavioral-health platform (EHR, billing, accounting, payroll, and electronic forms) connecting clinical, finance, and administrative teams in one system.

On Target Clinical — integrated EHR, billing, and payroll for behavioral health — case study
  • 5 modules (Billing, Payroll, Accounting, Webforms, EHR) shipped in one system scope record
  • Live in production at ontargetconnect.com/clinical since 2024 public site

Context

On Target Clinical is a behavioral-health operations platform: the clinical, accounting, and administrative teams of a behavioral-health practice working from a single system rather than five disconnected tools. The engagement covered the full module set: EHR, billing, payroll, accounting, and a webforms/e-signature flow.

Problem

The platform’s value lives at the seams. Billing has to reconcile against documented clinical work; payroll has to read timesheets that came from actual delivered care; accounting has to ledger both. Each module on its own is solvable; the integration is where most healthcare-ops products quietly fall apart at month six.

Approach

A microservices architecture with a React front end and a Java/Spring Boot service tier, with MySQL for relational records (billing, payroll ledgers) and MongoDB for documents and forms. The integration contract between modules was the load-bearing design decision: get the shared identifiers and event flow right and the modules compose; get them wrong and every cross-module report becomes a rescue project.

Solution

  • React front end for the clinical, billing, payroll, accounting, and webforms surfaces.
  • Java + Spring Boot service tier, organized as microservices per module.
  • MySQL for transactional billing, payroll, and accounting records; MongoDB for forms, documents, and unstructured clinical notes.
  • Laravel + PHP components for ancillary surfaces and a desktop application for in-clinic workflows.
  • ExpressJS endpoints where lightweight Node services fit the integration pattern.
  • Compliance posture aligned to behavioral-health data handling: admin team and clinicians configured the data structuring jointly with the engineering team.

Outcome

Above. The platform is live at ontargetconnect.com/clinical. The On Target product team advertises 100% billing collection rates as a product outcome. That figure is a client product claim, not an engineering metric, and is the operating success the integrated billing + clinical-delivery contract is built to support.

marketplace · 2024 · CLEAQUES (cleaques.com)

CLEAQUES — diaspora marketplace for African producers

Laravel + Blade marketplace connecting African producers and service providers to the African diaspora across the US and Canada.

CLEAQUES — diaspora marketplace for African producers — case study
  • Live in production at cleaques.com since 2024 public site

Context

CLEAQUES set out to connect African producers and service providers to African expats in the US and Canada: a diaspora marketplace bridging community, cultural commerce, and small-business support. The build needed to handle multi-currency listings, vendor onboarding, and a buyer experience that respected both sides of the Atlantic.

Problem

A diaspora marketplace lives or dies on trust: vendors need a clean store experience, buyers need confidence in the listings, and the platform owner needs admin controls without fighting the framework. Off-the-shelf marketplace plugins didn’t fit the cultural-commerce shape of the catalogue.

Approach

A Laravel + Blade build with custom theming on top of a marketplace data model designed for the diaspora use case. Server-rendered templates over a heavier SPA, because the audience reach matters more than client-side interactivity.

Solution

  • Laravel application with Blade templating for the storefront and admin surfaces.
  • Vendor onboarding and product-listing flows with admin moderation.
  • Theme customization layered on a base marketplace shell: product cards, category navigation, and storefront pages tuned for the diaspora catalogue.
  • Currency and locale handling for cross-border transactions.

Outcome

Above. The platform launched and is live at cleaques.com; the producer-to-diaspora connection is the load-bearing user story and the storefront is where most of the engineering investment landed.

logistics · 2024 · a MENA logistics company

Digital transformation for a regional logistics operator

Replaced a six-spreadsheet ops workflow with a multi-tenant web platform: dispatch, driver app, customer tracking, billing, all one source of truth.

Digital transformation for a regional logistics operator — case study
  • 11min → 90s dispatch time per shipment client ops lead
  • 5d → same-day billing reconciliation cycle client finance
  • 2 regions added by client without engineering involvement (post-handoff, 6 months) client ops lead

Context

A regional logistics operator with ~40 drivers, two dispatchers, and a fleet of spreadsheets that no two people read the same way. They had attempted two prior software builds: one died at scope, the other shipped but couldn’t handle peak load on Mondays.

Problem

The honest story was that the spreadsheets weren’t the problem; the workflow was. Each previous build had digitised the spreadsheet without questioning the underlying ops, and the result was a slower spreadsheet.

Approach

The team mapped the goal, not the feature list, then proposed cutting two of the six spreadsheets entirely (the duplicates) and merging the customer-facing one into the driver app via a shared shipment record. Lean scope, signed off in week one, no surprises at week six.

Solution

  • Multi-tenant Next.js admin panel for dispatch + a React Native driver app sharing a single Postgres source-of-truth.
  • Stripe Connect for marketplace billing: drivers paid out automatically against signed off-deliveries.
  • Live tracking page customers received as a per-shipment magic link (no account needed).
  • Real-time updates over Supabase Realtime, no custom websocket layer.
  • Clean handoff with docs, CI on Vercel + GitHub Actions, and a 30-day support window.

Outcome

Above. Six months later the company added two regions to the platform without engineering involvement. That was the proof the underlying schema was right.

transportation · 2024 · ATL Worldwide Limousine (atlworldwidelimousine.com)

ATL Worldwide Limousine — Atlanta airport and corporate transport site

Custom HTML / PHP / jQuery build (no WordPress, no template) for an Atlanta-based ground transportation company. Still on retainer.

ATL Worldwide Limousine — Atlanta airport and corporate transport site — case study
  • Live in production at atlworldwidelimousine.com since 2024 public site
  • 4 vehicle classes (Sedan, SUV, Premium SUV, Sprinter) with per-class pricing live site
  • Ongoing retainer for maintenance and fleet/pricing updates since launch engagement record

Context

ATL Worldwide Limousine runs ground transportation out of Hartsfield-Jackson Atlanta International Airport, with airport transfers as the primary service, alongside point-to-point, corporate, and charter work. 24/7 operation. The site had to make booking obvious from any entry point.

Problem

The fleet-and-pricing problem is genuine: a limo site that hides vehicle classes behind a contact form loses the buyer, and a site that exposes pricing without context loses the brand. The fleet showcase had to read as a service catalogue with confidence-cues (capacity, minimum hours, hourly rates) without becoming an algorithmic-looking marketplace.

Approach

A fully custom HTML / CSS / PHP / jQuery build: no WordPress, no page builder, no theme template. Every page hand-written. The decision was deliberate: the brief was tight enough that a CMS would have added more overhead than value, and the long-term retainer relationship meant the team would own ongoing maintenance directly.

Solution

  • Custom HTML / CSS / PHP / jQuery front end, no WordPress installation.
  • Multi-slide hero carousel rotating value propositions (family safety, corporate transport, airport transfers).
  • Fleet showcase with vehicle cards: Cadillac XTS (sedan, 3 pax, $100/hr), Chevrolet Suburban (SUV, 6 pax, $125/hr), Cadillac Escalade (premium SUV, 6 pax, $140/hr), Mercedes Sprinter (14 pax, $250/hr), each with capacity, minimum-hour rules, and a Book Now CTA.
  • Service pages for Airport Limousine, Point-to-Point, Corporate Transfers, and Charter Services.
  • City/area selection for Atlanta and Alpharetta.
  • Testimonials carousel, FAQ section, and contact surfaces (email, phone, airport address) surfaced site-wide.
  • Mobile-responsive layout with hand-tuned breakpoints.

Outcome

Above. Live at atlworldwidelimousine.com, continuing on retainer for content updates, fleet and pricing changes, and any new feature work.

healthcare · 2025 · a US healthcare provider

Healthcare patient-intake chatbot with safe-handoff

Conversational intake that gathered symptoms, insurance, and consent before a clinician saw the patient, with an explicit guardrail that escalated anything outside its scope to a human.

Healthcare patient-intake chatbot with safe-handoff — case study
  • 14min → 6min patient-intake completion time client clinical-ops lead
  • 100% safe-handoff fired on every red-flag case (200-conversation eval) joint eval with the clinical team
  • 87% clinician notes ready before the appointment client clinical-ops lead

Context

A multi-clinic provider whose intake desk was the throughput bottleneck, and whose previous chatbot trial had been pulled because it tried to give triage advice. They wanted intake automation, not medical advice.

Problem

The risk wasn’t the happy path. The risk was the model “being helpful” on a question it had no business answering: symptom severity, dosing, “is this an emergency”. One bad answer would close the experiment for good.

Approach

A guardrail-first design: enumerate the conversational moves the bot is allowed to make, and refuse everything outside them. Symptom capture (record what the patient says) is allowed; symptom interpretation is not. Safe-handoff is a first-class action: when in doubt, the bot says “let me get a clinician” and pages on-call.

Solution

  • Claude with a tightly-scoped system prompt + structured tool calls for the allowed moves (capture, schedule, request insurance, escalate).
  • A separate classifier in front of every user turn that ran a red-flag check (chest pain, suicidal ideation, severe bleeding, pregnancy red-flags, paediatric red-flags); it bypassed the model entirely and paged on-call when triggered.
  • Server-side conversation log with explicit consent capture and FHIR-shaped notes for the EHR.
  • HIPAA-aligned hosting with BAAs in place; no patient data in model providers’ retention windows.
  • 200-conversation eval set built with the clinical team, including adversarial prompts designed to extract triage advice.

Outcome

Above. The eval harness is what convinced the compliance team to sign off; they could see the safe-handoff firing on every case it should have.

real-estate · 2024 · Rentagade (therentagadeapp.com)

Rentagade — rental marketplace connecting renters with owners

Laravel + PHP rental marketplace bringing renters and property owners together in one platform.

Rentagade — rental marketplace connecting renters with owners — case study
  • Live in production at therentagadeapp.com since 2024 public site

Context

Rentagade set out to be a single platform for property rentals: renters on one side, owners on the other, with a shared listing and contact surface in the middle. The build had to feel coherent to both audiences.

Problem

A two-sided rental marketplace stands or falls on the renter’s clarity (what’s listed, what does it cost, what happens next) and the owner’s control (post a listing, manage enquiries, respond). Off-the-shelf listing themes hide the renter-side flow behind cards that read as classifieds, not rentals.

Approach

A Laravel + PHP build with a marketplace data model designed for the rental use case. Server-rendered for performance and reach; admin surfaces and listing forms tuned to the property-rental shape rather than a general classifieds template.

Solution

  • Laravel application with relational property and user models.
  • Listing creation flow with owner-side admin controls.
  • Search and discovery surface for renters with property-detail pages.
  • Contact and enquiry flow between the two sides.

Outcome

Above. Live at therentagadeapp.com since 2024.

marketplace · 2024 · JDM Buy/Sell (jdmbuysell.com)

JDM Buy/Sell — marketplace for Japanese Domestic Market vehicles

WordPress + Laravel marketplace for Japanese Domestic Market enthusiasts, connecting buyers with thousands of independent JDM sellers.

JDM Buy/Sell — marketplace for Japanese Domestic Market vehicles — case study
  • Live in production at jdmbuysell.com since 2024 public site
  • Thousands of independent sellers on the marketplace listing side client-reported, public site

Context

JDM Buy/Sell connects Japanese Domestic Market vehicle enthusiasts with independent sellers: a category-specific marketplace where the seller pool sits in the thousands and the buyer is looking for vehicles the standard auto-marketplace stack doesn’t categorize well.

Problem

JDM listings carry their own metadata (year, model variants, import status, chassis codes) that generic auto-marketplace plugins don’t handle natively. A WordPress build had to layer custom plugin and theme work over the marketplace base so the listing shape matched what JDM buyers actually scan for.

Approach

A WordPress base with custom plugin development and theme customization for the JDM-specific listing fields, layered on Laravel components where the marketplace logic warranted a non-WordPress service. Plugin customization carried the bulk of the differentiation work.

Solution

  • WordPress installation as the storefront and content surface.
  • Custom plugin development for JDM-specific listing metadata.
  • Custom theme development with the JDM enthusiast audience in mind.
  • Laravel components where the marketplace logic sat outside WordPress comfort zones.
  • Plugin customization to integrate the existing marketplace plugin set with the custom JDM fields.

Outcome

Above. Live at jdmbuysell.com; the marketplace supports thousands of independent sellers on the listing side.

healthcare · 2024 · Priority Health (credentialing platform)

Priority Health credentialing — medical credentialing and enrollment platform

Full-stack medical credentialing and enrollment platform: clinical professionals and administrators collaborating on structured staff-qualification data.

Priority Health credentialing — medical credentialing and enrollment platform — case study
  • 1 platform: credentialing + enrollment replacing the multi-document, multi-spreadsheet baseline scope record
  • Joint clinical + admin collaborative data structuring with healthcare professionals engagement record

Context

Priority Health specializes in medical credentialing and enrollment: the work behind making sure clinical staff have the qualifications, certifications, and approvals required to deliver patient care. The platform brief was a full-stack web application to carry that workflow.

Problem

Credentialing data has shape that off-the-shelf forms don’t capture well: certifications expire on different cycles, qualifications come from different bodies, enrollments are facility-specific. Without a careful data model designed with clinical staff, the platform becomes another spreadsheet replacement that drifts.

Approach

A full-stack TypeScript build with REST APIs in front of the credentialing data model, designed jointly with healthcare professionals and administrators. Data-security and patient-data-privacy concerns drove architectural decisions throughout: encryption posture, access control, audit, and retention rules built into the data layer, not bolted on after.

Solution

  • Full-stack TypeScript web application with REST APIs.
  • Credentialing and enrollment data model designed jointly with clinical staff and administrators.
  • Data-security and privacy controls integrated at the database and API layers.
  • Web surfaces for credentialing data entry, enrollment workflows, and admin oversight.

Outcome

Above. The collaborative-design loop with the clinical team is the load-bearing process decision: credentialing data without clinical-staff input ends up being a spreadsheet pretending to be a platform.

enterprise · 2026 · a Series A B2B SaaS

RAG refactor: from 64% to 91% answer accuracy in three weeks

An internal Q&A bot that the support team had stopped trusting, rebuilt over three weeks with a new retriever, a cite-or-refuse rule, and an eval set the client still runs against every prompt change.

RAG refactor: from 64% to 91% answer accuracy in three weeks — case study
  • 64% → 91% answer accuracy on a 200-question internal benchmark client QA team, owned end-to-end
  • 3 weeks scope-to-handoff, working from an existing codebase engagement record
  • 0 downstream changes the support team had to make to start trusting it client support lead, 14-day window

Context

A 35-person SaaS had shipped an internal “ask the docs” bot a year earlier, on GPT-3.5 with a basic vector retriever. The support team used it for a quarter, lost trust in it, and went back to grepping Notion. By the time the team called DatVerse, the bot was still in the product but nobody on the support side opened it.

Problem

The retriever was the bug, not the model. A 200-question benchmark the client put together showed 64% answer accuracy; most failures were either confidently-wrong answers where the retriever had missed, or refusals on questions whose answer was sitting in the corpus but in a format the embedder couldn’t see (tables, code blocks, threaded Slack screenshots that had been pasted as PDFs).

Approach

Three-week scope, no greenfield rewrite. The strategy was: keep the chat UI, keep the LLM, keep the database. Replace the retrieval layer with a hybrid BM25 + vector store, add a reranker pass, and bolt a cite-or-refuse rule onto the prompt. Then hand the support team an eval harness they could run themselves.

Solution

  • Hybrid retrieval. Postgres + pgvector for embeddings, alongside a Postgres tsvector BM25 index. Both run in parallel, top-k from each, deduped by source.
  • Reranker. A small cross-encoder run on the top-50 to recover answers buried in tables and code.
  • Cite-or-refuse prompt. Every answer either includes a Sources: section with direct links to the source docs, or refuses and points at the closest near-miss for the support engineer to follow up.
  • Eval harness. The 200-question benchmark the client built moved into a small FastAPI eval runner that fires on every prompt or retriever-config change in CI. Pass/fail diffs are posted to the team’s Slack.

Outcome

The accuracy number is in the headline metrics. The follow-on outcome the client cared more about: two weeks after handoff, the support team had reopened the bot in their daily flow without anyone asking them to. The eval suite is what made that switch credible; they could see exactly which questions had gotten better and which had not.

marketplace · 2024 · Cabo Verde TradeInvest

Cabo Verde TradeInvest — WordPress / WooCommerce marketplace

WordPress + WooCommerce marketplace platform for Cabo Verde TradeInvest, with a standardized buy/sell transaction flow and API integration.

Cabo Verde TradeInvest — WordPress / WooCommerce marketplace — case study
  • 1 standardized transaction flow: consistent buy/sell path across the marketplace scope record

Context

Cabo Verde TradeInvest wanted a marketplace where the buy and sell flows behaved the same way every time (same fields, same confirmation steps, same payment path) so neither side had to learn a new workflow per listing category.

Problem

WordPress + WooCommerce gives you a storefront; it does not give you a marketplace with a standardized cross-category transaction flow. The plugin set has to be picked and customized so listings, payments, and confirmations move through the same path.

Approach

A WordPress base with WooCommerce as the transactional engine, plus custom WordPress plugin and theme work to standardize the buy/sell flow across listing categories. API integration carries data into and out of the marketplace where it interacts with external systems.

Solution

  • WordPress installation with WooCommerce as the e-commerce core.
  • Custom WordPress plugin work for the marketplace-specific transaction flow.
  • Custom WordPress theme work for the listing and storefront surfaces.
  • API integration with JSON payloads where the marketplace exchanges data with external systems.
  • Standardized buy/sell path so each transaction follows the same field set and confirmation sequence.

Outcome

Above. The standardization is the operating value: both sides of the marketplace recognize the same flow on the next transaction.

enterprise · 2026 · a Series B operations-tooling SaaS

Dashboard stability audit — dbt freshness checks and schema contracts for a finance dashboard

A two-week audit of a finance dashboard that kept showing wrong numbers on Mondays, culminating in dbt freshness checks, schema contracts on three brittle upstream tables, and a one-page playbook the analytics team now runs themselves.

Dashboard stability audit — dbt freshness checks and schema contracts for a finance dashboard — case study
  • 0 / 60 days 'the dashboard is wrong' threads after handoff (previous baseline: 3–5/month) client analytics lead, team incident channel
  • 3 upstream tables now under schema contract: the ones that had been silently changing internal audit
  • Weekly close cadence the CFO signed off on after the audit (was monthly) client analytics lead

Context

A Series B SaaS with a healthy analytics function. dbt models in production, a Looker dashboard the CFO opened every Monday morning. The numbers were usually right. They were also wrong often enough (three or four times a month) that the analytics lead had started prefacing every Monday standup with “the dashboard looks weird, give me an hour.” Nobody could predict which Monday it would be.

Problem

Three upstream tables (one from the product backend, one from a billing tool, one from the CRM) were silently shape-shifting. A column would get renamed in the product database during a Friday deploy; dbt would happily ingest the rename as a new column with NULLs for the prior data; the model would compile clean; the dashboard would show -38% revenue on Monday. The analytics team had no way to see this coming because dbt was running on a schedule and the test suite didn’t cover schema drift.

Approach

Two-week scope, no model rewrites. The work was three things: identify which upstreams were the actual sources of the Monday surprises, add freshness and schema-shape checks that would catch the drift at the right layer, and hand the analytics team a one-page playbook for what to do when the checks fire.

Solution

  • Upstream audit. Ran a git-blame across the last 90 days of dbt model failures and analytics-channel incidents to identify the three tables responsible for the recurring Monday drift.
  • dbt freshness + schema-shape tests. Added dbt source freshness checks on the three tables, plus custom schema tests that fail loudly if column names, types, or NOT-NULL constraints change unexpectedly. Failures gate the dbt run so a bad upstream stops the pipeline instead of corrupting the dashboard.
  • Schema contracts. Documented the three upstream contracts in the dbt project README: names, types, allowed-null columns, deprecation policy. The contracts got posted to the product team’s Slack so backend changes to those tables hit a review step.
  • Playbook. One page: what each check covers, what the failure modes look like, who owns each upstream, how to silence a known-good change without disabling the check entirely.

Outcome

The Monday surprises stopped. The analytics lead reported zero “the dashboard is wrong” threads in the 60 days after handoff. More importantly, three upstream changes that would have broken the dashboard pre-audit were caught and resolved at the contract layer before they hit production. The CFO signed off on a move from monthly to weekly close on the back of the audit; the dashboard was finally trustworthy on the cadence the business actually wanted.

transportation · 2024 · Pak Express Limousine (pakexpresslimousine.com)

Pak Express Limousine — Dubai luxury transport booking platform

Custom WordPress + Elementor + WooCommerce build with an integrated taxi-booking engine for a Dubai luxury transport company. Still on retainer.

Pak Express Limousine — Dubai luxury transport booking platform — case study
  • Live in production at pakexpresslimousine.com since 2024 public site
  • Multi-city UAE coverage with Executive, VIP, and Special Event service tiers scope record
  • Ongoing retainer for maintenance, updates, and feature work since launch engagement record

Context

Pak Express Luxury Cars Transport LLC operates luxury ground transportation out of Dubai, with multi-city UAE coverage. The brief was a booking site that matched the “luxury, comfort, elegance” positioning end-to-end, from the first impression through the reservation flow.

Problem

Most off-the-shelf transport templates feel like off-the-shelf transport templates. The booking flow had to fit the brand, the vehicle-class selection had to read as a service catalogue rather than a checkbox grid, and the payment flow had to land as a confirmed reservation rather than a cart abandonment.

Approach

A from-scratch Elementor build on a Hello Elementor parent shell (no theme template), wired into WooCommerce for payments and an eCab Taxi Booking Manager configuration for the reservation flow. Custom header and footer via the Header & Footer Elementor plugin to keep brand control end-to-end.

Solution

  • WordPress 6.x + Hello Elementor parent shell, with every page section custom-built in Elementor 3.x + ElementsKit Lite.
  • WooCommerce 10.x for the payment side; eCab Taxi Booking Manager for vehicle selection, pickup/dropoff scheduling, and pricing tiers.
  • Metform-based contact and quote flows tied to admin notifications.
  • Custom header and footer via Header & Footer Elementor for full brand control.
  • Service and fleet pages for Executive, Special Event, and VIP Transfers, with multi-city UAE coverage.
  • Mobile-responsive tuning across breakpoints; SEO basics, page-speed pass, and schema markup.

Outcome

Above. Live at pakexpresslimousine.com, continuing on retainer for maintenance, content edits, and new feature work since launch.

real-estate · 2024 · a North-American real estate operator

Real estate buy/sell/rental platform with MLS RETS integration

Multi-mode real estate platform (buy, sell, rental) with RETS-standard MLS imports across North America and a back-office automation layer for agents.

Real estate buy/sell/rental platform with MLS RETS integration — case study
  • MLS across North America: RETS-standard listing import pipeline scope record
  • 3 transaction modes (buy, sell, and rental) in one platform scope record

Context

A real estate operator needed a single platform covering buy, sell, and rental flows (not three separate sites stitched together), with listing data flowing in from MLS sources across North America.

Problem

MLS data lives behind the RETS standard, and most agent-facing platforms either don’t speak RETS at all or hand-roll a fragile importer per market. Without a proper RETS pipeline, the listing surface goes stale within weeks of launch.

Approach

A Node.js application with a hybrid MySQL + MongoDB data layer, hosted on AWS, with a RETS-standard import pipeline as a first-class subsystem. A back-office automation layer for agents and sellers sits on top so the platform isn’t just a listing site; it carries the agent’s daily operational workflow.

Solution

  • Node.js application tier; MySQL for transactional records (transactions, agent accounts), MongoDB for listing documents and unstructured data.
  • AWS hosting and infrastructure.
  • RETS-standard MLS import pipeline pulling listings from feeds across North America into the platform database.
  • Back-office automation surfaces for real estate agents and sellers to run their operational workflow inside the platform.
  • Three transaction modes (buy, sell, rental) sharing the listing model rather than duplicating it.

Outcome

Above. The RETS pipeline is the load-bearing infrastructure decision: without it the listings surface becomes stale; with it the platform stays current against MLS sources across the continent.

healthcare · 2024 · a US healthcare technology company

Healthcare-pharma connect — a SaaS bridge between providers and pharma/biotech

HIPAA-aware SaaS connecting healthcare providers with pharmaceutical, biotech, and life-science companies to surface current treatment and device offerings.

Healthcare-pharma connect — a SaaS bridge between providers and pharma/biotech — case study
  • HIPAA aware build: protected health context across the provider-pharma exchange stack record

Context

Healthcare providers and pharmaceutical / biotech / life-science companies need each other (providers to learn what is currently available for their patients, pharma to reach the clinicians who would use it), but most of the existing connection paths run through sales reps and conferences. The platform was a SaaS bridge sitting in between.

Problem

Industry-context data point: estimates suggest more than 90% of physicians are not fully aware of the latest pharma and device offerings relevant to their practice. That gap is not a marketing-attribution problem; it is an information-architecture problem, and a HIPAA-aware one, because the provider-side context carries protected health framing.

Approach

A SaaS build with a HIPAA-aware data posture from the outset, not retrofitted. The platform surfaces current pharma and device offerings on the provider side; the pharma side gets a structured channel into clinical workflows. The web-design layer matters because the audience is clinical, not consumer.

Solution

  • SaaS platform build with provider-side and pharma-side surfaces.
  • HIPAA-aware data handling baked into the architecture: access control, audit, retention, and storage decisions made with the compliance posture in mind.
  • Provider-side discovery surface for pharma and device offerings.
  • Pharma-side channel into the provider workflow.
  • Web-design and IA tuned for a clinical audience.

Outcome

Above. The platform addresses a documented industry gap (the >90% awareness shortfall) with a SaaS connection model rather than a sales-rep channel.

saas · 2024 · a multi-location restaurant operator

AI-assisted restaurant reputation management SaaS

Review aggregation and response SaaS: pulling restaurant reviews from multiple platforms into one workflow with AI-assisted classification and response drafting.

AI-assisted restaurant reputation management SaaS — case study
  • 1 workflow: reviews from multiple platforms unified for analysis and response scope record

Context

Restaurants live and die on reviews across half a dozen platforms (Google, Yelp, TripAdvisor, delivery apps, and more), and the operator’s inbox can’t keep up. The brief was a SaaS that pulled reviews into one place, made sense of them, and helped the operator respond.

Problem

Treating each platform’s inbox as a separate workflow scales poorly past the first location. Treating them as one workflow without classification produces noise: a complaint about service and a comment about the parking lot need different responses, and right now they’re treated the same.

Approach

A React + Node.js SaaS that pulls reviews from the platforms each operator uses, applies AI-assisted classification (sentiment, category, response priority), and routes each review into a response queue with suggested drafts.

Solution

  • React front end for the operator’s review-management surface.
  • Node.js back end for platform integrations and review-state management.
  • AI-assisted classification of incoming reviews (sentiment, category) to set prioritization.
  • Response drafting assistance with operator-in-the-loop, never auto-publishing.

Outcome

Above. The unified-workflow shape is the load-bearing decision; the AI assist is the productivity multiplier on top.

e-commerce · 2024 · GoLeanPro

GoLeanPro — custom Shopify theme from Figma

Fully custom-coded Shopify theme translated from Figma, with parallax effects, scroll animations, and geolocation-based 7-day weather integrated into the storefront.

GoLeanPro — custom Shopify theme from Figma — case study
  • Figma → custom Shopify theme translated end-to-end without theme-shop reliance scope record
  • 7-day geolocation-based weather forecast widget in the storefront scope record

Context

GoLeanPro arrived with a designer’s Figma file and a Shopify storefront brief. The standard path (start from a paid theme, customize) was on the table; the brief was strong enough that a from-scratch translation made more sense.

Problem

A Figma-to-Shopify translation has more dragons than a Shopify-to-Shopify edit. Scroll animations and parallax want to fight Shopify’s section-rendering model unless the theme is built with them in mind from the start. A custom geolocation widget adds a permissions and progressive-enhancement story on top.

Approach

A fully custom-coded Shopify theme (Liquid templates, sections, and assets all written from the Figma spec) with parallax and scroll animations designed in alongside the section structure rather than retrofitted. The geolocation weather widget was scoped from the start as a hero-area feature, not an afterthought.

Solution

  • Custom Shopify theme coded from the Figma source: Liquid templates, sections, and snippets.
  • Parallax effects and scroll animations woven into the section model.
  • Geolocation-based 7-day weather forecast rendered in the storefront with a permissions-respecting fallback path.
  • CSS / HTML5 implementation tuned for the Shopify section model.

Outcome

Above. The custom-from-Figma path beats theme-shop customization when the design has parallax or scroll-state behaviour that touches the section structure.

e-commerce · 2024 · an e-commerce operator

Full-stack e-commerce platform on Node.js + React

Full-stack e-commerce build with a React front end and a Node.js back end over a multi-month engagement.

Full-stack e-commerce platform on Node.js + React — case study
  • 7–8 months engagement duration end-to-end engagement record

Context

A standard e-commerce brief (storefront, cart, checkout, admin) built end-to-end on a React + Node.js stack over a multi-month engagement.

Problem

E-commerce is well-trodden but unforgiving on details: payment flow, inventory state, and order-confirmation reliability each have their own failure modes. The brief was to ship a coherent build rather than glue together three half-integrated SaaS products.

Approach

A React front end and a Node.js back end with API integration between the two, sized for the operator’s catalog and order volume. The stack choice was deliberate: same language across the stack, easier hand-off, fewer integration seams.

Solution

  • React front end for the storefront, cart, and account surfaces.
  • Node.js back end for product, cart, order, and admin APIs.
  • REST API contract between the two layers.
  • Standard e-commerce primitives (catalog, cart, checkout, order history, admin) all built on the same stack.

Outcome

Above. The 7–8 month engagement delivered the full-stack platform; the React/Node stack kept the seams minimal for ongoing maintenance.

fintech · 2025 · DatVerse engineering portfolio

Options trading automation bot — rule-based execution engine

Rule-based options trading automation bot with a deterministic execution engine, backtest harness, and risk controls.

Options trading automation bot — rule-based execution engine — case study
  • Rule-based deterministic execution with backtest harness for new-rule validation architectural record

Context

Options trading automation lives in the space between strict rule-based execution and discretionary trader judgment. The build is a deliberate counterexample to the “throw an LLM at it” pattern: a deterministic rule engine where every execution path is auditable.

Problem

LLM-driven trading bots are interesting research but bad operating reality. They fail in ways that aren’t reproducible, and risk controls have to be bolted on around them. A rule-based engine inverts that: rules are explicit, backtestable, and reviewable.

Approach

A Python rule engine with backtest harness as a first-class subsystem: every new rule passes through the backtest before it gets a live-execution path. Risk controls (position size, daily loss caps, instrument filters) sit at the engine level, not as ad-hoc per-rule checks.

Solution

  • Python rule engine with deterministic execution paths.
  • Backtest harness for validating new rules against historical data before live deployment.
  • Configurable risk controls (position size, daily loss caps, instrument filters) at the engine layer.
  • Audit log of every execution decision for post-trade review.

Outcome

Above. The rule-based posture is the methodology contribution; deterministic execution and backtest-first validation are the operating discipline.

enterprise · 2025 · DatVerse engineering portfolio

RAG document-retrieval pipeline — classical IR baselines + modern vector RAG

End-to-end RAG retrieval pipeline benchmarked against classical IR baselines on the CISI corpus: semantic chunking, ChromaDB vector store, cross-encoder reranking, and LLM-grounded answer generation.

RAG document-retrieval pipeline — classical IR baselines + modern vector RAG — case study
  • 1,460 / 112 CISI benchmark: documents / queries with precision/recall scoring published IR corpus
  • 4 retrieval models: Boolean, TF-IDF VSM, dense semantic, cross-encoder-reranked internal benchmark

Context

RAG implementations get asserted, not measured. “It works for our use case” is the modal claim. The build was a deliberate counterexample: a complete RAG pipeline alongside classical IR baselines, all benchmarked against the same published corpus so the lift from each stage is quantifiable.

Problem

A modern RAG stack (bi-encoder embeddings → vector store → top-k retrieval → cross-encoder rerank → LLM generation) makes a lot of architectural decisions implicitly. Without baselines, “the rerank helps” and “the bi-encoder is good enough” are both unfalsifiable. With baselines, they’re measurable.

Approach

Four independently-runnable sub-pipelines: classical Boolean retrieval, classical TF-IDF Vector Space Model with cosine similarity, modern dense retrieval with bi-encoder embeddings, and dense retrieval with cross-encoder reranking. All evaluated on the published CISI corpus (1,460 documents, 112 queries with ground-truth relevance judgments) with precision/recall scoring. Each pipeline benchmarkable in isolation, swappable surgically.

Solution

  • Document ingestion with semantic chunking (not fixed-token windows).
  • Embedding generation via HuggingFace Sentence Transformers: deliberate bi-encoder for recall, cross-encoder for precision.
  • Local ChromaDB vector store with metadata-aware retrieval filtering.
  • LangChain orchestration chaining retriever, reranker, and generator into a reusable pipeline.
  • Classical Boolean retrieval with AND/OR/NOT operators implemented as a separate sub-pipeline.
  • TF-IDF Vector Space Model with cosine similarity ranking implemented as a separate sub-pipeline.
  • CISI-benchmark evaluation harness computing precision and recall against ground-truth relevance judgments.
  • Four-sub-project codebase layout: each retrieval approach in its own folder, with pinned dependencies and step-by-step setup.

Outcome

Above. The “RAG works because the eval says it does” framing is the methodology contribution. The codebase is paste-ready as a reference architecture for documentation / knowledge-base RAG projects where the retrieval quality has to be defensible, not assumed.

enterprise · 2025 · DatVerse engineering portfolio

LLM job-fit and skill-gap assistant — resume + JD parser with NLP fallback chain

Django + pgvector job-fit assistant: resume and job-description parsing through a transformer-NER + sentence-embedding fallback chain, with vector similarity colocated in PostgreSQL.

LLM job-fit and skill-gap assistant — resume + JD parser with NLP fallback chain — case study
  • 4-stage NLP fallback chain: transformer NER → SkillNER → spaCy → sentence-embedding similarity architectural record
  • pgvector vector similarity colocated in PostgreSQL with no separate vector service architectural record

Context

A candidate uploads a resume, pastes a job description, and wants a structured fit-score plus a prioritized skill-gap list, not a generic “you need more skills” hand-wave. The platform decision was whether vector similarity belongs in a separate vector service or colocated with the relational data.

Problem

Single-approach skill extraction misses skills. Transformer NER catches the long tail but misses taxonomy-anchored skills; SkillNER catches the taxonomy but misses informal language; spaCy gives a lexical baseline but misses semantic equivalents. Without a fallback chain, every approach silently drops candidates.

Approach

A multi-stage NLP fallback chain: transformer-based NER first (Nucha-BERT / JobBERT class models for high precision), SkillNER second (taxonomy-backed coverage), spaCy third (lexical baseline), sentence-embedding similarity via pgvector last (semantic long-tail). Each fallback catches what the previous layer missed. Vector similarity colocated with relational data in PostgreSQL via pgvector, so relational and vector queries hit the same store with no separate vector service.

Solution

  • Django 5 + Django REST Framework backend with drf-spectacular OpenAPI schema and JWT-secured endpoints (djangorestframework-simplejwt).
  • PostgreSQL + pgvector for colocated relational and vector data: skills, embeddings, and user/resume/JD records in one store.
  • Resume parser via pdfplumber with regression-test fixtures.
  • Multi-stage NLP fallback chain: transformer NER → SkillNER → spaCy → sentence-embedding similarity.
  • Modular Django apps per concern: resume parsing, JD parsing, gap analysis, progress tracking, learning resources, search/match.
  • TypeScript + Vite SPA frontend wired to the backend OpenAPI contract.
  • CPU-only PyTorch wheel installation path so the backend runs end-to-end on a developer laptop without the multi-GB CUDA payload.
  • pytest test suite, ruff linting, INSTALL.md verified on Python 3.13 (Windows).

Outcome

Above. The fallback-chain + pgvector-colocation pair is the architectural contribution. The “vector store inside PostgreSQL” approach matters when teams want one operational store rather than two, and is paste-ready as a reference for similar parsing / matching / scoring use cases.

enterprise · 2025 · DatVerse engineering portfolio

Retail data warehouse — Bronze/Silver/Gold medallion on Databricks Lakehouse

End-to-end retail data warehouse on Databricks Lakehouse: medallion architecture with a Kimball dimensional Gold layer answering one analytical question across five operational sub-systems.

Retail data warehouse — Bronze/Silver/Gold medallion on Databricks Lakehouse — case study
  • 13 / 14 Kimball dimensional Gold layer: conformed dimensions / facts warehouse manifest
  • 5 operational sub-systems unified: point-of-sale, payment gateway, marketplace, classifieds, returns scope record
  • Idempotent end-to-end: re-runnable from any layer without duplicate keys or fact drift architectural property

Context

The analytical question came from an actual cross-department disagreement: was the Back-to-School Tech & Essentials marketing campaign genuinely successful, or did every operational sub-system count campaign success differently? Point-of-sale, payment gateway, marketplace, classifieds, and returns each had their own answer, and none of them lined up.

Problem

Five sub-systems each producing their own version of “campaign success” is an information-architecture problem dressed as a reporting problem. Without conformed dimensions enforcing a single source of truth, every cross-department analytical question gets five answers and a Slack argument.

Approach

A Bronze / Silver / Gold medallion architecture on a Databricks Lakehouse with Unity Catalog Volumes for storage. The Gold layer is a Kimball dimensional model with 13 conformed dimensions and 14 facts, so cross-sub-system questions traverse one shared model rather than five duplicated ones. Idempotent transformations end-to-end with SHA-256 surrogate keys for deterministic re-derivation.

Solution

  • Bronze landing zone for raw transaction, classified-listing, point-of-sale, payment-gateway, marketplace, and seller data.
  • Silver layer with idempotent transformations and schema-aware cleanup; marketplace + classified-listing Silver layer owned end-to-end.
  • Shared silver_template notebook every Silver notebook inherits from, with consistent anomaly handling and validation patterns across owners.
  • Gold layer with 13 conformed dimensions and 14 facts in Kimball star-schema form.
  • SHA-256 surrogate-key generation for deterministic identity.
  • Snowflake integrated as a read/write target via the snowflake-connector-python adapter, replacing the spark-snowflake Maven JAR in a serverless-only environment.
  • Platform-adaptation work documented end-to-end: migrated from a deprecated Databricks Community-Edition assumption to Free Edition (serverless compute, Unity Catalog Volumes, widgets-for-secrets, Python Snowflake adapter).
  • Supporting SQL artifacts under a dedicated sql/ folder for downstream BI consumption.

Outcome

Above. The Gold layer answers the Back-to-School-campaign question consistently across operational sources rather than producing five competing answers; that is the warehouse’s operating value. Idempotent Silver + Gold transformations + SHA-256 surrogate keys make the pipeline safe to re-run from any layer without duplicate keys or fact drift.

enterprise · 2025 · DatVerse engineering portfolio

Azure Data Factory cloud ETL — canonical Git-published artifact layout

Azure Data Factory orchestration authored in the canonical Git-published artifact layout: datasets, pipelines, triggers, and linked services as version-controlled JSON, not GUI clicks.

Azure Data Factory cloud ETL — canonical Git-published artifact layout — case study
  • 5 canonical resource directories: dataset, factory, linkedService, pipeline, trigger layout record
  • Git as source of truth: every change a diffable JSON commit architectural property

Context

Azure Data Factory pipelines authored only in the GUI carry a hidden cost: the workspace becomes the source of truth, and code review, rollback, and multi-environment promotion all become harder than they need to be. The discipline of authoring against the canonical Git-published artifact layout solves it; the project is the proof of that discipline applied end-to-end.

Problem

“It works in the GUI” and “we can roll this back if it breaks” are not the same statement. Without a Git-tracked artifact layout, pipeline changes have no diff, no review, and no clean rollback path. ADF emits the canonical layout when source-control configuration is enabled, but only if practitioners author against it consistently.

Approach

Author every ADF resource as a version-controlled JSON artifact in its canonical subfolder: Linked Service, Dataset, Pipeline, Trigger each in their own top-level directory, plus the publish_config.json that wires the workspace to the publish/collaboration branch contract. Treat the layout as the authoritative state of the pipeline; the GUI is a renderer.

Solution

  • ADF artifacts authored in the canonical Git-published layout: dataset/, factory/, linkedService/, pipeline/, trigger/ subfolders.
  • publish_config.json configured for the publish/collaboration branch contract, so the workspace stays in lockstep with Git.
  • Every ADF resource as a one-JSON-per-resource definition: code-reviewable, branchable, promotable through standard Git tooling.
  • Linked services, datasets, pipelines, and triggers all version-controlled rather than GUI-only.

Outcome

Above. The artifact-layout discipline is the load-bearing engineering outcome: it is what separates a team that can roll back a broken pipeline from one that cannot. Operational specifics (pipeline counts, source/sink systems, schedule cadence) live in the private repo and are shared in client conversations rather than on the public surface.

manufacturing · 2025 · DatVerse engineering portfolio

Real-time IoT streaming pipeline — Bronze/Silver/Gold medallion on PySpark Structured Streaming

End-to-end IoT sensor-telemetry pipeline: Bronze ingestion, Silver enrichment and validation, Gold windowed aggregation with stream-stream joins on PySpark Structured Streaming.

Real-time IoT streaming pipeline — Bronze/Silver/Gold medallion on PySpark Structured Streaming — case study
  • 3 + 1 medallion layers (Bronze, Silver, Gold) plus a data generator for local testing architectural record
  • 1-min windowed aggregations with stream-stream joins at the Gold layer architectural record

Context

IoT sensor-telemetry workloads have a shape that batch pipelines handle badly: events arrive continuously, validation has to happen per-record, and downstream aggregations need windowing rather than full-dataset scans. The build is a proof that the lakehouse medallion architecture extends cleanly to the streaming case.

Problem

A streaming pipeline that mixes ingestion, enrichment, and aggregation into one notebook is impossible to debug. Per-record validation gets tangled with cross-event windowing; cross-source joins fight schema drift; the whole graph becomes hard to inspect when something silently goes wrong.

Approach

A Bronze / Silver / Gold medallion architecture applied to the streaming case. Bronze ingests raw JSON sensor records as an append-only, schema-aware stream. Silver enriches each event with metadata and runs per-record quality validation. Gold materializes 1-minute windowed aggregations using stream-stream joins for cross-source enrichment. Four independently-runnable terminals (generator + the three medallion layers) so each stage can be debugged in isolation.

Solution

  • Data generator emitting JSON sensor records to simulate live IoT emissions for local development.
  • Bronze layer with raw JSON-stream ingestion, append-only schema-aware writes.
  • Silver layer with metadata enrichment and per-record quality validation.
  • Gold layer producing 1-minute windowed aggregations with stream-stream joins across the enriched event streams.
  • validate_pipeline.py validation harness covering the three layers independently.
  • Pinned Python dependencies in requirements.txt for reproducibility.
  • Setup documentation for Apache Spark 3.x, Java 11 or 17, and Hadoop winutils for Windows with PowerShell-friendly env-var configuration.
  • Four-terminal run pattern (generator, Bronze, Silver, Gold) so each stage is independently debuggable.

Outcome

Above. The medallion-streaming pairing is the methodology contribution: the same Bronze / Silver / Gold pattern used in modern lakehouse deployments, applied here as a working proof on a realistic IoT telemetry workload. Reusable as a reference for any live-enrichment + windowed-aggregation pipeline.

enterprise · 2025 · DatVerse engineering portfolio

Hadoop MapReduce batch analytics — HDFS Streaming pipeline in Python

Three-stage Hadoop MapReduce pipeline (mapper → combiner → reducer) in plain Python via Hadoop Streaming, cluster-portable from single-node to multi-node.

Hadoop MapReduce batch analytics — HDFS Streaming pipeline in Python — case study
  • 3-stage MapReduce pipeline: mapper → combiner → reducer in Python over Hadoop Streaming architectural record
  • Same scripts single-node and multi-node, cluster-portable from pseudo-distributed to full Hadoop cluster reproducibility property

Context

Modern lakehouse work doesn’t replace the need to know on-prem big-data tooling: buyers in regulated or air-gapped environments still ask whether the team can ship to a Hadoop cluster they already operate. The pipeline is the proof that the foundational MapReduce pattern is intact and runnable end-to-end.

Problem

A MapReduce job authored in Java forces a JVM dependency on every node and slows iteration. A Streaming job in Python (stdin/stdout piping) keeps the cluster Java-only on the runtime side and lets the analytics logic live in plain scripts that any data engineer can read.

Approach

A three-stage Hadoop Streaming pipeline: mapper emits per-record tuples, combiner does local-node aggregation before the network shuffle, reducer produces final cross-key aggregations. Pure stdin/stdout piping, no Hadoop-Java dependencies. Same scripts run on a pseudo-distributed single-node install for local development and on a full multi-node cluster for production-style runs.

Solution

  • mapper.py parses tab-separated movie records (UID, title, genres, year, rating) and emits per-record key-value tuples.
  • combiner.py runs local-node aggregation before the network shuffle: the textbook MapReduce performance optimization that reduces shuffle volume.
  • reducer.py produces final aggregations across genre, rating, and year dimensions.
  • Optional years.txt filter so the same pipeline can be re-run scoped to a subset of years without recompiling logic.
  • Hadoop-Streaming-compatible: pure stdin/stdout piping, runnable through any Hadoop 3.x cluster.
  • Cluster-portable: same scripts on a pseudo-distributed single-node install or a full multi-node cluster.
  • Final reducer output captured in results.txt for inspection and downstream consumption.
  • README covering prerequisites (Java 8 or 11, Hadoop 3.x, Python 3.x per node) and the exact hadoop jar hadoop-streaming-*.jar invocation.

Outcome

Above. Paired with the PySpark streaming pipeline, the combo demonstrates full-stack distributed-processing range (on-prem legacy MapReduce alongside modern lakehouse streaming) for buyers whose environment spans both worlds.