AI delivery · Data foundations

Get your real data and access rules ready for an AI system that worked in the demo

If an AI system works on demo data but your real data is scattered, inconsistent and nobody is sure who may access what, you can commission me to build the missing foundations. I inventory the sources, map ownership and permissions, add freshness and quality checks, and deliver an ingestion plan with the first pipelines working, so the AI system can use real data under real access rules.

This is a good fit if…

  • A pilot or prototype works on a curated sample, and the real data sits across shared drives, SaaS tools, databases and spreadsheets.
  • Nobody can say with confidence who owns each source, how often it changes or which users may see which records.
  • An AI system needs to respect existing access rules, and those rules are not recorded anywhere a pipeline can read.
  • You have a data engineering manager or Head of AI who will own the pipelines afterwards.

Look elsewhere if…

  • You need a full data warehouse or data platform programme. This work is scoped to what a specific AI use case needs.
  • Sources and access are already well understood and you need the AI system itself built. Use RAG and enterprise search engineering or AI workflow automation.
  • You have not chosen a use case yet. Start with the AI readiness and implementation roadmap.

What you get

Source inventory, ownership and permission map, freshness checks and ingestion plan

  • A source inventory: every system the use case needs, what it holds, its format, volume, change frequency and owner.
  • An ownership and permission map showing who owns each source and how its access rules translate into something a pipeline can enforce.
  • Freshness and quality checks that alert when a source stops updating or its data changes shape.
  • An ingestion plan, with the first priority pipelines implemented and tested.
  • A list of gaps that need business decisions, each with a named owner.

How it runs

  1. 01

    Brief and fit check

    You describe the AI use case, what works in the demo and what breaks with real data. I reply with questions and a view on fit before anything is signed.

  2. 02

    Source inventory

    With system owners I list every source the use case needs, sample its data, and record format, volume, change frequency, quality issues and owner.

  3. 03

    Ownership and permission mapping

    For each source I trace how access is granted today and how it can be carried into the AI system, and record the decisions only the business can make.

  4. 04

    Pipelines and checks

    Priority sources get ingestion pipelines with permission metadata, freshness checks and quality checks, built in your environment.

  5. 05

    Acceptance and handover

    Agreed measures are checked and the pipelines, inventory and runbook pass to the data engineering owner.

What needs to be in place

  • A specific AI use case, ideally with a working prototype or demo.
  • Contacts for each source system who can explain its data and access rules.
  • Read access to the sources through your normal approval process.
  • A data engineering owner for the pipelines after handover.

Not included

  • Building an organisation-wide data platform or warehouse.
  • Cleaning or rewriting source content; quality issues are reported to owners with evidence.
  • Deciding access policy; I make current rules explicit and flag gaps for owners to decide.
  • Legal or data-protection advice; your privacy owners make those calls.

The situation

The demo was convincing. A model answered questions about a folder of documents, or an agent updated records in a test system, and everyone could see the potential. Then someone tried it on real data.

Real data lives in a document store, two SaaS tools, a database nobody has documented and a set of spreadsheets maintained by one person. Some sources are updated daily; some have not changed in a year but are still treated as current. Formats disagree. Customer IDs do not match between systems. Most importantly, nobody can say clearly who is allowed to see what, and the demo ignored that question entirely.

The model is rarely what is missing. What is missing is the foundation underneath it: known sources, clear owners, enforceable permissions and data that stays fresh.

What the work involves

Inventory what the use case actually needs. Not every system in the organisation, only the sources this AI use case depends on. For each, I sample real data with its owner and record what it holds, its format and volume, how often it changes, known quality problems and who is responsible for it.

Map ownership and permissions. For each source I trace how access works today: groups, folder permissions, record ownership, client or region codes, informal rules. Then I work out how those rules can travel with the data, as metadata a retrieval system or agent can enforce at query time. This follows the same principle as the memory layer of the Substrate Pattern I have published: what an AI system can see is scoped by identity in a layer it cannot bypass. Where rules exist only as habits, they are written down and confirmed by the owner.

Add freshness and quality checks. Each source gets checks that alert when updates stop arriving, volumes change unexpectedly or the schema shifts. Answers built on stale data are one of the most common reasons users stop trusting AI systems.

Build the first pipelines. The priority sources get ingestion pipelines in your environment, carrying permission metadata and running the checks, so the AI system can be tested on real data under real rules.

The signature deliverable

You receive the source inventory, ownership and permission map, freshness checks and ingestion plan. Illustrative example of an inventory extract:

SourceOwnerChangesAccess ruleStatus
Policy library (document store)HR operationsWeeklyGroup-based; contractors excludedPipeline built; freshness check live
Customer contracts (shared drive)LegalAd hocPer-client foldersRule confirmed; pipeline next
Pricing sheet (spreadsheet)No confirmed ownerUnknownAnyone with linkGap: needs owner before use
Support tickets (SaaS)Support leadContinuousPer-team queuesAPI access pending approval

Illustrative example showing the format, not a client deliverable.

How acceptance is judged

Acceptance measures are agreed after the inventory: every source in scope has a confirmed owner or a recorded gap; every source’s access rule is documented and, for built pipelines, enforced in the ingested data; freshness and quality checks run and alert correctly in a test; and the AI system can run on the ingested data in a test environment. The data engineering owner signs off.

Ownership and handover

Pipelines, checks and the inventory live in your repositories and tools. The owner receives the runbook for responding to alerts, adding a source through the same steps and handling access-rule changes. Gaps needing business decisions are handed to named owners. I do the work personally; any specialist help is disclosed and approved by you first.

Boundaries

Once the foundations are in place, the AI system itself is built through RAG and enterprise search engineering, AI workflow automation or document workflows. If you have not yet chosen which use case to pursue, start with the AI readiness and implementation roadmap.

Questions buyers ask

Isn't this just a data engineering project?

It uses data engineering, but it is scoped to one AI use case and its specific needs: permission metadata that retrieval or agents can enforce, freshness that answers depend on, and formats models can use. It is deliberately smaller than a platform programme, and it ends with pipelines working for that use case.

What if nobody knows who owns a source?

That is a common finding, and it is recorded as a gap with a proposed owner for the sponsor to confirm. A source without an owner will drift, and an AI system built on it will drift too. Sometimes the right decision is to leave that source out until someone takes responsibility.

How do you carry access rules into the AI system?

For each source, I trace how access is granted (groups, folders, record ownership, client codes) and capture it as metadata alongside the data, so retrieval or agents can filter by the user's rights at query time. Where rules exist only in people's heads, they are written down and confirmed by the owner before any pipeline relies on them.

Will you move our data somewhere new?

Only as far as the use case needs, and only into infrastructure you control and approve. Often the pipelines read from sources in place and write to an index or store the AI system already uses. Copying data creates a second thing to secure and keep fresh, so I keep it to a minimum.

What happens when a source changes?

Freshness checks alert when a source stops updating, and schema and quality checks alert when its data changes shape. The runbook explains how to respond, and the inventory records who to contact for each source.

Describe what needs to work

A short, non-confidential description is enough to start. I read every brief personally and reply within two business days, including when the answer is that I am not the right fit.

Step 1 of 2 · The basics