Data engineering
You end up with a data platform your team can run without us: modeled, tested, monitored, documented, and deployed through a pipeline anyone on the team can read.
A senior engineer reads it and replies within one business day. Or write to emilio@lyticaconsulting.com.
- Week one
- A written read of what you have and what we would build first
- Inside a month
- The first pipeline in production, tested and monitored
- From there
- Two-week cycles, with a written summary every week
The problem this solves
Most companies do not have a data problem. They have five copies of the same number and no agreement on which is right. Reports disagree because each was built against a different extract, by a different person, at a different time, and nobody wrote down which one was correct.
The usual response is to buy another tool. That rarely helps, because the warehouse was never the failure. The failure is that nothing enforces a definition, nothing tells you when a load fails, and the person who built the pipeline has left.
What we build
Artifacts you own, not activities we perform.
- Infrastructure defined in code
- Warehouse, storage, compute, networking, and access, on AWS or Azure, written down so the platform can be rebuilt rather than remembered.
- A modeled warehouse
- Facts and dimensions designed around the questions your business actually asks, rather than a mirror of whatever your source systems happen to emit.
- Ingestion and transformation
- ETL and ELT in Python and SQL, orchestrated in Airflow, with retries, backfills, and dependencies that are explicit instead of implied.
- Streaming, where it earns its keep
- Kafka or Flink when the business genuinely needs data in seconds. We will tell you when it does not, because streaming you do not need is a permanent tax.
- Monitoring and alerting
- Freshness, volume, and schema checks that reach a human before a business user finds the problem in a meeting.
- CI/CD
- Every change tested and deployed through a pipeline. No hand-edited production jobs, and no deploy that only one person knows how to run.
- Documentation
- Written for two readers: the analyst who needs to know what a column means, and the engineer who needs to change it without breaking anything.
A senior engineer reads it and replies within one business day. Or write to emilio@lyticaconsulting.com.
How the engagement runs
You should be able to picture week one before you sign anything.
Week one
Where we start
We get read access to your sources and your cloud account, and spend the week reading. You end the week with a written architecture assessment: what you have, what is broken, what we would build first, and what it costs to run today.
The first weeks
The first thing that runs
The first pipeline reaches production. Not a prototype — tested, monitored, and deployed through CI. We pick the one that unblocks the most people, which is usually not the one that is most interesting to build.
From there
The cadence
Two-week cycles, each ending with something running in your account under your control. You get a written summary every week, whether or not there is good news in it.
What we need, and what you get
What we need from you
- One person who can approve an architecture decision without convening a committee.
- Access to source systems inside week one. This is the single most common reason a build slips.
- A few hours a week from someone who knows what the data means to the business.
- The authority to decommission a source system when we find it is feeding nothing.
What you get from us
- Senior engineers only. Nobody is learning on your project.
- Work in your repository and your cloud account, from the first commit.
- A written weekly status, and a named engineer you can contact directly.
- Code, infrastructure, and documentation you own outright. No proprietary layer you have to keep paying us for.
Related
Tell us what you are trying to build
Describe the situation in your own words. A senior engineer reads it and replies, usually with questions rather than a brochure.
Or write to emilio@lyticaconsulting.com