Data engineering
Ingestion, storage, movement and migration. We have run readings from device fleets into time series storage, moved years of sensitive records with nothing lost, and put reporting in front of people who actually open it.
Book a 30-minute scoping call See the data work
Most stalled analytics projects are stuck here, not at the model.
| Readings ingested MQTT, CoAP and HTTP | Streaming |
|---|---|
| Rows reconciled Source and destination counts match | Matched |
| Query on a year of history Time series storage | Fast |
| Failed batch Retried, then escalated | 1 alert |
| Report refreshed Before the morning meeting | 06:00 |
The honest question first
Data engineering is expensive and invisible. It is worth doing when the alternative has started costing more.
Leave it alone when
Building a warehouse for this is a hobby, not an investment.
Fix the plumbing when
This is the work we do, and it is usually what an AI project actually needs first.
Plain definitions
The second one in particular gets answered by architecture diagrams when it should be answered by a count of your sources.
It is the work of getting data from where it is produced to where it is used, in a state somebody can rely on. Ingestion, storage, movement, reconciliation and the monitoring that tells you when a feed has stopped.
Data science gets the attention and data engineering gets the calls at six in the morning. When an analytics or AI project stalls, this layer is almost always the reason.
A warehouse holds structured data modelled in advance for reporting. A lake holds raw files of any shape and defers the modelling. A lakehouse is the attempt to have both, with warehouse-style querying over lake storage.
For most companies asking us, the honest answer is that none of the three is the first problem. One reliable feed and one agreed definition of the number people argue about will change more than any of these labels. We would rather build that and let the architecture question wait until the volume forces it.
What we build
What people ask us for
Almost every enquiry is one of these. Two of them are usually happening at once.
Our work
Case study, IoT platform
IoT Workz needed measurement data from device fleets without renting it back from a SaaS vendor. We built the ingestion path over MQTT, CoAP and HTTP, put readings into TimescaleDB, used Kafka where decoupling mattered, object storage for raw payloads, and Prometheus with Grafana so the pipeline reports on itself.
Case study, Norway
The client had to leave a system that could no longer handle its volumes or meet security standards, without interrupting the schools using it. Migration ran as its own workstream with a dedicated team, reconciled on both sides, alongside a phased rollout and 60 days of warranty support.
How the work runs
Week 1
Where data comes from, how much, how often, and which number people argue about.
Week 2 to 3
A single source ingested, stored, monitored and reported, so the shape is proven before scope grows.
Week 4 to 12
More sources, then any migration, rehearsed before the real run.
Handover
Pipelines, monitoring and documentation are yours, deployed in your own tenant.
Technology
Everything here is running somewhere in production. We have left off the tools we have only read about.
Who does the work
Pipelines get handed over and then live for years without us. That only works if the person who built yours wrote documentation somebody else can follow, which is easier to insist on across twenty people than across two hundred.
You interview whoever we put forward. Ask what their alerting looks like when a feed fails at two in the morning, because that answer separates the careful from the quick.
How we work
The models differ in one thing: how much of the management you hand over. Everything else, including who owns the code, is the same in all five.
Changing model later is normal. Augmentation into a dedicated team is the common direction, and the people stay.
Engineers join your team and work inside your process, your board and your code review.
A team that works only on your product, with a lead on our side running delivery.
A long-term engineering site under your standards, where we carry recruitment, HR, payroll and equipment.
04
One price for a scope that is genuinely settled, with the overrun carried by us.
Everything in the centre model, with the whole team moving into your own Vietnamese entity on an agreed date.
FAQs
Usually neither, at first. Most of the value comes from reliable ingestion and one agreed source for the numbers people argue about. We would rather fix that than sell you an architecture diagram.
Yes. We treat migration as its own workstream with reconciliation on both sides, and we rehearse it before the real run.
Yes. We build to what your team already uses, including Power BI and Grafana, rather than making everyone learn something new.
Your own tenant, in the region your policy requires. On the IoT platform the client hosts everything themselves.
The pipeline monitors itself and alerts. Finding out because a chart looks flat is not a monitoring strategy.
Almost always. Models fail on inputs far more often than on algorithms. If an AI project has stalled, this is usually why.
One feed ingested, stored, monitored and reported end to end is usually three weeks. A working platform across several sources, with migration behind it, takes three to four months. Migration off a legacy system across regions runs longer again. Which engagement model you pick decides how it is billed, and the models are set out above.
We have not shipped on them, and we will say so rather than learn on your budget. Our production work is PostgreSQL, TimescaleDB, Kafka and object storage, self-hosted or in your own cloud tenant. If your organisation has already standardised on one of the big platforms, hire a team that lives in it.
ETL transforms the data before it lands, ELT loads it raw and transforms it afterwards. ELT is the more forgiving default, because raw data you kept can be reprocessed and raw data you transformed away cannot. It matters less than the two things people skip, which are reconciliation and monitoring.
That argument usually points straight at the broken feed. Thirty minutes is enough to find it and say what fixing it involves.
