Testing an AI decision support system

A case study on testing an AI Decision Dupport system (DSS) for transport planners — and the constraint that revealed the wrong company was trying to solve it.

Decision support system jargon

Some terms we used internally for this work:
Global Driver Assignment Support (GDAS) algorithm, which optimises all jobs across all drivers at all providers.
A step down was Local Driver Assignment Support (LDAS), operating at a single provider level — which driver inside a provider is best for a certain job.

Strategic impact

• Established a low-cost, high-fidelity testing pattern (real jobs + co-located planner sessions) that reached a definitive go/no-go answer in weeks rather than months.
• Stopped further technical investment as the business stakeholders wanted to push out a product that wouldn't work well enough to see usage, and would need further technical investment.
• Drafted a roadmap of features we may need to build if the project was to progress and succeed.
  
• Findings shaped a CPO-level decision to deprioritise GDAS in January 2025, freeing the team to focus on Dispatch as the foundational product.

Context

The problem

Motorway wanted to offer the best collection, inspection and delivery service; to give us a competitive advantage over our competitors. This meant partnering with many different transport providers. Which introduced issues around scalability.

Planning point-to-point vehicle movements is a very manual and difficult process. Getting an optimal plan together is almost impossible. Having more jobs and more planners works for a bit, but you begin to hit diminishing returns pretty quickly.

How might Motorway and its partners plan around a thousand jobs a day across tens of different transport providers, with the ability to replan and re-optimise a plan as things change?

Our objective

Build a Decision Support System (DSS) planning assistant for transport planners, helping them become more efficient.
‍
We would go on to test a first iteration of LDAS, specifically focused at planning at the 'frontier' of existing plans.

My role

I was Lead Service Designer on discovery and design; and also the Product Manager towards the end of the project. I partnered closely with our ML Engineer and Tech lead throughout.

Collaborators: Andre (ML Engineer), Malcolm (Tech lead), Transport Operations team

What we knew already

Planners hold enormous context in their heads: which drivers need to finish early, who's reliable for high-value vehicles, which locations are awkward, how to balance workloads so drivers don't quit.

We observed they generally work in two modes. Either starting with a job and finding a driver for that job; or starting with a driver and finding jobs near that driver's last delivery location.

Experienced planners read postcodes like the Matrix, they occasionally jump to supporting information on a job, or explore public transport options on Google Maps. With less experienced planners relying more heavily on the latter information.

They used non-Motorway jobs to 'get' their driver near to a Motorway seller's collection address. This was done to reduce costs and public transport time.

This is some text inside of a div block.

Designing a testable prototype

Leveraging previous discovery work from building the Dispatch product, I took some of the earlier stage conceptual designs we'd explored and refined them.

Earlier planning concepts showing nearby jobs for a driver to do

Exploring another way to show suggested jobs

Later version of scheduling view

Explaining the concept of planning 'at the frontier' to Motorway stakeholders

Andre and I decided on building a lightweight front end that ingested real job data and had some basic functionality (drag and drop, working logic for accepting suggestion etc).

We settled on a visual schedule view as well as a supporting table view to test. These supported both ways a planner prefers to work (job first and driver first). Both UI'ss gave us enough space to convey key info on jobs that a planner would need, with supporting information a click away.

This also allowed planners to see more of their driver pool than their current planning tool, which in this providers case was a complex Excel sheet.

Some of our findings

The algorithm showed promise. Job duration estimates were accurate. Some job pairings presented surprised planners—they'd check the public transport links and say "Actually, that could work."

“This may be a good job, but there is a better job I can give the driver”

Our algorithm was

• Too optimistic.
• Suggesting three jobs per day when planners know two is more realistic, they preferred to build slack into the plan
• Many plans had drivers spending 4+ hours on trains, basically making them unprofitable
• The algorithm didn't account for the chaos of reality: cancelled sales after an inspection happened, delays from previous jobs, missed connections on public transport
• Couldn't see non-Motorway jobs the planners were using
• We simply didn't have enough volume to make this work
• A lack of diversity in buyers meant too many drivers were going to the same delivery location, with no nearby collections to link up with.

We were able to tweak the parameters of the algorithm to make it less optimistic, we increased time at the end of the collection, and increased the assumed time on public transport, whilst also dialling down the suitable public transport distance.

This produced fewer suggested jobs in the data set, but the suggested jobs for drivers were taken more seriously by the planners.
‍
The real blocker was our visibility into their business. Planners kept saying: "That's a decent job suggestion, but I've got a better one from another client for that driver." Our algorithm only saw Motorway jobs. Planners are optimising across all their clients simultaneously. Even at 80% Motorway work, that remaining 20% of jobs we couldn't see changed everything.

Not getting buy in from the business

Planners don't have a "Motorway planning problem." They have a planning problem that happens to include Motorway.

To make the algorithm genuinely useful, we'd need visibility into providers' other jobs. But without that visibility, planners had no reason to adopt our tool.

Our operations team wanted a tool that optimised Motorway's jobs, our costs, our efficiency. But we'd learned that solving for Motorway wasn't the same as solving for the user.

Ops didn't see value in helping providers plan non-Motorway work.

Our recommendation

After presenting these findings, I recommended scaling back to a different type of product MVP.

We still wanted to try to get this tool into the hands of planners. To do this Andre and I proposed a lightweight alternative, that would allow planners to upload their current plan and all their unplanned jobs regardless of which client it was.

A headless system that allowed planners to upload a CSV, have the algorithm run and produce a suggestion CSV the planner could download and review.

I proposed planners' involvement in the development process was critical for this to succeed.
We'd learn more quickly and get them used to working with AI. No deep integration. No workflow change. A way to build trust incrementally.

How a scaled back DSS could work

The outcome

Our CPO paused work on GDAS/LDAS, redeploying Andre to another project. Later the business did a re-org and shifted focus, and the project was ultimately shelved. The proposed CSV MVP was never built past a conceptual test. In January 2025 development on GDAS stopped.

The discovery was that ultimately Motorway might not be the right company to solve this problem. We're one client among many. A planning tool that works for providers needs to work for all their clients—and that's a different product, for a different market.

If we could build a solid foundational product in Dispatch and revisit this when all the support infrastructure and features are built, we may have a better chance at success.

What I learnt

• Real data exposes real constraints. Interviews said planners wanted decision support. Testing with actual jobs revealed why they couldn't use it
• Understanding where you sit in users' broader workflow, and who else they serve is essential
• Recommending "no-go" is sometimes the right answer. Better to learn at prototype stage than after months of development
• Sometimes the most valuable outcome is clarity about what not to build
• Cross-discipline collaboration drives better decisions. Working closely with Andre meant we reached a joint conclusion neither could have found alone.