OSDI '26 - Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms

Hi, my name is Gohar and today I'll be talking about our work Morakab, resource efficient agentic workflow orchestration in cloud platforms.

This is joint work with my colleagues from Microsoft Azure.

It goes without saying that a lot of us use agentic workflows and applications today. For example, productivity tools like Microsoft Copilot, cloud code for programming, and even domain-specific applications like video editing applications from Adobe are now built as agentic workflows.

In the context of this talk, we refer to models and tools composed into workflows as agentic workflows.

And in this talk, we will focus on the resource efficiency of agentic workflows and try to examine what the limitations of existing deployments are and how to improve those.

Given the diverse objectives and workloads, how do we optimally configure the system to achieve those objectives?

Let's take an example workflow for the rest of this talk. A video QA workflow that takes a set of videos and answers queries about them.

So, you input some questions and videos to this application and it passes them to a scene detection tool, which is one of the agents in this workflow.

This feeds into a frame extraction agent which may produce raw frames that are then fed into an object detection model which produces annotated frames.

In parallel, potentially, we can have the frame extractor also extract the audio track associated with sets of frames, which is fed into a speech-to-text model that produces a text transcript.

All of this intermediate output, along with the original question, is fed into a multimodal LLM to obtain an answer.

Note that in the context of this work, we consider models and tools being used in this workflow as all agents.

Now, there are many configurable pieces in this workflow at different levels.

For example, the number of frames to extract in the frame extractor agent or which model to use for the QA agent, are examples of agent level configurations.

Whether to enable some agent in the workflow, for example, do we want to use an object detector or not, and whether we want to parallelize pieces of the tag, are workflow level configurations, each presenting different trade-offs.

And finally, which resources or accelerators to run various agents on, are hardware level configurations.

So, let's see how such a workflow is defined and configured today.

So, here is a relatively generic, simplified example that illustrates some of the key points. This is not any specific framework, and things may vary based on how you choose to proceed, but the high-level message is roughly the same.

First, notice that there are various modules defined here, each with their own parameters.

You have a bunch of these modules for each task that we saw earlier. These comprise your application logic, and you construct a data flow between these tasks to execute as a workflow.

This is, of course, very important, but there is a lot more going on here as well. So, let's take a look at what that is.

You see resource configuration for most of these modules here. These are your hardware configurations.

Then you have agent configurations, specifying things like which models to use for each agent.

You have workflow configurations.

And these all are your execution details.

Whether in the definition itself, or partially when provisioning resources and deploying the workflow, all of these decisions need to be made up front.

The problem, however, is the application logic is very tightly coupled with the execution details, and the whole definition is baked in and deployed.

Just to put things in perspective, let's visualize part of this configuration space. There are just a few of these.

So, the Y-axis has accuracy, and the X-axis for each of the subplots has latency, cost per request, and energy consumption per request, respectively.

Let's consider the video QA workflow from earlier. So, we take the NVLM model as the multimodal LLM of choice for the QA agent, a couple of different options for the number of frames to extract in the frame extractor, and And disable speech-to-text, and use A100s with tensor parallelism four as the hardware choice.

This gives us us a few of these options for each objective.

We can add more GPU types, we can add more parallelism options, we can add more models, and this configuration space blows up really fast.

So, clearly, agentic workflows, even relatively simple ones, are not so easy to configure properly. And there is not a single correct configuration. It is a function of your request SLOs and deployment objectives.

And manual configuration, or trying to pick these configurations manually by hand, is often not Pareto optimal. You may end up being less than ideal in terms of accuracy while also wasting more resources and energy, for example.

So, all of this is our first source of inefficiency today. Rigid workflows that have configurations baked into the definitions are difficult to adapt later.

Now, assuming you have defined and configured this workflow, let's take a look at the remainder of the workflow's life cycle from its development to deployment and the different entities involved.

So, we saw how the application developer defines and configures the workflow, then deploy it and the pieces of this application across one or more cloud platforms, and then expose them to end users, say using an HTTP endpoint.

Now, notice how all of these entities have somewhat different objectives.

Different users may care about the three axes of cost, latency, and accuracy with varying degrees.

Developers care about profitability from their application and want to provide a good user experience.

And the cloud platform wants to maximize efficiency, minimize resources and energy consumption to drive up revenues.

But unfortunately, today, none of these layers has visibility into the other.

The cloud platform sees an opaque box statically configured application. It cannot adapt or reconfigure the workflow with changing system dynamics to yield better resource efficiency.

The developers are not really able to adapt to their users' preferences individually. They deploy an application that is one-size-fits-all.

Similarly, neither is the cloud platform able to capture the users' SLO and translate those into different resource allocation decisions to maximize efficiency.

And better efficiency, of course, would benefit everyone in the stack. For example, it can reflect in lower cost and better user experience. But we are far from this today.

So, this highlights the second inefficiency, which is a fragmented stack limits cross-stack optimization, leading to local optima rather than globally efficient configurations.

And this brings us to our goal. We want unified end-to-end orchestration of agentic workflows to deliver high resource efficiency while satisfying user-level SLOs.

To this end, we present our system Morakab that realizes part of this vision. And it does it through two key ideas.

First, it uses declarative workflow specifications that decouple application logic from execution details. In other words, we separate the what from the how.

The developer's job is to focus on expressing the intent of the application and the task composition. The configurations are taken out of the development phase of the workflow entirely.

Morakab then uses an adaptive SLO-aware orchestrator that generates and optimizes these agentic workflows holistically under a single umbrella. It leverages greater visibility to the different workflows, the incoming requests, and the system dynamics to be more resource-efficient while satisfying the SLOs. In other words, we make configuration a runtime decision, not a development time decision.

The workflow specification now, from the perspective of the developer, is just a simple spec, mostly in natural language like this, which is mainly just focusing on the application logic.

At request arrival time, there may be query-specific SLOs like low latency or maximize accuracy, etc., to guide the optimization process.

But here's the big challenge. How do you go from the representation on the left, which has hand-specified every detail, to the simpler declarative specification on the right that only expresses the intent of the application, but nothing else?

How does Morakab figure out all the execution details on its own?

There are two main pieces of information that need. First, which agents to use for each of the specified tasks, and two, how to configure the workflow at various levels like the agent, workflow, and hardware levels.

And Marabou does it through a centralized orchestrator and an optimizer.

We have a centralized orchestrator and library of agents. Each agent exposes an interface to invoke the agent, a set of configurations to configure the agent, and a natural language description of the agent so that the orchestrator can pick the right agent given a task in the specification.

An LLM with tool calling ability selects an appropriate agent for each task in the workflow, but does not select any configurations.

It only constructs a logical workflow transforming the specification into a DAG of agents that are still unconfigured.

The orchestrator is able to pick nodes for the logical workflow like shown here.

A frame extractor for the task of extracting frames, a question answer agent for answering questions given query and context, but notice the execution details shown in pink like the number of frames, the model, the accelerator type, parallelism are not locked in.

So, what about these configurations?

Well, this happens in the next phase.

Marabou does this through profile-guided optimization with a hierarchy of profiles. You have workflow level, agent level, and hardware level profiles, and each of these capture behavior of components in different parts of the stack.

And what makes this powerful is we can now compose these profiles from various levels of the stack. The agent level profiles are agnostic to which workflow uses them and can be reused across a wide variety of workflows.

Similarly, the hardware profiles are agnostic of how you are using a model as an agent. You can hence reuse these profiles for many different agents, each being used in many different workflows in turn.

So, profiling is tractable and reusable, which helps Marabou make runtime optimizations reasoning about the quality, latency, cost, and energy consumption of workflows.

So, the logical workflow without any configurations is taken as input by the optimizer along with the profiles. It then produces executable workflows with the execution details now locked in at runtime and also cached for later reuse until the optimizer runs again.

Now, the appropriate resources have also been allocated and provisioned, ready to execute the workflows.

Let's look at workflow profiles first.

For the video QA workflow example that we've been considering, on the x-axis, we have different configurations. The model chosen is shown in different hatch patterns, and the number of frames and speech-to-text enabled or disabled as the x values.

We measure the impact on accuracy on an open-source data set as a proxy for the workflow quality.

So, for example, with 10 frames and speech-to-text agent enabled, the QA agent using a Gemma model delivers about 65% accuracy on this set.

Now, next we consider the load of each of these configurations um on the agent itself. In this case, the multimodal LLM's token generation load.

We show a distribution of the number of tokens generated for each of these configurations. For example, with 10 frames and speech-to-text enabled, Gemma produces about 650 tokens at the median, while NVLLM produces about 250 tokens.

Now, depending on how conservative or relaxed the provider wants to be or how many resources they may want to provision, they can approximate how many tokens a new request for a given configuration of the video QA workflow would produce and accordingly provision resources.

This is then composed with the hardware-level profiles to determine how much time it would take to produce that number of tokens under various hardware configurations. So, we can select the appropriate configuration given the latency or cost SLOs.

And to handle deviations from this approximation or spikes in demand, Murakab uses an auto scaler that can always spin up more resources, but the optimizer can rebalance the system at a later point in time.

So, again, if we choose the Gemma model with 10 frames and speech-to-text enabled, at the 90th percentile, the requests would generate about 700 tokens.

And let's say we operate with eight H100 GPUs at 2,000 TPS, it would take about, 0.03 seconds * 700 tokens, so in total about 21 seconds to produce these responses. And that's just a toy example for now.

Similarly, we can also estimate the energy expenditure of this hardware configuration to ensure we are under our energy budget.

As an overview of the optimizer, we formulate it as a mixed integer linear program with multiple objectives. For example, minimizing operational cost, the number of GPUs, or the total energy expenditure under a given constraint of satisfying the total load given the SLOs.

The optimizer takes as input the profiles, arrival patterns, and resource availability information of the system.

And outputs a resource allocation plan, for example, how many GPUs, CPUs to allocate, which models and tools to run on each of them, and how to configure the workflow knobs and model selection to now use this optimized plan.

So, the optimizer, based on the profiles and SLOs, plugs the values in, and now the workflow is executable.

Finally, we're going to be able to optimize workflows jointly with respect to other workflows being served in the system to unlock even greater efficiency by multiplexing resources, something which a lack of visibility across the stack severely limits.

So, this opens up a richer optimization space, rather than looking at each workflow in isolation.

As we saw, each workflow SLO combination may have many feasible configurations.

Murakab's optimizer tries to steer workflows towards overlapping configurations, those that can use the same models and tools, so that it can achieve better allocation of resources and amortize the cost across users of different workflows.

It adapts to dynamism in the load or demand for different workflow SLO combinations, and the ever-changing resource availability in cloud platforms. For example, spot instances may come and go, energy sources may change, etc.

So, the key shift in paradigm is this.

There is more cross-stack visibility.

The orchestrator in Murakab now gets the workflow specification without any configurations baked into it, and also gets SLOs at query time, and can also observe the state of the platform in terms of what pieces are deployed, what resources are available, what the current per workflow load is, etc. In other words, it tracks all the sources of dynamism in real production scenarios.

And leveraging this greater visibility, it performs runtime optimization of all workflows to unlock much greater resource efficiency while satisfying SLOs.

So finally, let's take a quick look at how much resource efficiency does Marakoud unlock.

So we will look at a production workload from Azure for two LLM services, which we map onto two agentic workflows. One for video QA, and the other for a coding workflow, much like cloud code.

We assign 70% of the requests from each workflow to be high accuracy, while the remaining 30% to be latency critical.

We compare to LangGraph as a baseline, given that LangGraph is static by nature. We also evaluated given the same auto scaling policy that we used for Marakoud for a fairer comparison. Then we look at how Marakoud performs when optimizing workflows in isolation, and then also when it jointly optimizes across different workflows.

So we will look at the efficiency from two metrics, energy and cost. Here we overlay the aggregated workload trend for reference, that is shown in gray.

First, we run with default LangGraph with a hand-tuned configuration that tries to balance between accuracy and latency. But by default, this workflow is This framework is SLO agnostic, so all requests are treated the same way.

You can see how energy consumption varies with load, but the cost remains flat because of a static resource allocation.

With auto scaling added to LangGraph, it is able to turn on or off instances, much like Marakoud, in response to changing load, so that the cost picture now looks much better. But again, remember that it cannot reconfigure any of the workflows once deployed, so they remain static.

Marakoud, on the other hand, having the ability to do SLO aware orchestration, can optimize to the requirements of the different SLO classes, while also picking configurations that are most resource efficient. So without violating SLOs, it is able to run with much lower energy and cost footprint.

And given the ability to look across different workflows, it can adapt them at runtime given the demand for other workflows and consolidate them onto fewer, more efficient hardware and amortize the cost as well.

So, it brings down both the energy and cost even further.

All in all, One A Cloud gives you about 3.7 x improvement in energy consumption, 4.3 x improvement in cost, and about 2.8 x fewer GPUs.

So, in summary, in this talk we saw that today's agentic workflow execution is siloed and loses out opportunities for better optimization.

Decoupling the what from the how in the workflow definition opens up a vast space of automated runtime optimizations.

And visibility into all layers of the stack allows joint optimization across multi-tenant workflows and cloud platforms for much greater resource efficiency.

Thank you.