Conditionals

In many automations conditionals are important control flow elements. Because the ADK uses regular Python, conditionals can be expressed as simple if / else statements inside your automation code. If promises (i.e. step outputs) are used in conditionals, special considerations apply.

Static vs. dynamic conditionals

A static conditional is a conditional that can be evaluated immediately, because it does not involve a promise (i.e. a step output). In other words, when executing the automation script, the Python interpreter can decide immediately whether to follow the "if" or the "else" branch of the automation, without waiting for a step to finish execution first.

A dynamic conditional, on the other hand, is a conditional whose outcome depends on a promise, i.e. the output of another step. Only after the step of this promise finished execution, the Python interpreter knows how to proceed.

Static conditionals

Consider the following example that involves a static conditional. For a list of numbers, the goal is to compute squares for even numbers and doubles for odd numbers using steps.

Final result should be the sum of individual results. A sleep statement inside both steps' execute function again simulates long-running external service calls to better study the impact of parallelized execution.

from freyja import Step, Automation, Input, Output
from time import sleep
 
 
class Square(Step):
    num = Input(int)
    squared = Output(int)
 
    def execute(self):
        sleep(3)
        self.squared = self.num * self.num
 
 
class Double(Step):
    num = Input(int)
    doubled = Output(int)
 
    def execute(self):
        sleep(3)
        self.doubled = self.num * 2
 
 
class Main(Step):
    def execute(self):
        results = []
        for i in range(1, 5):
            if i % 2 == 0:  # even
                result = Square(f"Square-{i}", num=i).squared
            else:  # odd
                result = Double(f"Double-{i}", num=i).doubled
            results.append(result)
        print(f"Result is {sum(results)}")
 
 
if __name__ == "__main__":
    Automation(Main).run()

When executed, the following shows in the console.

$ python static_conditional.py run
2019-03-19 20:58:00,085    INFO [  freyja.log:  68]: (MainThread  ) Logging configured
2019-03-19 20:58:00,086    INFO [freyja.graph: 362]: (MainThread  ) Process: 14214
2019-03-19 20:58:00,086    INFO [freyja.graph: 489]: (MainThread  ) Instantiating Step <Main "main">
2019-03-19 20:58:00,086    INFO [freyja.graph: 217]: (MainThread  ) Step <Main ("main")> queued for execution
2019-03-19 20:58:00,086    INFO [freyja.graph: 648]: (main        ) Initiating execution for for: Step <Main ("main")>
2019-03-19 20:58:00,087    INFO [freyja.graph: 658]: (main        ) Execution started for: Step <Main ("main")>
2019-03-19 20:58:00,087    INFO [freyja.graph: 664]: (main        ) RUNNING: main
2019-03-19 20:58:00,087    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Double "Double-1">
2019-03-19 20:58:00,087    INFO [freyja.graph: 217]: (main        ) Step <Double ("main.Double-1")> queued for execution
2019-03-19 20:58:00,087    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Square "Square-2">
2019-03-19 20:58:00,088    INFO [freyja.graph: 217]: (main        ) Step <Square ("main.Square-2")> queued for execution
2019-03-19 20:58:00,088    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Double "Double-3">
2019-03-19 20:58:00,087    INFO [freyja.graph: 648]: (main.Double-1) Initiating execution for for: Step <Double ("main.Double-1")>
2019-03-19 20:58:00,089    INFO [freyja.graph: 217]: (main        ) Step <Double ("main.Double-3")> queued for execution
2019-03-19 20:58:00,089    INFO [freyja.graph: 658]: (main.Double-1) Execution started for: Step <Double ("main.Double-1")>
2019-03-19 20:58:00,090    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Square "Square-4">
2019-03-19 20:58:00,092    INFO [freyja.graph: 648]: (main.Double-3) Initiating execution for for: Step <Double ("main.Double-3")>
2019-03-19 20:58:00,092    INFO [freyja.graph: 664]: (main.Double-1) RUNNING: Double-1
2019-03-19 20:58:00,092    INFO [freyja.graph: 648]: (main.Square-2) Initiating execution for for: Step <Square ("main.Square-2")>
2019-03-19 20:58:00,093    INFO [freyja.graph: 217]: (main        ) Step <Square ("main.Square-4")> queued for execution
2019-03-19 20:58:00,093    INFO [freyja.graph: 658]: (main.Double-3) Execution started for: Step <Double ("main.Double-3")>
2019-03-19 20:58:00,093    INFO [freyja.graph: 658]: (main.Square-2) Execution started for: Step <Square ("main.Square-2")>
2019-03-19 20:58:00,094    INFO [freyja.graph: 648]: (main.Square-4) Initiating execution for for: Step <Square ("main.Square-4")>
2019-03-19 20:58:00,094    INFO [freyja.graph: 664]: (main.Double-3) RUNNING: Double-3
2019-03-19 20:58:00,094    INFO [freyja.graph: 658]: (main.Square-4) Execution started for: Step <Square ("main.Square-4")>
2019-03-19 20:58:00,094    INFO [freyja.graph: 664]: (main.Square-2) RUNNING: Square-2
2019-03-19 20:58:00,095    INFO [freyja.graph: 664]: (main.Square-4) RUNNING: Square-4
2019-03-19 20:58:03,097    INFO [freyja.graph: 690]: (main.Square-2) Execution finished for: Step <Square ("main.Square-2")>
2019-03-19 20:58:03,099    INFO [freyja.graph: 690]: (main.Square-4) Execution finished for: Step <Square ("main.Square-4")>
Result is 28
2019-03-19 20:58:03,100    INFO [freyja.graph: 690]: (main.Double-1) Execution finished for: Step <Double ("main.Double-1")>
2019-03-19 20:58:03,105    INFO [freyja.graph: 690]: (main.Double-3) Execution finished for: Step <Double ("main.Double-3")>
2019-03-19 20:58:03,106    INFO [freyja.graph: 690]: (main        ) Execution finished for: Step <Main ("main")>
2019-03-19 20:58:03,111    INFO [freyja.graph: 114]: (Executor-main) Executor done
2019-03-19 20:58:03,114    INFO [freyja.graph: 406]: (MainThread  )
-----------------------------------------------------------------------
Execution summary:
    Steps instantiated: 5
    Steps incomplete:   0
    Steps executed:     5
    Steps failed:       0
-----------------------------------------------------------------------

The log shows that squares are computed for even numbers and doubles for odd numbers. The final result is 28. The script finishes in three seconds, which indicates that all steps executed in parallel, as expected.

Dynamic conditionals

Now let's look at a different example that involves a dynamic conditional. We want to compute the sum of squares for all squares that are greater than 8. Again, this can be scripted with a simple if statement inside the loop.

from freyja import Step, Automation, Input, Output
from time import sleep
 
 
class Square(Step):
    num = Input(int)
    squared = Output(int)
 
    def execute(self):
        sleep(3)
        self.squared = self.num * self.num
 
 
class SumOfSquares(Step):
    def execute(self):
        squares_greater_8 = []
        for i in range(1, 5):
            square = Square(f"Square-{i}", num=i)
            if square.squared > 8:
                squares_greater_8.append(square.squared)
        print(f"Result is {sum(squares_greater_8)}")
 
 
if __name__ == "__main__":
    Automation(SumOfSquares).run()

This time we see an unexpected behavior.

$ python dynamic_conditional_blocking.py run
2019-03-19 21:13:25,343    INFO [  freyja.log:  68]: (MainThread  ) Logging configured
2019-03-19 21:13:25,343    INFO [freyja.graph: 362]: (MainThread  ) Process: 14939
2019-03-19 21:13:25,343    INFO [freyja.graph: 489]: (MainThread  ) Instantiating Step <SumOfSquares "main">
2019-03-19 21:13:25,343    INFO [freyja.graph: 217]: (MainThread  ) Step <SumOfSquares ("main")> queued for execution
2019-03-19 21:13:25,344    INFO [freyja.graph: 648]: (main        ) Initiating execution for for: Step <SumOfSquares ("main")>
2019-03-19 21:13:25,344    INFO [freyja.graph: 658]: (main        ) Execution started for: Step <SumOfSquares ("main")>
2019-03-19 21:13:25,344    INFO [freyja.graph: 664]: (main        ) RUNNING: main
2019-03-19 21:13:25,344    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Square "Square-1">
2019-03-19 21:13:25,345    INFO [freyja.graph: 217]: (main        ) Step <Square ("main.Square-1")> queued for execution
2019-03-19 21:13:25,345    INFO [freyja.graph: 648]: (main.Square-1) Initiating execution for for: Step <Square ("main.Square-1")>
2019-03-19 21:13:25,345    INFO [freyja.graph: 658]: (main.Square-1) Execution started for: Step <Square ("main.Square-1")>
2019-03-19 21:13:25,346    INFO [freyja.graph: 664]: (main.Square-1) RUNNING: Square-1
2019-03-19 21:13:28,351    INFO [freyja.graph: 690]: (main.Square-1) Execution finished for: Step <Square ("main.Square-1")>
2019-03-19 21:13:28,352    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Square "Square-2">
2019-03-19 21:13:28,355    INFO [freyja.graph: 217]: (main        ) Step <Square ("main.Square-2")> queued for execution
2019-03-19 21:13:28,357    INFO [freyja.graph: 648]: (main.Square-2) Initiating execution for for: Step <Square ("main.Square-2")>
2019-03-19 21:13:28,359    INFO [freyja.graph: 658]: (main.Square-2) Execution started for: Step <Square ("main.Square-2")>
2019-03-19 21:13:28,360    INFO [freyja.graph: 664]: (main.Square-2) RUNNING: Square-2
2019-03-19 21:13:31,366    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Square "Square-3">
2019-03-19 21:13:31,367    INFO [freyja.graph: 690]: (main.Square-2) Execution finished for: Step <Square ("main.Square-2")>
2019-03-19 21:13:31,368    INFO [freyja.graph: 217]: (main        ) Step <Square ("main.Square-3")> queued for execution
2019-03-19 21:13:31,372    INFO [freyja.graph: 648]: (main.Square-3) Initiating execution for for: Step <Square ("main.Square-3")>
2019-03-19 21:13:31,374    INFO [freyja.graph: 658]: (main.Square-3) Execution started for: Step <Square ("main.Square-3")>
2019-03-19 21:13:31,375    INFO [freyja.graph: 664]: (main.Square-3) RUNNING: Square-3
2019-03-19 21:13:34,381    INFO [freyja.graph: 690]: (main.Square-3) Execution finished for: Step <Square ("main.Square-3")>
2019-03-19 21:13:34,382    INFO [freyja.graph: 489]: (main        ) Instantiating Step <Square "Square-4">
2019-03-19 21:13:34,385    INFO [freyja.graph: 217]: (main        ) Step <Square ("main.Square-4")> queued for execution
2019-03-19 21:13:34,387    INFO [freyja.graph: 648]: (main.Square-4) Initiating execution for for: Step <Square ("main.Square-4")>
2019-03-19 21:13:34,389    INFO [freyja.graph: 658]: (main.Square-4) Execution started for: Step <Square ("main.Square-4")>
2019-03-19 21:13:34,390    INFO [freyja.graph: 664]: (main.Square-4) RUNNING: Square-4
Result is 25
2019-03-19 21:13:37,396    INFO [freyja.graph: 690]: (main.Square-4) Execution finished for: Step <Square ("main.Square-4")>
2019-03-19 21:13:37,397    INFO [freyja.graph: 690]: (main        ) Execution finished for: Step <SumOfSquares ("main")>
2019-03-19 21:13:37,399    INFO [freyja.graph: 114]: (Executor-main) Executor done
2019-03-19 21:13:37,402    INFO [freyja.graph: 406]: (MainThread  )
-----------------------------------------------------------------------
Execution summary:
    Steps instantiated: 5
    Steps incomplete:   0
    Steps executed:     5
    Steps failed:       0
-----------------------------------------------------------------------

Steps executed one after the other, each with three seconds delay. Total execution time was 12 seconds (4 x 3s). In other words, parallelization did not work.

To understand what is going on we need to remember that the conditional evaluates a promise. In order to test if the square is greater than 8, the square needs to be computed first.

This means that at each iteration of the loop the Python interpreter has to stop at the conditional and wait for the step to finish execution.

Otherwise, the interpreter will not know how to continue with script execution after this point.
Unfortunately, this also means that additional steps are not being instantiated, are not being added to the execution graph, and hence not being run in parallel.

Resolving execution blocks (caused by dynamic conditionals)

Dynamic conditionals tend to be less common than static conditionals. Should they be required, the easiest way to work around execution blocks is to pull the conditional out of the loop into a separate step.

When put in its own step, each conditional will evaluate in its own thread, and therefore does not block the loop of the parent step.

The following script demonstrates this technique, turning a blocking into a non-blocking automation script.

from freyja import Step, Automation, Input, Output
from time import sleep
 
 
class BigSquare(Step):
    num = Input(int)
    squared = Output(int)
 
    def execute(self):
        sleep(3)
        squared = self.num * self.num
        if squared > 8:
            self.squared = squared
        else:
            self.squared = 0
 
 
class SumOfSquares(Step):
    def execute(self):
        squares_greater_8 = []
        for i in range(1, 5):
            square = BigSquare(f"BigSquare-{i}", num=i)
            squares_greater_8.append(square.squared)
        print(f"Result is {sum(squares_greater_8)}")
 
 
if __name__ == "__main__":
    Automation(SumOfSquares).run()

This time we observe the expected behavior:

$ python dynamic_conditional_nonblocking.py run
2019-04-10 17:11:40,810    INFO [  freyja.log:  68]: (MainThread  ) Logging configured
2019-04-10 17:11:40,811    INFO [freyja.graph: 362]: (MainThread  ) Process: 19336
2019-04-10 17:11:40,811    INFO [freyja.graph: 489]: (MainThread  ) Instantiating Step <SumOfSquares "main">
2019-04-10 17:11:40,811    INFO [freyja.graph: 217]: (MainThread  ) Step <SumOfSquares ("main")> queued for execution
2019-04-10 17:11:40,811    INFO [freyja.graph: 648]: (main        ) Initiating execution for for: Step <SumOfSquares ("main")>
2019-04-10 17:11:40,812    INFO [freyja.graph: 658]: (main        ) Execution started for: Step <SumOfSquares ("main")>
2019-04-10 17:11:40,812    INFO [freyja.graph: 664]: (main        ) RUNNING: main
2019-04-10 17:11:40,812    INFO [freyja.graph: 489]: (main        ) Instantiating Step <BigSquare "BigSquare-1">
2019-04-10 17:11:40,812    INFO [freyja.graph: 217]: (main        ) Step <BigSquare ("main.BigSquare-1")> queued for execution
2019-04-10 17:11:40,812    INFO [freyja.graph: 489]: (main        ) Instantiating Step <BigSquare "BigSquare-2">
2019-04-10 17:11:40,813    INFO [freyja.graph: 648]: (main.BigSquare-1) Initiating execution for for: Step <BigSquare ("main.BigSquare-1")>
2019-04-10 17:11:40,813    INFO [freyja.graph: 217]: (main        ) Step <BigSquare ("main.BigSquare-2")> queued for execution
2019-04-10 17:11:40,813    INFO [freyja.graph: 658]: (main.BigSquare-1) Execution started for: Step <BigSquare ("main.BigSquare-1")>
2019-04-10 17:11:40,814    INFO [freyja.graph: 648]: (main.BigSquare-2) Initiating execution for for: Step <BigSquare ("main.BigSquare-2")>
2019-04-10 17:11:40,813    INFO [freyja.graph: 489]: (main        ) Instantiating Step <BigSquare "BigSquare-3">
2019-04-10 17:11:40,814    INFO [freyja.graph: 658]: (main.BigSquare-2) Execution started for: Step <BigSquare ("main.BigSquare-2")>
2019-04-10 17:11:40,814    INFO [freyja.graph: 664]: (main.BigSquare-1) RUNNING: BigSquare-1
2019-04-10 17:11:40,814    INFO [freyja.graph: 217]: (main        ) Step <BigSquare ("main.BigSquare-3")> queued for execution
2019-04-10 17:11:40,815    INFO [freyja.graph: 489]: (main        ) Instantiating Step <BigSquare "BigSquare-4">
2019-04-10 17:11:40,815    INFO [freyja.graph: 664]: (main.BigSquare-2) RUNNING: BigSquare-2
2019-04-10 17:11:40,815    INFO [freyja.graph: 648]: (main.BigSquare-3) Initiating execution for for: Step <BigSquare ("main.BigSquare-3")>
2019-04-10 17:11:40,815    INFO [freyja.graph: 217]: (main        ) Step <BigSquare ("main.BigSquare-4")> queued for execution
2019-04-10 17:11:40,816    INFO [freyja.graph: 658]: (main.BigSquare-3) Execution started for: Step <BigSquare ("main.BigSquare-3")>
2019-04-10 17:11:40,816    INFO [freyja.graph: 664]: (main.BigSquare-3) RUNNING: BigSquare-3
2019-04-10 17:11:40,816    INFO [freyja.graph: 648]: (main.BigSquare-4) Initiating execution for for: Step <BigSquare ("main.BigSquare-4")>
2019-04-10 17:11:40,817    INFO [freyja.graph: 658]: (main.BigSquare-4) Execution started for: Step <BigSquare ("main.BigSquare-4")>
2019-04-10 17:11:40,817    INFO [freyja.graph: 664]: (main.BigSquare-4) RUNNING: BigSquare-4
2019-04-10 17:11:43,817    INFO [freyja.graph: 690]: (main.BigSquare-2) Execution finished for: Step <BigSquare ("main.BigSquare-2")>
2019-04-10 17:11:43,818    INFO [freyja.graph: 690]: (main.BigSquare-1) Execution finished for: Step <BigSquare ("main.BigSquare-1")>
2019-04-10 17:11:43,820    INFO [freyja.graph: 690]: (main.BigSquare-3) Execution finished for: Step <BigSquare ("main.BigSquare-3")>
Result is 25
2019-04-10 17:11:43,820    INFO [freyja.graph: 690]: (main.BigSquare-4) Execution finished for: Step <BigSquare ("main.BigSquare-4")>
2019-04-10 17:11:43,821    INFO [freyja.graph: 690]: (main        ) Execution finished for: Step <SumOfSquares ("main")>
2019-04-10 17:11:43,821    INFO [freyja.graph: 114]: (Executor-main) Executor done
2019-04-10 17:11:43,822    INFO [freyja.graph: 406]: (MainThread  )
-----------------------------------------------------------------------
Execution summary:
    Steps instantiated: 5
    Steps incomplete:   0
    Steps executed:     5
    Steps failed:       0
-----------------------------------------------------------------------

We get the same result (25) but total execution time is just 3 seconds, which means all steps are running again in parallel.

Whenever your automation script does not parallelize as expected, watch out for execution blocks and remember this simple trick as one possibility to resolve them.

A more comprehensive automation example that features the implementation of both a static and dynamic conditional can be found in our GitHub repository.