Conditionals
In many automations conditionals are important control flow elements. Because the ADK uses regular Python, conditionals can be expressed as simple if / else statements inside your automation code. If promises (i.e. step outputs) are used in conditionals, special considerations apply.
Static vs. dynamic conditionals
A static conditional is a conditional that can be evaluated immediately, because it does not involve a promise (i.e. a step output). In other words, when executing the automation script, the Python interpreter can decide immediately whether to follow the "if" or the "else" branch of the automation, without waiting for a step to finish execution first.
A dynamic conditional, on the other hand, is a conditional whose outcome depends on a promise, i.e. the output of another step. Only after the step of this promise finished execution, the Python interpreter knows how to proceed.
Static conditionals
Consider the following example that involves a static conditional. For a list of numbers, the goal is to compute squares for even numbers and doubles for odd numbers using steps.
Final result should be the sum of individual results. A sleep statement inside both steps' execute function again simulates long-running external service calls to better study the impact of parallelized execution.
from freyja import Step, Automation, Input, Output
from time import sleep
class Square(Step):
num = Input(int)
squared = Output(int)
def execute(self):
sleep(3)
self.squared = self.num * self.num
class Double(Step):
num = Input(int)
doubled = Output(int)
def execute(self):
sleep(3)
self.doubled = self.num * 2
class Main(Step):
def execute(self):
results = []
for i in range(1, 5):
if i % 2 == 0: # even
result = Square(f"Square-{i}", num=i).squared
else: # odd
result = Double(f"Double-{i}", num=i).doubled
results.append(result)
print(f"Result is {sum(results)}")
if __name__ == "__main__":
Automation(Main).run()
When executed, the following shows in the console.
$ python static_conditional.py run
2019-03-19 20:58:00,085 INFO [ freyja.log: 68]: (MainThread ) Logging configured
2019-03-19 20:58:00,086 INFO [freyja.graph: 362]: (MainThread ) Process: 14214
2019-03-19 20:58:00,086 INFO [freyja.graph: 489]: (MainThread ) Instantiating Step <Main "main">
2019-03-19 20:58:00,086 INFO [freyja.graph: 217]: (MainThread ) Step <Main ("main")> queued for execution
2019-03-19 20:58:00,086 INFO [freyja.graph: 648]: (main ) Initiating execution for for: Step <Main ("main")>
2019-03-19 20:58:00,087 INFO [freyja.graph: 658]: (main ) Execution started for: Step <Main ("main")>
2019-03-19 20:58:00,087 INFO [freyja.graph: 664]: (main ) RUNNING: main
2019-03-19 20:58:00,087 INFO [freyja.graph: 489]: (main ) Instantiating Step <Double "Double-1">
2019-03-19 20:58:00,087 INFO [freyja.graph: 217]: (main ) Step <Double ("main.Double-1")> queued for execution
2019-03-19 20:58:00,087 INFO [freyja.graph: 489]: (main ) Instantiating Step <Square "Square-2">
2019-03-19 20:58:00,088 INFO [freyja.graph: 217]: (main ) Step <Square ("main.Square-2")> queued for execution
2019-03-19 20:58:00,088 INFO [freyja.graph: 489]: (main ) Instantiating Step <Double "Double-3">
2019-03-19 20:58:00,087 INFO [freyja.graph: 648]: (main.Double-1) Initiating execution for for: Step <Double ("main.Double-1")>
2019-03-19 20:58:00,089 INFO [freyja.graph: 217]: (main ) Step <Double ("main.Double-3")> queued for execution
2019-03-19 20:58:00,089 INFO [freyja.graph: 658]: (main.Double-1) Execution started for: Step <Double ("main.Double-1")>
2019-03-19 20:58:00,090 INFO [freyja.graph: 489]: (main ) Instantiating Step <Square "Square-4">
2019-03-19 20:58:00,092 INFO [freyja.graph: 648]: (main.Double-3) Initiating execution for for: Step <Double ("main.Double-3")>
2019-03-19 20:58:00,092 INFO [freyja.graph: 664]: (main.Double-1) RUNNING: Double-1
2019-03-19 20:58:00,092 INFO [freyja.graph: 648]: (main.Square-2) Initiating execution for for: Step <Square ("main.Square-2")>
2019-03-19 20:58:00,093 INFO [freyja.graph: 217]: (main ) Step <Square ("main.Square-4")> queued for execution
2019-03-19 20:58:00,093 INFO [freyja.graph: 658]: (main.Double-3) Execution started for: Step <Double ("main.Double-3")>
2019-03-19 20:58:00,093 INFO [freyja.graph: 658]: (main.Square-2) Execution started for: Step <Square ("main.Square-2")>
2019-03-19 20:58:00,094 INFO [freyja.graph: 648]: (main.Square-4) Initiating execution for for: Step <Square ("main.Square-4")>
2019-03-19 20:58:00,094 INFO [freyja.graph: 664]: (main.Double-3) RUNNING: Double-3
2019-03-19 20:58:00,094 INFO [freyja.graph: 658]: (main.Square-4) Execution started for: Step <Square ("main.Square-4")>
2019-03-19 20:58:00,094 INFO [freyja.graph: 664]: (main.Square-2) RUNNING: Square-2
2019-03-19 20:58:00,095 INFO [freyja.graph: 664]: (main.Square-4) RUNNING: Square-4
2019-03-19 20:58:03,097 INFO [freyja.graph: 690]: (main.Square-2) Execution finished for: Step <Square ("main.Square-2")>
2019-03-19 20:58:03,099 INFO [freyja.graph: 690]: (main.Square-4) Execution finished for: Step <Square ("main.Square-4")>
Result is 28
2019-03-19 20:58:03,100 INFO [freyja.graph: 690]: (main.Double-1) Execution finished for: Step <Double ("main.Double-1")>
2019-03-19 20:58:03,105 INFO [freyja.graph: 690]: (main.Double-3) Execution finished for: Step <Double ("main.Double-3")>
2019-03-19 20:58:03,106 INFO [freyja.graph: 690]: (main ) Execution finished for: Step <Main ("main")>
2019-03-19 20:58:03,111 INFO [freyja.graph: 114]: (Executor-main) Executor done
2019-03-19 20:58:03,114 INFO [freyja.graph: 406]: (MainThread )
-----------------------------------------------------------------------
Execution summary:
Steps instantiated: 5
Steps incomplete: 0
Steps executed: 5
Steps failed: 0
-----------------------------------------------------------------------
The log shows that squares are computed for even numbers and doubles for odd numbers. The final result is 28. The script finishes in three seconds, which indicates that all steps executed in parallel, as expected.
Dynamic conditionals
Now let's look at a different example that involves a dynamic conditional. We want to compute the sum of squares for all squares that are greater than 8. Again, this can be scripted with a simple if statement inside the loop.
from freyja import Step, Automation, Input, Output
from time import sleep
class Square(Step):
num = Input(int)
squared = Output(int)
def execute(self):
sleep(3)
self.squared = self.num * self.num
class SumOfSquares(Step):
def execute(self):
squares_greater_8 = []
for i in range(1, 5):
square = Square(f"Square-{i}", num=i)
if square.squared > 8:
squares_greater_8.append(square.squared)
print(f"Result is {sum(squares_greater_8)}")
if __name__ == "__main__":
Automation(SumOfSquares).run()
This time we see an unexpected behavior.
$ python dynamic_conditional_blocking.py run
2019-03-19 21:13:25,343 INFO [ freyja.log: 68]: (MainThread ) Logging configured
2019-03-19 21:13:25,343 INFO [freyja.graph: 362]: (MainThread ) Process: 14939
2019-03-19 21:13:25,343 INFO [freyja.graph: 489]: (MainThread ) Instantiating Step <SumOfSquares "main">
2019-03-19 21:13:25,343 INFO [freyja.graph: 217]: (MainThread ) Step <SumOfSquares ("main")> queued for execution
2019-03-19 21:13:25,344 INFO [freyja.graph: 648]: (main ) Initiating execution for for: Step <SumOfSquares ("main")>
2019-03-19 21:13:25,344 INFO [freyja.graph: 658]: (main ) Execution started for: Step <SumOfSquares ("main")>
2019-03-19 21:13:25,344 INFO [freyja.graph: 664]: (main ) RUNNING: main
2019-03-19 21:13:25,344 INFO [freyja.graph: 489]: (main ) Instantiating Step <Square "Square-1">
2019-03-19 21:13:25,345 INFO [freyja.graph: 217]: (main ) Step <Square ("main.Square-1")> queued for execution
2019-03-19 21:13:25,345 INFO [freyja.graph: 648]: (main.Square-1) Initiating execution for for: Step <Square ("main.Square-1")>
2019-03-19 21:13:25,345 INFO [freyja.graph: 658]: (main.Square-1) Execution started for: Step <Square ("main.Square-1")>
2019-03-19 21:13:25,346 INFO [freyja.graph: 664]: (main.Square-1) RUNNING: Square-1
2019-03-19 21:13:28,351 INFO [freyja.graph: 690]: (main.Square-1) Execution finished for: Step <Square ("main.Square-1")>
2019-03-19 21:13:28,352 INFO [freyja.graph: 489]: (main ) Instantiating Step <Square "Square-2">
2019-03-19 21:13:28,355 INFO [freyja.graph: 217]: (main ) Step <Square ("main.Square-2")> queued for execution
2019-03-19 21:13:28,357 INFO [freyja.graph: 648]: (main.Square-2) Initiating execution for for: Step <Square ("main.Square-2")>
2019-03-19 21:13:28,359 INFO [freyja.graph: 658]: (main.Square-2) Execution started for: Step <Square ("main.Square-2")>
2019-03-19 21:13:28,360 INFO [freyja.graph: 664]: (main.Square-2) RUNNING: Square-2
2019-03-19 21:13:31,366 INFO [freyja.graph: 489]: (main ) Instantiating Step <Square "Square-3">
2019-03-19 21:13:31,367 INFO [freyja.graph: 690]: (main.Square-2) Execution finished for: Step <Square ("main.Square-2")>
2019-03-19 21:13:31,368 INFO [freyja.graph: 217]: (main ) Step <Square ("main.Square-3")> queued for execution
2019-03-19 21:13:31,372 INFO [freyja.graph: 648]: (main.Square-3) Initiating execution for for: Step <Square ("main.Square-3")>
2019-03-19 21:13:31,374 INFO [freyja.graph: 658]: (main.Square-3) Execution started for: Step <Square ("main.Square-3")>
2019-03-19 21:13:31,375 INFO [freyja.graph: 664]: (main.Square-3) RUNNING: Square-3
2019-03-19 21:13:34,381 INFO [freyja.graph: 690]: (main.Square-3) Execution finished for: Step <Square ("main.Square-3")>
2019-03-19 21:13:34,382 INFO [freyja.graph: 489]: (main ) Instantiating Step <Square "Square-4">
2019-03-19 21:13:34,385 INFO [freyja.graph: 217]: (main ) Step <Square ("main.Square-4")> queued for execution
2019-03-19 21:13:34,387 INFO [freyja.graph: 648]: (main.Square-4) Initiating execution for for: Step <Square ("main.Square-4")>
2019-03-19 21:13:34,389 INFO [freyja.graph: 658]: (main.Square-4) Execution started for: Step <Square ("main.Square-4")>
2019-03-19 21:13:34,390 INFO [freyja.graph: 664]: (main.Square-4) RUNNING: Square-4
Result is 25
2019-03-19 21:13:37,396 INFO [freyja.graph: 690]: (main.Square-4) Execution finished for: Step <Square ("main.Square-4")>
2019-03-19 21:13:37,397 INFO [freyja.graph: 690]: (main ) Execution finished for: Step <SumOfSquares ("main")>
2019-03-19 21:13:37,399 INFO [freyja.graph: 114]: (Executor-main) Executor done
2019-03-19 21:13:37,402 INFO [freyja.graph: 406]: (MainThread )
-----------------------------------------------------------------------
Execution summary:
Steps instantiated: 5
Steps incomplete: 0
Steps executed: 5
Steps failed: 0
-----------------------------------------------------------------------
Steps executed one after the other, each with three seconds delay. Total execution time was 12 seconds (4 x 3s). In other words, parallelization did not work.
To understand what is going on we need to remember that the conditional evaluates a promise. In order to test if the square is greater than 8, the square needs to be computed first.
This means that at each iteration of the loop the Python interpreter has to stop at the conditional and wait for the step to finish execution.
Otherwise, the interpreter will not know how to continue with script execution after this point.
Unfortunately, this also means that additional steps are not being instantiated, are not being added to the execution graph, and hence not being run in parallel.
Resolving execution blocks (caused by dynamic conditionals)
Dynamic conditionals tend to be less common than static conditionals. Should they be required, the easiest way to work around execution blocks is to pull the conditional out of the loop into a separate step.
When put in its own step, each conditional will evaluate in its own thread, and therefore does not block the loop of the parent step.
The following script demonstrates this technique, turning a blocking into a non-blocking automation script.
from freyja import Step, Automation, Input, Output
from time import sleep
class BigSquare(Step):
num = Input(int)
squared = Output(int)
def execute(self):
sleep(3)
squared = self.num * self.num
if squared > 8:
self.squared = squared
else:
self.squared = 0
class SumOfSquares(Step):
def execute(self):
squares_greater_8 = []
for i in range(1, 5):
square = BigSquare(f"BigSquare-{i}", num=i)
squares_greater_8.append(square.squared)
print(f"Result is {sum(squares_greater_8)}")
if __name__ == "__main__":
Automation(SumOfSquares).run()
This time we observe the expected behavior:
$ python dynamic_conditional_nonblocking.py run
2019-04-10 17:11:40,810 INFO [ freyja.log: 68]: (MainThread ) Logging configured
2019-04-10 17:11:40,811 INFO [freyja.graph: 362]: (MainThread ) Process: 19336
2019-04-10 17:11:40,811 INFO [freyja.graph: 489]: (MainThread ) Instantiating Step <SumOfSquares "main">
2019-04-10 17:11:40,811 INFO [freyja.graph: 217]: (MainThread ) Step <SumOfSquares ("main")> queued for execution
2019-04-10 17:11:40,811 INFO [freyja.graph: 648]: (main ) Initiating execution for for: Step <SumOfSquares ("main")>
2019-04-10 17:11:40,812 INFO [freyja.graph: 658]: (main ) Execution started for: Step <SumOfSquares ("main")>
2019-04-10 17:11:40,812 INFO [freyja.graph: 664]: (main ) RUNNING: main
2019-04-10 17:11:40,812 INFO [freyja.graph: 489]: (main ) Instantiating Step <BigSquare "BigSquare-1">
2019-04-10 17:11:40,812 INFO [freyja.graph: 217]: (main ) Step <BigSquare ("main.BigSquare-1")> queued for execution
2019-04-10 17:11:40,812 INFO [freyja.graph: 489]: (main ) Instantiating Step <BigSquare "BigSquare-2">
2019-04-10 17:11:40,813 INFO [freyja.graph: 648]: (main.BigSquare-1) Initiating execution for for: Step <BigSquare ("main.BigSquare-1")>
2019-04-10 17:11:40,813 INFO [freyja.graph: 217]: (main ) Step <BigSquare ("main.BigSquare-2")> queued for execution
2019-04-10 17:11:40,813 INFO [freyja.graph: 658]: (main.BigSquare-1) Execution started for: Step <BigSquare ("main.BigSquare-1")>
2019-04-10 17:11:40,814 INFO [freyja.graph: 648]: (main.BigSquare-2) Initiating execution for for: Step <BigSquare ("main.BigSquare-2")>
2019-04-10 17:11:40,813 INFO [freyja.graph: 489]: (main ) Instantiating Step <BigSquare "BigSquare-3">
2019-04-10 17:11:40,814 INFO [freyja.graph: 658]: (main.BigSquare-2) Execution started for: Step <BigSquare ("main.BigSquare-2")>
2019-04-10 17:11:40,814 INFO [freyja.graph: 664]: (main.BigSquare-1) RUNNING: BigSquare-1
2019-04-10 17:11:40,814 INFO [freyja.graph: 217]: (main ) Step <BigSquare ("main.BigSquare-3")> queued for execution
2019-04-10 17:11:40,815 INFO [freyja.graph: 489]: (main ) Instantiating Step <BigSquare "BigSquare-4">
2019-04-10 17:11:40,815 INFO [freyja.graph: 664]: (main.BigSquare-2) RUNNING: BigSquare-2
2019-04-10 17:11:40,815 INFO [freyja.graph: 648]: (main.BigSquare-3) Initiating execution for for: Step <BigSquare ("main.BigSquare-3")>
2019-04-10 17:11:40,815 INFO [freyja.graph: 217]: (main ) Step <BigSquare ("main.BigSquare-4")> queued for execution
2019-04-10 17:11:40,816 INFO [freyja.graph: 658]: (main.BigSquare-3) Execution started for: Step <BigSquare ("main.BigSquare-3")>
2019-04-10 17:11:40,816 INFO [freyja.graph: 664]: (main.BigSquare-3) RUNNING: BigSquare-3
2019-04-10 17:11:40,816 INFO [freyja.graph: 648]: (main.BigSquare-4) Initiating execution for for: Step <BigSquare ("main.BigSquare-4")>
2019-04-10 17:11:40,817 INFO [freyja.graph: 658]: (main.BigSquare-4) Execution started for: Step <BigSquare ("main.BigSquare-4")>
2019-04-10 17:11:40,817 INFO [freyja.graph: 664]: (main.BigSquare-4) RUNNING: BigSquare-4
2019-04-10 17:11:43,817 INFO [freyja.graph: 690]: (main.BigSquare-2) Execution finished for: Step <BigSquare ("main.BigSquare-2")>
2019-04-10 17:11:43,818 INFO [freyja.graph: 690]: (main.BigSquare-1) Execution finished for: Step <BigSquare ("main.BigSquare-1")>
2019-04-10 17:11:43,820 INFO [freyja.graph: 690]: (main.BigSquare-3) Execution finished for: Step <BigSquare ("main.BigSquare-3")>
Result is 25
2019-04-10 17:11:43,820 INFO [freyja.graph: 690]: (main.BigSquare-4) Execution finished for: Step <BigSquare ("main.BigSquare-4")>
2019-04-10 17:11:43,821 INFO [freyja.graph: 690]: (main ) Execution finished for: Step <SumOfSquares ("main")>
2019-04-10 17:11:43,821 INFO [freyja.graph: 114]: (Executor-main) Executor done
2019-04-10 17:11:43,822 INFO [freyja.graph: 406]: (MainThread )
-----------------------------------------------------------------------
Execution summary:
Steps instantiated: 5
Steps incomplete: 0
Steps executed: 5
Steps failed: 0
-----------------------------------------------------------------------
We get the same result (25) but total execution time is just 3 seconds, which means all steps are running again in parallel.
Whenever your automation script does not parallelize as expected, watch out for execution blocks and remember this simple trick as one possibility to resolve them.
A more comprehensive automation example that features the implementation of both a static and dynamic conditional can be found in our GitHub repository.
Updated over 5 years ago