Refactoring #

Refactoring is a code change that does not change the external behavior of the Code.

It should be driven by change, not by personal opinions on code aesthetics.

Reasons and moments to refactor could be:

• If a new feature should be implemented, and there is no simple way to add the new code
• Reducing the future cost of change. Modularity can improve the speed in which new functionality can be added (if we are aware of the upcoming functionality)
• Emphasize architecture decisions that sometimes get buried under short term decisions
• Comprehensive Refactoring to understand complicated code
• Litter pickup to always leave the code cleaner that you encountered it
• Refactor as review style

Refactoring can always be performed step by step, with reliable tests running, and do not need a massive change of the code base with hours of trying to get a now completely different code running to the same tests. Every refactoring is possible in a loop of one line change, tests passing, and the next one line change. Need to refactor a property into a getter? Write the getter, replace property access by property access with the getter. Need to extract a function (and don’t want to use the IDEs refactor function)? Move it line by line into the function, while keeping the tests passing. Having a hard time with many variables? Replace them with function calls during refactoring to enable you to move slowly with passing tests.

Because refactoring does not change the external behavior, it can be completely separated from feature development, coining the phrase First make the change easy, then make the easy change. This emphasizes feature adding and refactoring are two steps, valuing different skills and knowledge, and aiming for different goals - different jobs that can be imagined as two hats be worn to different occasions, never simultaneously.

Refactoring and Performance #

Most performance issues are located in a very small fraction of the code. Most of the time, refactoring makes it easier to identify and tackle those bottlenecks, which in practise is often more relevant than performance drawbacks from refactoring e.g. running through the same loop twice to separate two steps of processing.

Code Smells #

Mysterious Name #

Naming is hard, and renaming a function, property, or variable is one of the most common refactorings. While it is easy to call out “rename it” as a solution, complicated names can often be a sign of deeper design issues.

def caliginous_string() -> str:
    return (f"""
      -------
      │  ?  │
    dbqpdbqpdbq
      ( o.o )
      @≥ - ≤@
    """)

Change the declaration of a function, property or variable #

Functions represents building blocks of the software system. A well-defined function name can replace the need of finding out what the function does and how it is used. This also applies to parameters, properties and variables - a clear names enables the future developer to understand the code, its purpose and how it is used without needing to read the implementation. The more widely something is used, the more important is the clarity of its name.

Duplicated Code #

Using the same code in multiple places can indicate a missed opportunity for abstraction. Abstracting would emphasize the differences between similar code, instead of costing the reader time to question if there is a difference. If the difference might even be big and consistent enough to be separated into multiple functions.

To ensure euphoric deduplication is not making the code more complex than it needs to be, the “Rule of three” can be used as a guidance to allow duplications until at least three variants did present themselves to understand how different variants might behave.

def twin_monster() -> str:
    return (f"""
      -------
      │  ?  │
    dbqpdbqpdbq
      ( ^.^ )
      @≥ - ≤@
    """)

Slide (Move) Statements #

Code is easier to understand when related statements are close to each other. Sliding statements means moving statements together. This can be a preparation for an Extract Function.

Long Function #

A long function hides where the main logic is defined, and will cause every reader time to parse through the entire function to understand it and where the new feature should be placed. This can be avoided by shortening and structuring the function in way, that makes it easy to understand and place a new feature. The main goal should not be to reach a maximum line number a function should have, but to separate intention and implementation. A function should have a specific reason and context to exist in, which might be a concrete implementation, or stating an intention that calls implementations.

from datetime import datetime


def long_function_monster() -> str:
    now = datetime.now()

    # Start building the snake
    snake = "    ___   \n"
    snake += "  >{_σ │  \n"

    # Changed to left turning snake by 2020
    if now > datetime(2020, 3, 15):
        snake += "      \ \  \n"

    # Rolling release in 2025
    if now > datetime(2025, 11, 1):
        snake += "   ____) )\n"
        if now < datetime(2025, 1, 10):
            snake += " / _____/ \n"
        if now < datetime(2025, 3, 1):
            snake += "( (___    \n"
        if now < datetime(2025, 5, 1):
            snake += " \___ \   \n"
        if now < datetime(2025, 7, 1):
            snake += "     \/   \n"

    # Feature flag with auto enable in 2022
    if now < datetime(2022, 6, 20):
        snake += " \___ \   \n"
        if now > datetime(2022, 9, 1):
            snake += "     \/   \n"
        else:
            snake += "  ___/ /  \n"

    if now > datetime(2023, 2, 14):
        snake += " / _____/   \n"

    if now < datetime(2023, 7, 4):
        snake += "( (___ __ \n"
        if now > datetime(2023, 10, 31):
            snake += " \_____  \\n"
            # Parse the override string
            pass

    # Special handling for empty results (added for client X)
    if now < datetime(2024, 4, 1):
        snake += " / _____/ \n"

    # Tail handling
    snake += "( (___    \n"
    snake += " \___ \   \n"
    snake += "     \/   \n"

    return snake

Similar to this, Large classes cause the same problems, which might or might not allow the extraction of whole class. Additionally, here identifying and replacing possible subclasses with different types might help, this way properties and functions that only belong to some subtypes are moved out of the way to only leave the things that are common and used among all subclasses.

Extract a Function #

A function symbolizes a differentiation between intent and implementation. Extracting a function means replacing the old code fragment with a function call named after the intent of the code fragment. The parameters of the extracted function highlight dependencies and differences between its usages. With this goal in mind, having a function that just hase one line of code can be valid if it describes the intent of the code fragment better than the code fragment itself. In that way extracting a function can replace the need of a comment.

A large block of code can have a clearer intention if the code blogs within conditionals are extracted, leaving just the conditional easily readable in the original function. How the conditions are structured can also have drawbacks and benefits. Sometimes it is a clean solution to have on consolidated check that replaces a big nested check can make things easier to understand than a big conditional with many branches.

Pulling up a method means moving a method from a subclass to its superclass. This is useful when the method is duplicated in multiple subclasses, or only differs in small parts, naming, or parameters.

One hint, that a function is too long and should be splitted. If a variable is the result of a complex calculation, but is then re-assigned, that can be a sign that the function and that variable is doing too much and might work better splitted.

Replace Variable, Property, or Parameter with Query #

Variables or Parameters can capture and name a value, but they also introduce something the developer needs to keep in mind. Replacing variables with small functions can make the code easier to read and to perform other refactorings because the local dependency of the variable is removed.

Replacing properties with queries can also remove mutable data that can also be calculated freshly, removing any danger to corrupt or stale the value.

Replacing a variable with a query only makes sense if the redone calculation of the variables will not change the outcome. While formatting for example would always produce the same result, a database query might produce different results each time it is executed (which can be intentional in the same function, but sometimes is not).

Replace Function with Command #

A command in this context is a function that is encapsulated into an object. In this form, the function can have its own set of subfunctions and properties. This can be useful when the function is too complex to be easily understood, or when it is intended to be used as a single unit of work that can be executed, undone, or queued.

Replace Conditional with Polymorphism #

Complex conditional logic is often the hardest thing to code in a readable, intuitive way. One way to solve this is to separate the conditions into different circumstances that are wrapped into different subclasses, each handling an understandable subset of the conditional logic. This can be Variants with given parameter options, variants that match certain assumptions, or variants that belong to different types.

Split Loop (intp Pipeline) #

It is a common anty pattern that a loop is used to do things at the same. This comes with the drawback, that whenever the loop is touched, both things need to be understood. Splitting such a loop into two loops, each doing its own task, can be a good solution for that. Many hesitate to do this because of performance concerns, which are covered in Refactoring and Performance

If that loop does multiple things, this can be further clarified by using Pipelines like Map, Filter, Reduce to further emphasise how the data is processed.

Long Parameter List #

Long parameter lists happen, sometimes for historic reasons, sometimes because more and more functionality was added over the iterations. It indicates in any case that the behavior of functions may vary in many ways, which ways exactly is obfuscated by the number of variables that the reading developer needs to keep in mind to parse the function.

One straight forward way is to replace a Parameter with Query to reduce the number of parameters, although this would keep the coupling.

def parameter_hungry_monster(
        has_teeth: bool,
        teeth_count: int,
        tongue_count: int,
        tonsils: bool,
        extra_jaw_size: int,
) -> str:
    teeth_count = max(min(teeth_count, 5), 0)
    monster = " _" + "_" * teeth_count + "δδ) \n"
    monster += "() " + " " * teeth_count + " | \n"
    monster += "  " + ("v" if has_teeth else "~") * teeth_count + "\ |\n"
    if tonsils:
        monster += " " * teeth_count + "   |│\n"
    for tongue in range(tongue_count):
        monster += " >≈" + "≈" * teeth_count + "|│\n"
    for extra_jaw_size in range(extra_jaw_size):
        monster += " " * teeth_count + "   |│\n"
    monster += " (" + "^" * teeth_count + "  │ \n"
    monster += " " * teeth_count + "ΣΣ ß"
    return monster

Introduce Parameter Object #

Often a set of parameters travel together even through multiple functions. Such data clump can be replaced by a data class, emphasizing their relationship. Also, this will make the code more consistent and reducing the parameter list. They also open up the possibility to restructure the code around this new-found abstraction.

It sometimes happens that a couple of properties of an object are separated just to be put into one function. This is prone to future touches as any change of the object might require touching the function. This can be avoided if the whole object would become the parameter. Although sometimes the dependency to the object can be a problem itself.

Another example of replacing parameters is replacing boolean flag arguments with more meaningful enums, even replace other parameters.

Combine Functions into Class #

Classes bind functions together in a shared environment, exposing some behavior for outside functions to interact. Moving sets of functions into a class makes the shared environment they act in visible.

Mutable Data #

Mutable data often leads to a change of data here causes unintended side effects in a different place. The easiest, but not always straight forward solution is to prefer values to references. Instead of changing data objects, they then should be re-instantiated with the new values.

One way to solve this is to extract the re-instantiation into a function or replacing the mutable variable directly with a query.

def mutable_data_monster() -> str:
    monster = (" ,---.\n"
               "( @ @ )\n"
               " ).-.(\n"
               "'/|||\`\n"
               ") '|` (\n")
    return _bubbly(_mutate_tentacle(monster))


def _mutate_tentacle(monster: str) -> str:
    monster = monster.replace("|||", "( )")
    monster = monster.replace("/", ")")
    return monster.replace("\\", "(")


def _bubbly(monster: str) -> str:
    monster = monster.replace(".", "~")
    monster = monster.replace("`", ")")
    monster = monster.replace("'", "(")
    return monster.replace(",", "●")

Encapsulate Variable into a function #

Data is trickier to refactor than functions, as functions can be easily called from multiple places through multiple functions; but changing a data structure often requires change in all cases in which the data is accessed. This is why routing access to a mutable data property can be routed through a function (like an accessor / getter function) allows for easier refactoring in the future.

The next step to this would be to remove setting methods to replace them by enforcing settings on initialisation, making the configuration of a class immutable. If that initialisation becomes more and more complex a factory function can help to organise the initialisation of classes in a readable and flexible way.

Separate Query from Modifier #

The differentiation between functions with and without side effects - Commands and Query Separation. A common pattern is that functions that return a value should not have side effects (queries).

Divergent Change #

Divergent Change happens if a module is touched for many reasons. If a class or function has too many responsibilities, it needs changes often and becomes harder to understand as it touches multiple contexts. The goal here is straight forward to sort the different contexts into different places in a way that allows the developer to ignore the rest when changing just one context.

from typing import Literal


def divergent_monster(phase=Literal["Protocell", "Prokaryote", "Eukaryote"]) -> str:
    membrane = get_membrane(phase)
    core = get_cell_core(phase)
    extra = get_mitochondrion(phase)

    monster = (f"       {membrane}{membrane}  {membrane}\n"
               f"     {membrane}       {membrane}\n"
               f"  {membrane}            {membrane}\n"
               f"{membrane}                 {membrane}\n"
               f"{membrane}     {core}{core}{core}             {membrane}\n"
               f"{membrane}    {core}  {core}{core}       {extra}    {membrane}\n"
               f"{membrane}     {core}{core}{core}                {membrane}\n"
               f"{membrane}                       {membrane}\n"
               f" {membrane}       {extra}      {extra}    {membrane}\n"
               f"    {membrane}             {membrane}\n"
               f"     {membrane}      {membrane}\n"
               f"       {membrane}{membrane}{membrane}\n")
    return monster


def get_membrane(phase=Literal["Protocell", "Prokaryote", "Eukaryote"]) -> str:
    match phase:
        case "Protocell":
            return "  "
        case _:
            return "||"


def get_cell_core(phase=Literal["Protocell", "Prokaryote", "Eukaryote"]) -> str:
    match phase:
        case "Eukaryote":
            return "▒"
        case _:
            return " "


def get_mitochondrion(phase=Literal["Protocell", "Prokaryote", "Eukaryote"]) -> str:
    match phase:
        case "Prokaryote":
            return " "
        case _:
            return "⬤"

Split Phase #

If a function or class models a process or sequence of steps, those steps can be separated into different phases. This can if the steps in question happen by business logic and are called in an order, the phases are needed for technical reasons like a compiler requiring parsing as a separate step. The first step in this direction would be extracting a function per phase.

Reorganise or Move #

To separate different context not only programmatically but also visibly they can be moved into different modules, files, classes. This can be valid for moving fields, functions, classes. This should leave only the relevant code in the view of the developer or the functions that call it, but also gather all needed code in one place.

Extract Class #

Similar to extracting functions, a class might often be too big to be understood. Indicators of where to split a class are sets of functions that rely on the same set of properties or data, sets of function that are always called together, or sets of function that are called for subtypes of the class.

Shotgun Surgery or Feature Envy #

This smell is the opposite of the Divergent Change, it is code that if you want to make one change to, you have to go to many places to do that change. When those changes are scattered over multiple files and classes, it is time costly to find and easy to miss one of them. Sometimes reorganisation is not straight forward to do.

def shotgun_monster(name: str) -> str:
    """
    Todo: Customer requested that the cat should have whiskers
    """
    ears = " /\_/\ \n"
    face = "( o.o )\n"
    legs = "(n   n)"
    if name.endswith("a"):
        ears = " /\_/@ \n"
    if "Sir " in name or "Lady " in name:
        face = "( δ.δ )\n"
    if len(name) > 10:
        legs = "| || |\n(∞   ∞)=======//"
    if name.startswith("Big"):
        face = "(  o.o  )\n"
        legs = "( Ω   Ω )"

    return ears + face + legs

Combine Functions into Transform #

The goal of this refactoring is to identify if multiple derived properties of a data instance are used and requested in multiple spots in the code. A single function or class is introduced to calculate all derive properties and hold them in one object to be used throughout the code. This makes it easier to track how different derived properties affect the data flow in the code and makes it easy to add features.

In a extreme form, this can lead to the combination of all the functions into one new class.

Inline Function, Variable, or Inline Class #

This is the opposite of extracting a function or class - copying the the body to all occurrences in which the function would be called. This can sometimes help to restructure the class or function design completely and can free one from the past refactoring decisions made.

Feature Envy #

Feature Envy can be a subset of this smell. Programs are usually modularized into parts with high cohesion, minimizing interaction between code of different zones. If a function spends more time communicating with code in different modules that can be a code smell and the code should be restructured.

def envy_monster() -> str:
    bark = get_membrane("Eukaryote")
    leaf = get_mitochondrion("Eukaryote")
    return (f"  {leaf}{leaf}  {leaf}\n"
            f"{leaf} {leaf} {leaf}{leaf}\n"
            f"  {leaf}{leaf} {leaf}\n"
            f"    {bark}\n"
            f"    {bark}\n"
            f"    {bark}\n")

Data Clumps #

If the same data items show up throughout the code, they might belong together, especially if they only make sense with all present they should be encapsulated into their own class or object to emphasize their combined meaning.

Primitive Obsession #

Primitives are the basic typing foundations provided in programming - integers, floats, strings, booleans in more or less granulated types are common. Slightly more defined types are welcomed, like dates or currencies. But many developers seem to avoid creating dataclasses that hold primitives with a meaning like coordinates that come in pairs of floats between 0 and 360, ranges that imply start is smaller than end, or distance that hold a unit. Such different types can be refactored into Polymorphism

For the practice of passing strings around, although custom types seem way more appropriate the term “stringly typed” is coined.

def primitive_monster(length: int) -> str:
    assert length > 0
    assert length <= 255
    assert length % 2 == 0
    return (" )  (\n"
            f"( '_'){'} ' * length} \n"
            f"      {'^ ' * length} \n")

Replace Primitive with Object #

Packing a primitive into an object provides a single point of validation, data extraction, and defines the value. Objects can answer the questions about the data like “is this amount in cent or euro?”, “Do telephone numbers always include a country code?”, “Can this length be negative?”. Such objects also provide space for comparing or calculating. This is strongly tied, but not limited to Parameter Objects

Speculative Generality #

There are cases in which the code is made more flexible that it really needs to be. Sometimes because a refactoring aimed for features that never came, sometimes because business mentioned featured that never reached the code but their preparation did, sometimes a developer is eager to implement a pattern that is too complex for the current code.

Code that is not reachable can be removed. Version control system enable us to just delete code, not just commenting them out. If removing code enables, then also the unused (or now always equal) function parameters should be removed

from abc import ABC


class Monster(ABC):
    identifier: str

    def draw(self) -> str:
        pass


class SpeculativeMonster(Monster):
    identifier: str = 'SpeculativeMonster'

    def draw(self) -> str:
        return self.speculative_monster()

    @staticmethod
    def speculative_monster() -> str:
        return (f"""
   .-----.   
 /         \ 
|\/(o) (o)\/|
|           | 
 \         / 
   \_____/
        """)

Collapse Hierarchy #

If a class hierarchy is no longer needed, remove them. Changes to the parent class always affect children - if this behavior is causing more work than benefits, an interface might provide the same benefits without the strong coupling. Work step by step and move all functions from the parent to the child in question, and remove the heritage. Collapse Hierarchy

Message Chains or Middle Mans #

Following the flow of data sometimes reveals a series of getters. Navigating this path of getters means the original client caller is coupled to each class stepped through. Often this can be simplified by the extraction of these methods to a place in which multiple clients may access the data.

A special case of this would be a Middle Man, a class that only offers or redirect delegates. Maybe the class had a reason to exist in the past, or some idea of encapsulation got carried out a bit too much. One first step might be to inline the functions and see if the class is actually bringing benefit.

class Client:
    def get_output(self) -> str:
        return Service().get_monster().speculative_monster()


class Monster:
    def speculative_monster(self) -> str:
        return (f"""
     \ ________
      |          \  
     /_/~~~~| |\ /9`\__‚  
    |b      |b   \/~~~~/
            """)


class Service:
    def get_monster(self) -> Monster:
        return Monster()

Hide Delegate #

Is some client code calls a method that is defined on an object provided by some service, the client requires knowledge about the delegate object. This coupling can be removed by adding a function on the service that will call the method of the delegate and only returns its result, hiding the delegate. Changes made to the delegate do not propagate to the (or even multiple) client(s), only to the service.

Replace Middle Man by Delegate #

If I hide a delegate too often, this can also cause a lot of code that is hard to maintain, especially the class used to hide the delegate does not provide any other benefit. Then a getter function may be introduced to get the whole object and all get_delegate_property functions may be replaced by that.

This can applicable as well for Superclasses. If it does not make sense for a subclass to use or overwrite all the functions of its superclass, it might be that inheritance is too strong of a coupling, and a delegate object instead of a superclass might be more fitting. Same for the subclass if more axes variations is needed than practical for inheritance, or is the coupling between subclasses causes problems.

Alternative classes with different interfaces #

One of the best benefits of classes are the option to substitute classes. Matching interfaces (so matching function declarations) allow substitution in the long term and helps to find patterns in the current code.

Comments #

If you need a comment to explain what the code does it either has a bad name, can be a function, or should be encapsulated and validated by a dataclass.

Comments may be helpful to explain business decisions (“This contradicts industry standard, but we decided to accept the message anyway”), historical learnings (“Cannot be deleted to maintain backwards compatibility”), or explain uncertainty (“Checking data consistency because imported data might be inconsistent here”).

Conclusion #

The book provides a nice overview over code smells and how to fix them. It was quite fun to explore the code smells, think about how to display them in code, and also keeping an eye on my work code base spotting both code smells and nice refactorings.

Besides all the hard skill learnings, the idea of refactoring line by line, with passing tests, always ensuring no external behavior is changed is the most enlightening skill I took from this book. It is a challenge, and I am not saying that I will never again look at smelly code and just creating a new file rewriting it from scratch, I now know I always have the option to only change it in a save and sane way, bit by bit (and might request the same for code I review).

Happy Coding :)

From Individual Problem to Team Culture Issue #

The term [Imposter Syndrom] pathologizes the insecurities regarding ones capabilities […] to a mental illness to be cured, not as a reaction to a lack of recognition, missing role models, absence of promotion opportunities, discrimination in the workplace.

Sophia Fritz, Toxische Weiblichkeit (Toxic Femininity), my translation

Reading this changed my perspective on Imposter Syndrome, moved it from something that everyone has to deal with on their own, to something that is influenced by the team culture and environment. Changing team culture will not solve systemic issues, nor will it prevent any feeling of insecurity. The point is that it is not just a personal task to build confidence, but a shared team responsibility to create an environment where every team member feels recognised, authentic, and competent.

Looking into Scientific Literature #

Following the Hierarchy of Evidence, I looked for Meta-Studies and Systematic Reviews on the topic of Imposter Syndrome. I would recommend “Workplace Imposter Thoughts, Imposter Feelings, and Impostorism”¹; other papers focused only on the pathology of Imposter Syndrome in individuals ², or did not as dive deep into the comparison of results of multiple studies³.

One very interesting paper is the first paper that coined the term “Imposter (Phenomenon)” a 1978 Paper from two Psychologists “The Imposter Phenomenon in High Achieving Women” ⁴ which observed and named the feeling based on observations of high achieving woman during a time in which woman where quite new in (academic) careers.

History, Definition, and Symptoms #

“Imposter Syndrome” is a very modern term, that got more popular since around 2015.

Throughout the years the term is defined by the cognitive belief that others overestimate one’s competence at work. But it also includes feeling of intellectual incompetence, concerns about being exposed as a fraud, or feeling unauthentic. There are multiple hypotheses about the cause of Imposter Syndrome³:

• Perfectionism and Fear of Failure: Perfectionism, especially socially defined perfectionism makes individuals highly sensitive to perceived failures. This can cause a black-and-white thinking of anything that is not a perfect success must be a failure.
• Social Comparison Theory: People define their own worth based on how they compare to others, Imposter feelings are a result of comparing one with high-achieving peers - even if the comparison is biased like comparing new joiners with long term employees.
• Cognitive Dissonance Theory: If people have the belief that they are not competent, any success will create a dissonance that is resolved by downplaying the success or attributing it to external factors.

Imposter Syndrome is linked to lowered job satisfaction, higher stress levels, even depression and burnout. Although, some studies found the imposter phenomenon was a critical driver of work productivity of management consultants or working hours of students.¹

Trait or State? #

There are studies that link Imposter Syndrome to situational factors - starting a new role that differs from experiences, working after career breaks, or as a result of competitions.

Still, Imposter Syndrome is often described as a personality trait, focusing on correlations with low self-esteem, neuroticism, and perfectionism. Multiple studies found higher prevalence among woman and other marginalized groups. Not all studies find such correlations though, especially when comparing people within the same role (the surveyed roles were leadership roles or surgeons); while students and post-graduates showed the highest gender disparity¹.

To conclude, the beliefs that feed Imposter Syndrome can come both from individuals themselves or the situations they are in. In either cases, it is not a trait that someone has throughout their life or never at all - but a state that can change over time and with context, and for which some people might be more susceptible.

Team Culture Patterns and Anti-Patterns to reduce Imposter Feelings #

During my research I learned that there are different types of psychotherapy. Depth psychology focuses on the unconscious and experiences to resolve current problems; Cognitive Behavioral Therapy focuses on changing the current beliefs and behaviors to resolve current problems; and (quite new) Systemic Therapy focuses on the interactions between people and their environment to resolve current problems. Systemic Therapy aims to recognize that one cannot change people or systems, and to find creative nudges to which helps the system to change themselves.

Regardless of the question of Trait vs State, in a team context one can always assume that team culture is easier to change than personality traits of individuals. Therefore, I want to share some nudges that can help to create a team culture that reduces feelings of inadequacy and fosters feelings of belonging and competence.

A feeling of belonging #

Celebrate Learning new Things in a Safe Environment Social rules that are stated explicitly and are valued by the whole team are a basis for creating a safe environment. A good example are the Social Rules of the Recurse Center ⁵

• “No feigning surprise” - don’t act surprised when someone does not know something
• “No well actually’s” - don’t correct someone if the correction does not benefit the conversation
• “No Backseat Driving” - don’t give unsolicited advice on how to solve a problem you are not working on
• “No Subtle -isms” - no sexism, racism, ableism, ageism, or any other -ism that can make people feel excluded

Anticipate New Joiners “You are here because we want you to be here and believe in you!” This feeling that is easier for teams to spread that have the autonomy to select new members, but still possible in more hierarchical structures. Something I mostly observed for new joiners, who are well known within a community, but a habit that I would wish every new joiner: In the first days showing excitement about the new team member joining, show interested in past projects, skills, and experiences. Maybe the manager already provided some information, a blog, a past project, or open source contributions. This helps to create a feeling of being wanted and welcomed for the skills and experiences.

Show a Shared Vision Belonging to a team means working on a shared goal. A critical part of that is also to be able to connect the current tasks to the big picture. There will be “high impact” tasks that are directly contributing, and they should be shared in the team. On the other hand, the “low impact” tasks require a context to connect them to the vision.

Have a Diverse Team and Diverse Leaders. Missing representation can lead to feelings of not belonging. If there is no one that resembles your background / culture / gender in the hierarchy above you, it is harder to believe that you can also reach such a position. Diverse teams also help to create an environment where different perspectives are valued, and different communication and working styles are accepted.⁶

Asking for Help is Normalized Asking for pairing should not just be done by the team members who are looking for guidance - but also by team members who want to share knowledge. It does not require a “I need some help here”, it can be “Is someone up for discussions on the technical approach?” or “Would be great to share knowledge about this domain”. Any habit that is only perceived from juniors or new joiners will create a hesitation to pursue such habit - this can be asking for help, seeking validation, or admitting mistakes. ⁷

Make it easy to Achieve #

Enable Achievements Outside of Code Software Development is more than coding - Shadowing Users, Writing documentation, improving processes, code reviews, sharing knowledge … Such tasks should be recognized as valuable contributions to the team vision, should be mentioned, should be celebrated.

Let Ticket Refinements Happen in the Team A refinement can be a safe space to ask questions, clarify the business need behind the ticket, and discuss possible implementations. This can easily change the perspective from a ticket that sounds “too complex or to hard for me” to a ticket for which “I know the solution and the pitfalls to avoid”. It is easier to pick up such a refined ticket, easier to start working without big code investigation, it instantly provides a feeling of competence for the task.

There Should be Easy Tickets An easy ticket should include all necessary information, enable one immediately to start coding, and be achievable in less than a day. This is important for new joiners, as distraction from highly complex tasks, or on dull afternoons. Never will all tickets be easy - but there should always be some easy tickets available to pick from.

Tickets Should Have a Definition of Done. Having to go over and over a task because the result is unclear or requires multiple feedback loops is a common source of frustration and self-doubt. Why didn’t I got it right the first time? Why don’t I understand what is expected from me? A clear definition of done helps to avoid such situations and also helps to estimate ones resources required to finish the task.

Have a Culture of Recognition - not Competition #

Love Power Sharing - Hate Power Trips In this context “Power” does not mean “authority”, Power means having a privilege another person is lacking - not having Imposter feelings, knowing the codebase well, not being part of a marginalized group… A team member that calls out the achievements of others, shows vulnerability, shares knowledge altruistically, is open for cooperation / pairing shares power⁸.

Post only Team Metrics - Never Individual Metrics Any metric that compares the performance of individuals will be exploited by team members who are driven by competition and will foster a feeling of inadequacy in those who are not. No metric will adequately reflect productivity. It cannot capture learning curves, mentoring efforts, or the contributions in discussions, documentations, mentoring, code reviews, or team culture. AFrom looking up “the number of PRs per month for each team member” to weekly postings of “who closed the most tickets” - any of those metrics should be handed with care.

Everyone Should be Part of Celebration Celebrating team successes helps to create a feeling of belonging and achievement - but only if everyone is included. Meaning, on that celebration slide every team member should find at least one point they connect to their contributions. This does not mean that everyone has to be named (I personally would even discourage that), but that everyone can find their part in the success story.

Mentoring Mentors or Buddies can provide constant guidance and affirmation. They can help to navigate the new environment, provide technical guidance, and be a safe space to discuss insecurities. ³ They can ask coaching questions to help overcome self-doubt and imposter feelings.

Personal Strategies #

There are also personal strategies to deal with Imposter feelings. While the papers showed the success of Cognitive Behavioral Therapy, that is a very personal decision to pursue, and nothing I have any qualification to comment on.

I want to provide some questions and reflections that can help to better understand ones own feelings

• Observe When do I have feelings of Incompetence? Are there specific situations that trigger these feelings? When am I not feeling this way? What do I feel when achieving something, and what if I fail?
• Question Believes Was there ever a person stating that I lack a skill or trait, or is it my interpretation of the situation? If the latter, which situations trigger this thought? What beliefs do I have about my own competence? If I allow Myself to be curious about these beliefs, where do they come from?
• Investigate Alternatives How do I define Success and Competence? Do I like my definition? Are there other team members struggling with similar issues? Can I talk to them about it?
• What can Help? What does help I with these feelings? How can I ground Myself in moments of insecurity, and remind Myself of my skills and achievements? Can I do a training that helps me to get more confident in the skill I feel lacking?

Conclusion #

Imposter Syndrome was and is a companion in my career, but it’s strength and impact is highly influenced by the team culture I am in. I worked at companies where I did felt valuable and competent after a few days, others in which I was struggling with feelings of inadequacy for months. I worked in teams where a belonging and communication was a high priority, and other where it wasn’t. I had managers who emphasized my growth and opportunities, and others who only cared about results and numbers of whatever metric they valued.

I am striving to create a culture in my team in which Imposter Feelings are acknowledged and quickly replaced by feelings of belonging and value.

Happy Coding Communicating :)

Registry by Factory #

The most straightforward solution is to use a Factory that returns the appropriate implementation based on the request. Often this is the best solution, especially if there are only a few implementations to choose from, if the selection logic is not based on one property but on complex business rules, or it is unlikely that new implementations will be added in the future.

class Collie:
    pass


class Husky:
    pass


class Labrador:
    pass


class DogFactory:
    def get_dog_for(self, request_type: str) -> Collie | Husky | Labrador:
        match request_type:
            case "Pulling":
                return Husky()
            case "Herding":
                return Collie()
            case "Service Dog":
                return Labrador()
            case _:
                return Labrador()

The implementation is straightforward and easy to understand - no magic involved. Whenever a new implementation is needed, the Factory needs to be modified, violating the Open/Closed Principle.

Testing the Factory #

The most simple test is to check that the Factory returns the expected implementation for a given request.

def test_dog_factory():
    assert isinstance(DogFactory().get_dog_for("Pulling"), Husky)
    assert isinstance(DogFactory().get_dog_for("Herding"), Collie)
    assert isinstance(DogFactory().get_dog_for("Service Dog"), Labrador)

The Factory can not be tested in an abstract way, it is barely possible to write a test that ensures that every implementation is reachable, or that a new implementation will be added to the factory without re-implementing the selection logic of the Factory in the test.

Registry by config file or list #

Applying the Open/Closed Principle leads to the next simplest solution: A configuration file or list that contains the mapping. In its simplest form, this can be a dictionary in Python code, but especially in use cases in which different environments require different mappings, an external configuration file (YAML, JSON, etc.) is a good tool. Even further goes the idea of a database table that contains the mapping, which can be changed at runtime without a new deployment, is nice to test, but requires seeding / other solution for local development.

class BaseDog:
    pass


class Collie(BaseDog):
    pass


class Husky(BaseDog):
    pass


class Labrador(BaseDog):
    pass


DOG_CONFIG: dict[str, type[BaseDog]] = {
    "Pulling": Husky,
    "Herding": Collie,
    "Service Dog": Labrador,
}


class DogRegistry:
    def get_dog_for(self, request_type: str) -> BaseDog:
        found = DOG_CONFIG.get(request_type)
        if found:
            return found()
        else:
            return Labrador()

This solution is still straightforward and easy to understand. The selection logic is now separated from the mapping, new implementations can be added without modifying the code.

A pattern that I personally find “topologically equal” is a having a function that registers all handlers in a dictionary handler_registry.register("Pulling", Husky) individually. That is verbose, and allows programmatic " un-register"¹. The post also discovers very similar patterns. Mind that in that blog post mentioned solution of importing and looping through all subclasses causes the similar import-based problems, as a missing Handler in an init file will cause the Handler to remain undiscovered.

Testing the Config Registry / Test to ensure a new implementation is listed in the config #

Tests can be separated into testing the selection logic and testing the configuration. Also, tests that ensure that every implementation is reachable and that a new configuration is added correctly are possible.

from magic_registry import magic  # example function at the end of this post

from config import registry
from config import dogs


def test_config_returns_husky():
    assert isinstance(registry.DogRegistry().get_dog_for("Pulling"), dogs.Husky)


def test_config_is_complete():
    """ Test to ensure that all BaseDog implementations are registered in the DOG_CONFIG """
    all_dogs_registered = set(registry.DOG_CONFIG.values())
    import config as module
    all_dogs_in_module = set(magic.autodiscover(in_module=module, of_subclass=dogs.BaseDog))
    assert all_dogs_registered == all_dogs_in_module

Registry by init_subclass #

Looking at the Python features, there is a built-in hook method called __init_subclass__ that is invoked whenever a subclass is defined imported. The corresponding PEP even provides an example of a registry pattern ². Based on this, there is a BaseDog base class that registers every subclass to a dictionary, and the DogRegister using that to return the appropriate implementation.

By this solution, it is fair to assume that there will be a directory, which contains all implementations, like this:

Project/
├─ init_registry/
│  ├─ __init__.py
│  ├─ dogs/
│  │  ├─ __init__.py
│  │  ├─ husky.py
│  │  ├─ common.py
|  │  └─ corgi.py
|  └─ registry.py
└─ tests/ 
...

#  ------------- registry.py ------------- 
class BaseDog:
    def __init_subclass__(cls, request_type: str, **kwargs):
        super().__init_subclass__(**kwargs)
        DOG_REGISTRY[request_type] = cls()


DOG_REGISTRY: dict[str, BaseDog] = {}


class DogRegister:
    # import init_subclass.dogs.corgi # Import corgi to register it
    def get_dog_for(self, request_type: str) -> BaseDog:
        found = DOG_REGISTRY.get(request_type)
        if found:
            return found
        else:
            from init_subclass import dogs
            return dogs.Labrador()


#  ------------- dogs/common.py ------------- 
from init_subclass import registry


class Collie(registry.BaseDog, request_type="Herding"):
    pass


class Labrador(registry.BaseDog, request_type="Service Dog"):
    pass


#  ------------- dogs/husky.py ------------- 
from init_subclass import registry


class Husky(registry.BaseDog, request_type="Pulling"):
    pass


# -------------  dogs/corgi.py ------------- 
from init_subclass import registry


class Corgi(registry.BaseDog, request_type="Mascot"):
    pass


# -------------  dogs/__init__.py ------------- 
from . import husky, common

While the other implementations had the logic in one file, this one requires the developers action to look in the base class. And then, probably many developers would need to look up what __init_subclass__ is doing, as it is not that commonly used, but considering the very similar example in the PEP, I would consider its readability okay.

😈 Anyway, how long would it take a dev to understand the code is not working and why?

Import-based Registry - and what can go wrong #

Based on the above example, the first test will pass, but the second test will fail:

from init_subclass import registry
from init_subclass import dogs


def test_init_subclass_returns_husky():
    assert isinstance(registry.DogRegister().get_dog_for("Pulling"), dogs.husky.Husky)  # passes


def test_init_subclass_will_not_find_secret_corgi():
    assert not isinstance(registry.DogRegister().get_dog_for("Mascot"), dogs.common.Labrador)  # fails

The DogRegister().get_dog_for("Mascot") should return a corgi.Corgi, but it returns the default common.Labrador. The reason for that is that the corgi.Corgi class is never imported, and therefore never registered in the DOG_REGISTRY. Which is especially tricky to test, because if there was a test implemented to test the corgy.py. It would import the corgi.Corgi class and would also make the above test pass.

Python has one dictionary called sys.modules that contains all imported modules. Looking into that dictionary during a debugging session shows that only the husky and common modules are imported, but not the corgi module. Importing the corgi module during the debugging session registers the corgi.Corgi class, and then the test would pass.

# Importing sys
>>> (Pdb) import sys
# sys.modules is a dict of all imported modules, looking for Husky module
>>> (Pdb) sys.modules['init_subclass.dogs.husky']
<module 'init_subclass.dogs.husky' from '/Users/sofia/dev/PythonProject/src/init_subclass/dogs/husky.py'>
# Looking for Corgi module, but failing
>>> (Pdb) sys.modules['init_subclass.dogs.corgi']
*** KeyError: 'init_subclass.dogs.corgi'
# Importing the Corgi module
>>> (Pdb) from init_subclass.dogs import corgi
# Now the Corgi module is in sys.modules
>>> (Pdb) sys.modules['init_subclass.dogs.corgi']
<module 'init_subclass.dogs.corgi' from '/Users/sofia/dev/PythonProject/src/init_subclass/dogs/corgi.py'>
# And the Corgi module used points to the same via sys.modules
>>>(Pdb) dogs.corgi
<module 'init_subclass.dogs.corgi' from '/Users/sofia/dev/PythonProject/src/init_subclass/dogs/corgi.py'>

Side Quest: Understanding Python imports #

The import statement in python creates a module object, calling its module body, and by this defining a __name__ as the key, and __file__ as the file path value in sys.modules. If the module is already in sys.modules, the existing module object is used instead, which saves time.

The module in which the execution of a python project starts has __main__ both as __name__ and key in sys.modules. Modules may check if they are run as __main__ by checking if __name__ == "__main__":. For example Django is doing that in its manage.py file to set up the project settings and import Django.³

The imported Django module is actually a package. A package is a module that contains other modules, and is identified by the presence of an __init__.py file in the directory, which is its module body . When a package is imported, the module body in the __init__.py file will be executed, importing all sub-modules listed in it. If a sub-module is not listed there, it is not imported.

This also explains why circular imports raise an exception, as the module body execution and the registration in sys.modules is suspended while waiting for the imports inside the module body to finish. If the first module is eventually imported inside its own body, the module is not yet in sys.modules, and would start an infinit loop, if not excepted before. ⁴

Every module uses a __dict__ attribute as its namespace, which is a dictionary-like object that maps its attributes to their values. Any class or attribute defined in the module body is added to this namespace, which means under the hood that my_module.my_class is (somewhat) equivalent to my_module.__dict__['my_class']. Therefore, for very attribute, for example a class defined in the module body, Python creates an object by calling its Metaclass (usually type.__new__) to create the class object, It calls the __set_name__ method, super(), in which __init_subclass__ of the base class is called. The fully initialized class object is then bound to the namespace that the module body is executed in. ⁵

Manipulating sys.modules is also possible (maybe not the best solution). In theory, deleting the module from sys.modules will ensure that the tests of import-based registries work deterministically, even if other tests import some implementations.

import sys

module_identifier = "init_subclass.dogs"
modules_to_delete = [name for name in sys.modules if name.startswith(module_identifier)]
for name in modules_to_delete:
    del sys.modules[name]

Registry by decorator #

Another way to register implementations is to use a decorator. It looks a bit more modern that __init_subclass__, but it has the same problem: If the module containing the implementation is not imported, the implementation is not registered, as the decorators are evaluated also on creation of the class type. I wrote a post about decorators before, so I will not go into detail here.

def register_dog(request_type: str):
    def decorator(cls):
        DOG_REGISTRY[request_type] = cls
        return cls

    return decorator


DOG_REGISTRY: dict[str, type] = {}


@register_dog(request_type="Herding")
class Collie:
    pass


@register_dog(request_type="Pulling")
class Husky:
    pass


@register_dog(request_type="Service Dog")
class Labrador:
    pass


class DogRegister:
    def get_dog_for(self, request_type: str) -> Husky | Labrador | Collie:
        found = DOG_REGISTRY.get(request_type)
        if found:
            return found()
        else:
            return Labrador()

Registry by pkgutil.iter_modules #

If python has a way to iterate through all sub-modules of a given module, and registers all attributes including their types, it is possible to implement a registry that imports all modules in a given package, and registers all.

Frameworks often utilize similar mechanisms to auto-discover modules or classes to register them automatically, and provide magically working experience for the framework user. Examples of this are in Django django.utils.module_loading.autodiscover_modules which auto-discovers modules named a certain way (e.g., admin.py); that can be assumed are imported. This is utilized in Django’s admin interface to auto-register models for the admin interface. Another example, in which Django is not expecting a certain module name, is loading all migration files ⁶.

This discover_classes() function uses the same pkgutil to iterate through all sub-modules of a given module, collecting all python files even if only a subset is listed in the __init__.py file. It returns pkgutil.ModuleInfo(module_finder, name, ispkg), which is the mentioned module object. Its name can be used to import the sub-module with importlib.import_module; and ispkg can be used to check if the sub-module is a package itself, in which case the function is called recursively. Then, inspect.getmembers is used to get all members of the sub-module, filtering for classes that are subclasses of the provided Dog base class.

# ------------- registry.py -------------
import importlib
import inspect
import pkgutil
from collections.abc import Generator
from types import ModuleType


class BaseDog:
    request_type: str


class DogRegister:
    _registry: dict[str, type] = {}

    def register_dogs(self):
        import iter_modules.dogs as module
        for dog in self.discover_classes(in_module=module, of_subclass=BaseDog):
            assert isinstance(dog, type(BaseDog))
            self._registry[dog().request_type] = dog

    def get_dog_for(self, request_type: str) -> BaseDog:
        found = self._registry.get(request_type)
        if found:
            return found()
        else:
            from iter_modules import dogs
            return dogs.common.Labrador()

    def discover_classes(self, in_module: ModuleType, of_subclass: type) -> Generator[type]:
        for sub_module_info in pkgutil.iter_modules(in_module.__path__):
            # import sub-module
            sub_module = importlib.import_module(f"{in_module.__name__}.{sub_module_info.name}")
            # get classes matching Base
            for name, obj in inspect.getmembers(sub_module):
                # check if it's a class and subclass of Base (and not Base itself)
                if inspect.isclass(obj) and issubclass(obj, of_subclass) and obj is not of_subclass:
                    yield obj
            if sub_module_info.ispkg:
                # recurse into sub-package
                yield from self.discover_classes(in_module=sub_module, of_subclass=of_subclass)


# ------------- dogs/any file, any __init__.py content-------------
from iter_modules import registry


class Collie(registry.BaseDog):
    request_type = "Herding"


class Labrador(registry.BaseDog):
    request_type = "Service Dog"


class Husky(registry.BaseDog):
    request_type = "Pulling"

Testing the pkgutil Registry #

Similar to the config-based registry, the selection logic and the completeness of the registry can be tested separately.

from iter_modules import registry
from iter_modules import dogs


def test_init_subclass_returns_husky():
    dog_registry = registry.DogRegister()
    dog_registry.register_dogs()
    assert isinstance(dog_registry.get_dog_for("Pulling"), dogs.husky.Husky)

When to trigger registration? This solution is not triggered automatically on import by anything, and only uses the iteration over modules to ensure all are imported. So, when to call the register_dogs() function? One option is to utilize framework hooks like Django’s AppConfig ready() method ⁷, or implement similar hooks to control when the registration should happen.

Conclusion #

Magically working dev experience and hard to debug code often go hand in hand. Still, things can be done to minimize the knowledge required from the developer to use the code. Tests that fail with dev-friendly messages when something is not imported or registered correctly can help a lot, and move the magic to the tests instead of the business code. If the magic is in the business code, good documentation to spread awareness about the pitfalls of the current implementation and minimizing the errors developers can make is crucial.

Again big thanks to Markus for the debugging help on import related bugs, many insights about python and inspiration for this post!

Happy coding :)

If the goal is Observability, what is the data to collect? #

Raw data is a oxymoron, there are always decisions made what to collect and how to store it. (Pydata 25 Melbourne, “Falshoods Devs believe”)

Three pillars of observability - how to collect data #

Observability starts with collecting data about the system. The three pillars of observability are the main three ways in which data may be collected. ¹

• Logs - timestamped, semi-structured text records of discrete events, like error messages, transaction records, user actions. Mind that power of logs comes from their structure, the inclusion of meta-data like request IDs, request type, user group, trace id,… makes logs much more useful to search and to correlate. Compared to events, logs are more flexible and allow for more experimentation to find the right structure. Mind that logs should never contain sensitive data.
• Metrics - numerical data points collected over time, like request latency, error rates, business cases processed. They are an efficient way to monitor the health data over time, and can be aggregated and visualized in dashboards. While logs often are limited to some recent time window, metrics can be stored for longer periods to analyze trends.
• Traces - A static trace id in a request can be used to track a request life-cycle across multiple components. Traces are useful to understand the flow of requests, identify bottlenecks, and debug issues in complex distributed systems.

Service Level Indicators (SLIs), Service Level Objectives (SLOs) and Service Level Agreements (SLAs) #

Service Level Indicators (SLIs) measures on service level, for example request latency, availability, throughput. Which indicators are important to a system depends on what the users care about.

Service Level Objectives (SLOs) are target values or ranges for a service level indicator over a period of time or fraction of requests, like 99.9% of requests should be below 100ms latency, or 99.9% availability over a month. Choosing and publishing SLOs can communicate expected performance to users, and can guide developer decisions.

Service Level Agreements (SLAs) are explicit or implicit contracts between service providers and users. Explicit SLAs can be legally binding requirements, while implicit SLAs are expectations of the users.

A special form of SLA is the error budget, which defines for developers how much unreliability is acceptable within a certain period, until developers need to focus on reliability improvements over feature development.

How to choose what to monitor #

The golden signals are a good starting point for choosing what to monitor: ²

• Latency - Time to process a request
• Traffic - Amount of requests
• Errors - Rate of failed requests
• Saturation - Resource utilization with focus on bottlenecks

First principles are basic assumption about the system that are not deduced from other assumptions. The idea of first principles is usefully as good observability should allow a developer to debug a system without a lot of prior knowledge about it. As software systems grow in complexity, it becomes more valuable to have a solid foundation to reason on facts rather that relying on years of experience with a system. ³

Alerting and Monitoring #

Assuming there are metrics and logs collected about the system, anomalies need to be detected and acted upon. There are three main ways to deal with anomalies: ²

• Collecting information about anomalies that do not require action. For example, higher load on mondays morning is a good thing to be aware of, but not something that needs action.
• Tickets can be created for anomalies that need to be investigated, but do not require immediate action. For example, an increased error rate for one user group.
• Alerts are created when immediate action is required and possible. If the throughput of a data pipeline drops below a certain threshold, an alert can notify to investigate and mitigate the issue.

Alerts should be designed carefully, as they create cognitive load for developers and operators. The worst alert are repeatedly false alarms, alarms on which no action can be taken, or constant alarms that are ignored over time.

Alert Design #

An alert should be actionable. It should be common practice to define runbooks for every alert. What are possible causes, and how to investigate, and how to mitigate the issue? Information like who will be notified, escalation paths, or user impact should also be part of the runbook. Alerts should also require intelligence to investigate - if the runbook looks so well-defined, that it can be automated, it should be.

Choosing Targets for Alerting #

Externally Service Level Indicators (SLIs) do not need to match internal alerting targets. If the external targets are higher than the current (sub)system performance, alerting will be frequent and therefore ignored in the long run; and if the external targets are lower than the current (sub)system performance, alerts can be used to strive for higher performance.

Using Dashboards #

Observability should enable developers to understand and debug the system. Dashboards and Monitors should be designed around usability - next to the alert that some request time is out of bounds, developers should be able to quickly find information about possible causes, like request rates spiking, database connections being exhausted, or error rates increasing. Looking at a dashboard full of flashing graphs is not useful, even if it might look impressive.

Distribution over Averages #

Data viewed in distribution reveals patterns, anomalies, and offers more actionable insights compared to simple averages. For example, the average response time of a web service is nice to have, the information that 95% of requests are below 200ms while 5% take over 2 seconds provides a actionable insight that can lead to performance improvements or clearer expectations for users. Software system often have non-normal distributions, with outliers and long tails that would be hidden by averages, but are visible by using distribution aware tools like percentiles. ²

Distributions is not limited to distribution over amount of data points, but can also be used to show distribution over time, resources, user groups, …

• 95% of requests are served in under 200ms
• On 95% of days, the response time was under 200ms
• 95% of our users in the low-data throughput group experienced response times under 200ms
• 90% of our containers served requests in under 200ms

This already hints at what meta-data is useful to collect alongside the primary metrics.

Dashboards like Medical Monitors #

Ever looked at a medical monitor with heart rate, blood pressure, oxygen saturation and wondered if the values are good or not? Reading ECG requires training and experience, knowledge about the p-wave, qrs-complex and t-wave, and what how specific patterns indicate health issues. The moment a health metric drops or rises above a certain threshold, the monitor will indicate that by a color change, a easy description (e.g. “HR low”), and an alarm sound that fits the severity of the issue. This enables medical staff to act fast, without needing to interpret the raw data to understand the situation, and without knowledge about the type of monitor.

If you think about dashboard designs like medical monitors, the time series data requires a lot of training and experience to read, and while it can be interesting, it is often hard to judge what is normal and what is not.

Dashboards should therefore include baselines, indicating what is normal for a metric:

• Display a shaded area around the time series, indicating the average over the past weeks
• Provide high and low thresholds for normal ranges
• Color indications when the metric is outside normal ranges with severity levels
• For every numerical metric, provide the average metric over the past weeks as a reference point
• Provide context about external factors that influence the metric, like deployments, incidents, marketing campaigns,
• Have meaningful titles, and add hints, explanations, links to runbooks to help interpreting the data

Dashboard Graphs should tell a Story #

On the question of how to use graphics to communicate data effectively, I learned a lot from the book “Communication Patterns”, which I wrote about in my previous blog post

Conclusion #

I searched for good resources about observability, monitoring and alerting, and found way less than I anticipated. Especially good practices about what data to collect, how to design dashboards are hard to find (but honestly, so are good designed Dashboards). I hope to learn more about this topic in the future, and share my learnings in future blog posts.

Happy Coding :)

Communication Patters - about the book #

Successful communication is the art and science of sharing or exchanging ideas and information, using a common set of symbols, signs, or behaviors, resulting in shared understanding. — Jacqui Read, Communication Patterns ¹

Communication can be hard, and the cost of miscommunication can be high. Refining existing communication patterns into named patterns can form a toolbox for good communication and awareness of dark patterns.

A Pattern is a reusable solution to a commonly occurring problem within that is known to be effective. An Anti-pattern on the other hand is a solution that looks right, but its consequences outweigh its benefits.

Think about the Users Perspective #

Identifying the audience / user to communicate with is key to successful communication.

• What Role does the User fulfill? How much technical knowledge does the User have? Can they understand technical terms, or technical diagrams like UML?
• What does the User want to know? What do you want from the User? In what level of detail do they need the information?
• What do they already know? Do they might know outdated information that needs to be corrected?

Visual Communication #

Which diagram to choose? #

Giving the following bad example:

❌ Anti Pattern: Choosing the wrong Diagram for the User #

Different Users will understand and appreciate different diagrams. Talking in stereotypical roles, the steak holder who wants to understand the system would prefer a User Journey over a Class Diagram, while a Developer would probably gain quite some information from the latter.

The goal should be to choose a diagram that the User is familiar with, that will match their question or conveys the message they should receive, and it should contain the amount of detail that the User needs.

❌ Anti Pattern: Mixing Levels of Abstraction - 🛠 Tool: C4 Model #

Developer Example:³

def feed(corgi: Corgi) -> None:
    # easily readable level of abstraction
    if corgi.is_fed():
        return
    food = _get_food()
    corgi.ate.appent(food)
    # awkwardly detailed level of abstraction
    corgi.last_fed = datetime.now()
    corgi.save()
    # the last 2-3 lines could have been a method,
    # making this much more readable. 

Every diagram will be an abstraction of something, but not everything that can be useful for the reader should be compressed into one diagram. Individual tickets do not belong into a roadmap, as it would clutter the model or even raise the questions what these little details mean if they are so important to be mentioned here.

One Solution to this is offered by the C4 Hierarchy of abstractions. Starting from a big picture it starts with the System Context in which the system interacts with its environment. The Container level displays the components of system with their interaction in between them or the external entities. While the Component levels zooms into one Container of the last level and dissects its subcomponents. The finest level is the Code, which reveals actual implementation.

🏅 Best Practice: Representational Consistency #

When working with multiple diagrams, for example after applying the C4 Model, all references to the same component, role, entity should be easily identifiable as the same. With diagrams, we often do not have the luxury to step in and out of details with one click or having a compiler checking that naming is consistent over multiple diagrams. A role could be named the same and use the same pictogram; a component could be named the same and be framed by the same symbol and color in a system context diagram and a container diagram.

💡 Best Practice: Single Responsibility #

This commonly known software pattern describes that a building block should only have a single responsibility. The most repeated reason for this is maintainability, as there is also only one reason to change; giving a block of code a single responsibility makes it wat easier to understand. The same applies to diagrams, if a diagram tries to convey multiple messages, it will be hard to understand and less effective in conveying that message.

The main differentiation of responsibilities are between Behavior and Structure. Behavior diagrams like Sequence Diagrams show how the system achieves which goal. Structure diagrams like Class Diagrams show what the system is made of and who it interacts with.

Diagram Layout #

After choosing a fitting diagram type, the layout of the diagram should be designed to easily convey the message to the User.

A bad example would be:

❌ Anti Pattern: Boxes in Boxes in Boxes #

Boxes are a great tool to group related items together, if done correctly they can be used to indicate consistent entities over multiple diagrams, e.g. if the box that represents a Component in a Container Diagram is the same as the box that contains a component Diagram. However, boxes within boxes can quickly lead to confusion, and can then cost more time to understand the diagram.

A possible developer analogy might be nested functions, or nesting function logic into multiple levels of classes.

=> White Space reduces Cognitive Load, while lines and boxes increase it.

❌ Anti Pattern: Relationship Spiderweb #

The connection between diagram elements should have a defined meaning, which should be consistent over the diagram and easy to be identified by the User.

Examples of Relationships are:

• Hierarchical: Illustrating parent-child connection, like in taxonomies or organizational charts.
• Sequential: Progress of steps or stages, like in flowcharts or step-by-step guides.
• Causal: Showing cause-effect relationships, like in decision trees.
• Spatial: Representing physical or conceptual proximity, like in mind maps.
• Proportional: Indicating relative scale or size, like in pie charts or bar graphs.

Mixing multiple types of connections or not planning the layout to avoid crossing lines can quickly lead to a spiderweb of connections that is hard to understand.

🏅 Best Practice: Create Visual Balance #

Balance is a innate human expectation. Similar to photos, a diagram that is visually balanced will be easier to understand and appealing to look at. There is no character limit or maximum lines of code per file for diagrams, and fiddling with mermaid directions to achieve a visually balanced diagram can be annoying, but it is worth the effort.

💡 Best Practice: Match the Flow the User Expects, Tell a Story #

A human will find a diagram easier to understand if the flow of information matches their expectations, providing a start point (of a user journey, a state machine, the main class things evolve around), evolving around the message or story the diagram should tell, and a conclusion or end point.

While there are cultural differences, western cultures will expect information to flow from left to right and top to bottom. Anyway, it is easier to follow a flow that is consistent throughout the diagram, and if possible unidirectional.

Diagram Elements #

🏅 Best Practice: Balance Text #

Big walls of text obfuscate the message of a diagram. Often test can be reduced or moved into a footnote or legend.

Like Code, Diagrams should be understandable without comments or additional explanations. Good naming, understandable composition, carefully chosen colors and style, and known symbols can aid a well formulated diagram message.

🏅 Best Practice: Add a Legend #

Not including a legend means assuming that the User understands all elements, abbreviations, or symbols used. That assumption will in the end reduce the audience of the diagram, not only because Users unfamiliar with technical knowledge will not understand it, but also Users of other cultural backgrounds might interpret symbols or colors differently.

❌ Anti Pattern: Color Overload #

Colors should have a meaning - Using too many colors can quickly lead to confusion. Colors are a great tool provide an additional layer of information, e.g. grouping related items, or indicating status, risk levels, or role.

It is a best practice not to rely solely on colors to convey information, and always use a pattern or at least color hue to differentiate items, and included their meaning in the legend.

💡 Best Practice: Use Meta Style #

Style can also convey information. Using a dashed border for deprecated items or a scribbled style for ideas or future features. While many choose a style based on personal preference, choosing between Excalidraw or Mermaid can convey much more information than taste. Every Shape can have meaning by convention or association, and can be much more than style.

❌ Anti Pattern: Misleading the User #

Dark Patterns are well famous in UX Design, but can also be applied to Diagrams. Starting an axis at a non-zero value, using non-linear scales, emotional colors or pictures to influence the User … Every diagram has a message to convey, but misleading the User is plain unethical.

Written Communication #

🏅 Best Practice: Big picture first #

As a User of written communication, knowing the order of the things I am going to learn, will help to organize the information into my mental model. Starting with the big picture provides a narrative, helping to navigate into the details a user might look for and improves retention of information.

🛠Minto Pyramid: The Minto Pyramid Principle is a technique to structure information in a way that is easy to understand.

1. Start with the key message. For the User it is easier to understand the details if the key idea is known upfront.
2. Write out the details of information to convey with the message.
3. Group related information together, and structure them again into key messages and details.

❌ Anti Pattern: Testing the Audiences Knowledge #

The goal of communication is to convey a message. Any word, phrase, or concept that is unknown to the Audience but used by the communicator will make it harder to convey the message. It is important to know the Audience and what knowledge to expect from them in regard to technical terms, domain knowledge, language skills / vocabulary, diagram standards, and cultural context.

💡 Know about Biases #

Humans show systematic patterns of deviation from rationality in judgment, known as cognitive biases. They can affect how important we perceive information, how we interpret messages, and how we make decisions based on what we read or hear. As communicators, being aware of these biases can help us craft messages that are clearer and more effective.

🏅 Knowledge Management Principles #

• Product over Project: Keep the long term focus on domain knowledge that lives beyond the life cycle of a project.
• Ensure Visibility: Make knowledge accessible to those who need it. Centralize documentation in a known location, with open access for people even from different teams if they work on the same product or domain.
• Enable Reusability: Templates and standards for documentation can help to create consistent and reusable knowledge artifacts, while also decreasing entry barriers for contributors.
• Prioritize Searchability: Structure documentation using Metadata, Tags, and a clear structure.

🏅 Best Practice: Get Feedback #

Getting feedback from the multiple Audiences is crucial to successful communication. It is the best way to identify errors and possible optimisations early, establish a dialogue with the Audience, align expectations, evaluate risks, and share the load of communication and documentation. Like Pull Requests in Software Development, having a review process is an effective way to ensure quality and establish shared ownership of communication artifacts.

Conclusion #

Looking at the parallels of modeling a system as code and modeling a system in visual communication provides very interesting point of views. Modern code is designed to be reused, easy to understand, and easy to maintain. Why not apply the same principles and patterns to communication artifacts as well?

The book also covers much more information about verbal communication, meetings, and presentations, goes into great detail about the individual patterns, and provides many more examples - after all these are just my personal takeaways, biased by the knowledge I already had and the ignorance of topics I currently do not appreciate.

Happy coding :) (and communicating)

Graphic Recording is a visual practice that involves capturing and synthesizing information in real-time during meetings, workshops, or presentations. It combines elements of drawing, writing, and design to create large-scale visual representations of the content being discussed.

I want to learn Graphic Recording to enhance my ability to communicate complex ideas and concepts in a more engaging way. It combines my interests in art and communication. To learn this skill I started with the book “Graphic Recording - das 1 x 1 der Live-Visualisierung” ¹ by Martina Grigoleit, and various online resources and booklets.

While I learned about graphic recording when visiting the DevOps Days in 2023, I also got inspired by the blog posts from Julia Evans ².

Git Step by Step from File to Commit #

Working with Git is a common task for developers. Different projects have different expectations on git should be used. In the last months I had to upgrade my git skills and wanted to share some of the things I learned.

‼️ This post about what I learned additionally, therefore some basic git knowledge is assumed.

Within this blog post I mainly used the Git Book ¹ and the O’Reilly Course ².

Hashes as Object IDs #

Git uses a lot of hashes of files referred to as Object IDs (OID). Hash the content of the file. This hashing creates a unique (40 char) identifier for the content (using SHA-1). For two files with the same content, the hash will be the same, the file name or location does not matter. Adding two files with the same content will result in the same hash being created.

> cat README.md | git hash-object --stdin
0588748a531675ba06e473ae4d74532cdb2f94d3

Blob is one Git Object #

Git will create a blob in the .git directory with the hash as name and the zipped content of the file as content. Git is a Content addressable file system, which means that the content of the file is used to determine the file location.

.git/objects
├── 05
│   └── 88748a531675ba06e473ae4d74532cdb2f94d3
|       "Blob of the file content (zipped)"
├── info
└── pack

If multiple filed are added, one blob is created for each file. Two files with the same content will be added as one blob. Editing a file will create a new blob with a new hash, the old blob will still be there. Without a commit, this blob is not reachable, git only keeps history of committed files. Git is Immutable - a blob cannot be changed, only new blobs can be created. The git garbage collector will remove the blobs that are not referenced by any link eventually.

Git is not copying the files. Unchanged files are not copied, only the tree object is created with the hash of the blob of the unchanged file pointing to one content.

A Tree is a Git Object #

To store the relation of the file name and the blob, git creates a tree object. A Tree object is a file that contains a list in which the file type, file name, and the Hash Object ID. The Tree file hashed will again be the Object ID for the tree, therefore it will also be the same if all file contents and the file names in the tree are the same.

> git write-tree
2d66c1b551aae5efa0d242cc105b8531d3ae5e4b
> git cat-file -p 2d66c1b551aae5efa0d242cc105b8531d3ae5e4b
040000 tree 9b6bc3d88d357e1f02cda9e279f0bc9a9c5c11a3	content
040000 tree b8e6c8eb08befc5a0fde3b0c2821c806c7be2b36	public
100644 blob 0588748a531675ba06e473ae4d74532cdb2f94d3	readme.md
100644 blob 0588748a531675ba06e473ae4d74532cdb2f94d3	readme_copy.md
040000 tree bac8dc35897c3fe58c6f3d321deea5ee7dfff50f	resources

git write-tree creates a tree object from the staged files, this is usually done automatically when committing. In the tree file, different files with the same content will point to the same hash (blob). The output shows the tree object with the following structure:

100644 blob 0588748a531675ba06e473ae4d74532cdb2f94d3	readme.md
100644     -> Git file Type (binary 1000 (regular file), 1010 (symbolic link) and 1110 (gitlink) 
              and Linux file mode, binary authorization (0644/-rw-r--r--)
blob       -> The type of the object, in this case a blob. Subdirectories (like public) 
              are modeled as trees. 
0588748a...-> The OID (Object ID), Hash of the blob
readme.md  -> The name of the file

While staged the tree objects are also added to the .git/objects. directory.

.git/objects
├── 2d   # The root tree object
│   └── 66c1b551aae5efa0d242cc105b8531d3ae5e4b 
├── 9b   # The tree object for the content directory
│   └── 6bc3d88d357e1f02cda9e279f0bc9a9c5c11a3 
├── b8   # The tree object for the public directory
│   └── e6c8eb08befc5a0fde3b0c2821c806c7be2b36
├── 05   # The blob object for the readme.md file
│   └── 88748a531675ba06e473ae4d74532cdb2f94d3
...
├── info
└── pack

A Commit is a Git Object #

Git creates a commit object, which again is a file containing the hash of a tree object, the hash of the parent commit, an author and committer, and the commit message. The commit object also has a hash stored in the .git/objects directory.

❯ git cat-file -p 46cbcbdd595811bb59cbbc307d061dfaa7478a85
  tree 6a58a71284938e520f1c21f24971dbe14821fbd7
  parent 45dda006756054c4fd77fd9a0423a7033f21dbfd
  author Sofia Fischer <sofia@philodev.com> 1752449645 +0200
  committer Sofia Fischer <sofia@philodev.com> 1752449645 +0200
  
  ✨ Post about bitemporal data

Branches, Heads, Tags #

Branches #

A Branch is a named reference to a commit. The branches are stored in the .git/refs/heads directory.

❯ ls .git/refs/heads
master  bitemporal

❯ cat .git/refs/heads/master
96b73cbfd9cb66356737ce3282986c2f8aa225b1

Deleting a branch will only delete the file with the reference to one commit. If the commit is not fully merged, this might delete the only reference to the commit making the commit “orphaned” and unreachable.

Head #

The head is a reference to the current branch. The head is stored in .git/HEAD and contains the name of the current The head references the commit that will be the parent of the next commit.

❯ cat .git/HEAD
ref: refs/heads/main

Head may also point to a commit directly, this is called a “detached head”. What sounds spooky, is actually a nice way to travel back in time and look at the state of the repository at a specific commit. The reason why “detached head” causes panic is that the next commit will not be on a branch, and will be orphaned if head is changed. It is a feature of branches that they change on commit to point to the latest commit on the branch. So the solution is to just switch back to a branch.

One way to “save” a commit in a detached head state is to add a tag to the commit. Tags are references to a specific commit. Now when head is changed, the commit will not be orphaned, as the tag will still point to the commit.

Making pretty Git Commits in Practice #

While the git commits of some developers look like the developer had a plan for their feature, which unfolded flawlessly - one commit contained refactoring to make the change easy, one commit added unreachable new code with tests, and one connected it in multiple places. The dream of every reviewer is most commonly not the result of a developer who planned all actions upfront in perfection, but rather the result of using git to create readable, easy to review commits.

Commiting some lines of a file #

Many of the features I can implement start out as a single commit, but then I realise some refactoring that should go in a separate commit. Git allows to stage only parts of a file with git add -p or git add --patch. This will show the changes in the file and allow you to select a subset of the changes to stage. For each patch of code git will ask you if you want to stage it:

• y - yes, stage this hunk
• n - no, do not stage this hunk
• a - stage this and all the remaining hunks in the file
• d - do not stage this hunk nor any of the remaining hunks in the file
• g - go to the selected hunk
• / - search for a hunk matching the given regex
• j - leave this hunk undecided, see next undecided hunk
• J - leave this hunk undecided, see next hunk
• k - leave this hunk undecided, see previous undecided hunk
• K - leave this hunk undecided, see previous hunk
• s - split the current hunk into smaller hunks
• e - edit the current hunk manually
• ? - print help”

In PyCharm (or Intellij) you can also stage parts of a file by selecting the lines you want to stage in the preview of the commit dialog.

Change something in a commited commit #

It is also possible to change something in a past commit to ensure a clean commit history. The easiest way to do this is using git commit --amend. This will take the current staged changes and add them to the last commit. This is only possible if the commit to change is the last commit.

If the commit is not further down the commit history, git allows to change a past commit by adding a new commit with the message fixup! <commit message>. As an easier way to do this, you can use git commit --fixup <commit hash>, in the GUI of PyCharm by right clicking on the commit to change and selecting “Fixup commit”.

As an example, to fix some spelling mistakes in a blog post ✨add post about bitemporal data after I changed the some internal things for my blog, I could end up with the following commits:

d01fd6 fixup! ✨add post about bitemporal data
c028f4 🚀 use go modules instead of git submodules
b032c2 ✨add post about bitemporal data

The fixup commit will be squashed into the b032c2 commit on interactive rebase or on merge if used with --autosquash.

Interactive Rebase #

Interactive rebase is a powerful tool to change the commit history. It allows you to reorder, squash, or edit commits.

git rebase -i <commit hash> will open an editor with a list of commits starting from the given commit hash.

> git rebase --interactive 96b73cbfd9cb66356737ce3282986c2f8aa225b1

pick 1e57725 # ✨  Post about git
pick be9293c # 🖋continue on git post
pick 3003a57 # 🖋add disclaimer for git post

# Rebase 96b73cb..3003a57 onto 96b73cb (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
#                    commit's log message, unless -C is used, in which case
#                    keep only this commit's message; -c is same as -C but
#                    opens the editor
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label

Don’t Panik - this is vim 😬 In this text file the pick can be replaced with for example break which will stop the reabase at this commit and allow making changes to the commit. Also, the order of the commits can be changed. With the decision what should be done with each commit, :wq will save the changes and exit the editor, and start the rebase. If all the commits are picked, the rebase will continue without stopping as a regular rebase. If there is a commit set to break, the rebase will stop at that commit, any changes can be made to the files; after which I can add / stage the changes and continue the rebase with git rebase --continue.

Mind that creating a new commit in this stage will insert a new commit into the history before the commit that is currently being edited.

In the GUI of PyCharm, the interactive rebase can be started by right-clicking on a commit and selecting “Interactive Rebase from here”. The GUI will show a list of commits than can be reordered, or selected and breaked by clicking on the “pause” button. The IDE will then stop at the selected commit and allow to make changes to the files, then again adding and continuing is also possible graphically (in the current version of PyCharm, the continue button is in the top left).

Why the fuzz? Conclusion #

Many of these features are not easy to learn, and feel spooky even the third time. Many people who use git find it hard to understand and not very user-friendly - which is just true. Git is so unintuitively that it even has the concept of " porcelane commands" which are commands that are easier to use and follow a mental model of git storing changes instead of trees; and “plumbing commands” which are the commands that require understanding of how git works internally.

But there are reasons why an advanced usage of git is worth the effort: A clean commit history provides information about how a file or repository evolved, which can be useful for new developers or those who revisit their project after some time. Clean commits in a Pull Request makes reviews much more understandable, and therefor easier and faster. It can separate the steps taken to implement a feature, and allow to understand the reasoning behind the changes. Also, I found it a good practice to split up the work in smaller steps that still all pass tests and work, makes my implementation more reliable, easier to map in my head, and easier to debug.

Happy Coding :)

From Temporal Data to Bitemporal State Data #

Temporal Data #

Temporal data is data modelling historic changes of something. Most data in modern dataases is temporal data just by storing a created_at. But intentionally temporal data is often implemented by a model having a actual_from and actual_to properties (or valid_to, active_to, ended_at …), which correspond to non-overlapping time periods.

Some examples for this:

• Price tracking over time: The cupcake did cost 3,55€ in the last week, but this week the price was increased to 3,75€, and on monday there will be a sale and it will only cost 3€
• library book rental history: The book was rented by Ari from 01.03 until 01.04, nobody took it for a month, then Noa took it and still has it.
• Chameleon color change over time: The color was green yesterday, and today the color changed from green to magenta.

From a technical point of view, this enables a nice implementation; a database constrain can ensure that there are no overlapping time periods.

import datetime


class ChameleonColor:
    actual_from: datetime.datetime
    actual_to: datetime.datetime | None
    color: str

Non linear data reporting #

The moment the data becomes available not in a linear way, the simple temporal data model becomes more complex to handle. What if the information on what color the chameleon has arrives not at the moment it changes, but only as it has been recorded to the system.

Keeping those reports can be necessary to proof what knowledge was present in a point in time, or just to understand what has happened when an error occurred, or to being able to react to unreliable data that might be canceled later.

This adds a new time dimension to the existing model: the point in time in which the actual information was recorded, and the point in time in which the recorded time was overwritten by more recent data. The naming of those properties vary (because naming is hard), a common combinations are actual_from and recorded_at coined by Martin Fowler ¹, valid_from and recorded_at coined by Richard Snodgrass ².

import datetime


class ChameleonColor:
    actual_from: datetime.datetime
    actual_to: datetime.datetime | None
    color: str
    recorded_at: datetime.datetime
    record_overwritten_at: datetime.datetime | None

Understanding Bitemporal Data #

Do we need a record_overwritten_at?

Just looking at a query to retrieve the state of the chameleon at a specific point in time, there is no need to add a column that just records when a period was overwritten. But to utilize database constraints to ensure that there are no overlapping time periods in the calculated reality, a record_overwritten_at that is None for the valid period makes everything easier.

How to read bitemporal data?

Looking at an example of a chameleon that was recorded green at 2025-01-01, then changed to magenta at 2025-01-05, but the system only recorded the change at 2025-01-09.

The SQL query to retrieve the current state of the chameleon at 2025-01-07 would look like this and return magenta:

SELECT color
FROM chameleon_color
WHERE actual_from <= '2025-01-07'
  AND (actual_to IS NULL OR actual_to >= '2025-01-07')
  AND (record_overwritten_at IS NULL) LIMIT 1

Visually, this can be represented as follows:

- - - - - - - - - - - - - - - -  Point in time imagine we are - now
2025-01-09 ------█|██   magenta
2025-01-09 -█████----   new green period
2025-01-01 -░░░░░░|░░   green, invalidated
                  ↑ Point in time we are looking for at 2025-01-07

The SQL query to retrieve the state of the chameleon at a specific point in time (like 2025-01-07) from the perspective of one day later (2025-01-08) asking effectively “it was 2025-01-07, which color would we think the chameleon had yesterday?” The SQL query would look something like this and return green:

SELECT color
FROM chameleon_color
WHERE actual_from <= '2025-01-07'
  AND (actual_to IS NULL OR actual_to >= '2025-01-07')
  AND recorded_at <= '2025-01-08'
  AND (record_overwritten_at IS NULL OR record_overwritten_at >= '2025-01-08')
ORDER BY recorded_at ASC LIMIT 1

Visually, this can be represented as follows:

2025-01-09 ------█|██   magenta
2025-01-09 -█████----   new green period
- - - - - - - - - - - - - - - -  Point in time imagine we are at 2025-01-08
2025-01-01 -░░░░░░|░░   green, invalidated
                  ↑ Point in time we are looking for at 2025-01-07

Mind that we do not get the “real” state of the chameleon at 2025-01-07, but the state as the system thought it was.

In case of a collision, should we add more rows or update?

Looking at an example of trying to insert a new period, that should overwrite an existing period:

A ------███-  more recent period
B -███████--  old period

We could just split the old period, saving pressures database storage:

A ------███-  more recent period
B -█████----  valid part of old period
B ------░---  invalid part of old period

While this works in theory, it is much harder to recalculate the reality of the data at a specific point in time. What accountants use since centuries to add a new period for to replace invalid old data preserves the history of the data. Making the rows immutable except for the record_overwritten_at column, allows much more trust in the data ensures no data is lost.

A ------███-  more recent period
B -█████----  valid state old period
B -░░░░░░░--  invalid state of old period

How to insert an period

An new period is inserted by

• find any collisions with existing periods
• invalidate the existing, older periods by setting their record_overwritten_at to the new periods recorded_at
• insert new periods to represent the current actual state of the data with recorded_overwritten_at set to NULL

Inserting Bitemporal Data #

Inserting a new period in chronological order with always open-ended periods is easy: invalidate the existing period, and create a new “existing” period that ends as the new period starts, and the new period.

But what if finite periods can be inserted in any order? Lets look at this example:

2025-01-05 --█-------  teal
2025-01-04 ------██--  turquoise
- - - - - - - - - -  Point in time magenta is inserted spanning over the whole time
2025-01-02 ██--------  green
2025-01-01 ----█-----  blue

With new data
2025-01-05 --█-------  teal
2025-01-04 ------██--  turquoise
- - - - - - - - -  
2025-01-03 --------█-  magenta 
2025-01-03 ---███----  magenta
2025-01-03 -█--------  magenta
2025-01-03 █---------  green
2025-01-03 -░░░░░░░░-  magenta
- - - - - - - - - 
2025-01-02 ░░--------  green
2025-01-01 ----░-----  blue

To insert the magenta period,

• the blue and green periods need to be invalidated, as they are older and colliding with the new period
• the new magenta period needs to be inserted, but invalidated as there are more recent, colliding periods
• a new green period needs to be inserted, because while the green period is partially colliding with the magenta period, it started before the magenta period and is still valid until the start of the magenta period.
• a new magenta period needs to be inserted to fill the gap between the green and teal period
• a new magenta period needs to be inserted to fill the gap between the teal and turquoise period
• a new magenta period needs to be inserted for the fraction of the magenta period that lasts longer than the turquoise period.

Types of Collisions (Side Quest) #

Following a first thought, I wanted to go by the flow of time and deciding for each period if it is newer or older, and what insertions or updates need to be performed.

The relationships between two periods can be one out of 13 defined relationships, mathematically defined by James Allen [^snodgrass]. While periods can be ordered by many things (the length, the start, the end, creation date), ordering by the start date leaves 7 relationship between each period and the next one.

Out of the 7 relationships, 6 will result in a collision that needs to be handled, and will result in at least two rows to be touched. Considering that any in any the 6 relationships, any of the periods can be the newer one, there are 12 cases to handle - a number of ifs that can quickly grow out of hand.

But looking at the complex example, the solution might be much simpler.

Handling sets of collisions #

Instead of trying to handle each period individually, it is much easier to handle sets of periods that are colliding.

1. The first set ist the set of all periods that are colliding with the new period, but are older and must be invalidated.
2. There might be one period that started before the new period started, and was invalidated by the new period. It has a fraction that must remain valid until the start of the new period. That fraction needs to be inserted as a new valid period.
3. There might be one period that ended after the new period ended, and was invalidated by the new period. It has a fraction that must remain valid after the end of the new period. That fraction needs to be inserted as a new valid period.
4. Create the new period, valid if there are no newer periods colliding with the new period, otherwise invalid.
5. The last set consists of all periods that are colliding with the new period, but are more recent than the new period. For each gab between those periods, a new period needs to be inserted that covers the gap and corresponds to the new period.

import datetime


def insert_period(actual_from: datetime.datetime, actual_to: datetime.datetime | None, color: str,
                  recorded_at: datetime.datetime):
    colliding_periods = queries.get_ordered_colliding_valid_periods(actual_from, actual_to)
    colliding_older_periods = [
        period for period in colliding_periods if period.recorded_at <= recorded_at
    ]
    colliding_newer_periods = [
        period for period in colliding_periods if period.recorded_at > recorded_at
    ]

    # 1. invalidate all colliding periods that are older than the new period
    for period in colliding_older_periods:
        if period.recorded_at <= recorded_at:
            period.recorded_overwritten_at = recorded_at
            period.save()

    # 2. check for start fraction of older periods
    if (first_period := colliding_older_periods[0]) and first_period.actual_from < actual_from:
        create_period(
            actual_from == first_period.actual_from,
            actual_to=start,
            color=first_period.color,
            recorded_at=recorded_at)
    # 3. check for end fraction of older periods
    if ((last_period := colliding_older_periods[-1])
            and actual_to is not None
            and (last_period.actual_to is None or last_period.actual_to > actual_to)):
        create_period(
            actual_from=actual_to,
            actual_to=last_period.actual_to,
            color=last_period.color,
            recorded_at=recorded_at)

    # 4. create new period
    maybe_overwriten_at = colliding_newer_periods[0].recorded_at if colliding_newer_periods else None
    create_period(
        actual_from=actual_from,
        actual_to=actual_to,
        color=color,
        recorded_at=recorded_at,
        record_overwritten_at=maybe_overwriten_at)

    # 5. handle gaps between newer periods
    fractions = []
    current_start = actual_from
    for period in colliding_newer_periods:
        if current_start < period.actual_from:
            fractions.append((current_start, period.actual_from))
        if period.actual_to is not None or period.actual_to > actual_to:
            fractions.append((current_start, actual_to))

Bitemporal Databases #

These patterns do not feel like most relational database patterns. After some reasearch, I found databases that are specialised to bitemporal data, like XTDB. The database is written in Clojure, and uses event storage under the hood to handle second dimensions of time ³. Looking at the pattern, the partial immutability of the data, and the inserting of new periods as new entries, even if it to update existing data, id displays indeed many similarities to event sourcing.

Some Thoughts on downsides of Bitemporal Data #

Looking back at the example, inserting 5 rows for a new single new period is a lot. There is not really a way to euphemise an algorithm that will worst case insert n + 1 rows for a new record, with n being the number of existing records. Tables of this bitemporal state data can grow very large, and the database will need to handle a lot of rows. The patterns discussed here are not trivial, and do not match what at least I learned in university about normalized databases.

But looking at alternatives; updating the rows and then handling all the case of overlapping periods will result in an even more complex algorithm, and will provide less trust in the data. Also, this concept of bitemporal data is not new, some sources I found date back to 2000, and there are many resources to find on the topic. Having the need for bitemporal data, and trying to optimize for storage consumption, is a trade-off that will cost in readability, complexity, and reasearchability of the used patterns - if you think otherwise I would be happy to hear your thoughts!

Conclusion #

Bitemporal data is always complex to wrap around. The amount of articles and resources on this topic is extensive, and I did read through many of posts, one book, and one podcasts to understand some more granular details of the patterns. They all use different visualizations, often different namings, with mostly the same approaches to the problem. So while I am confident that the approach I presented here is one proven way to handle bitemporal data.

The main inspiration for this article was Simeon (big thanks!), who phrased it in his talks as “You understand bitemporal data, until you don’t. And when you need it, you understand it again”.

This article, as all my public learnings, is my way to understand things, and hopefully have an easy time to re-understand the concepts and patterns.

Happy coding :)

Meta Programming #

As the name suggests, Metaprogramming means writing programs that interact with the code itself.

The most simple example is checking the type of a variable, so gathering information about the code at runtime, and making decisions based of that. Especially when coding within a framework, there are many contact points with meta programming, for example when using decorators that “magically” transform a function into a job, inheriting from a class that “magically” transforms a class into a model that triggers migrations on change. Metaprogramming is a powerful tool, especially when writing code that is used by other developers consuming their code, like a framework.

When to use it? #

Metaprogramming should be used with caution. It is often used to hide complexity from the developer who just wants to use the magic. This hiding is what makes many frameworks so magically working without the need to understand the internal processes. But on the other hand, it makes it really hard to go through the code, following the flow of data, because it is hidden - objects or classes are touched, manipulated, or read at places where it is not expected.

It sometimes can be tempting to implement a program using metaprogramming, but it is often better to use simpler, understandable code to ensure the next developer can continue.

Metaprogramming in Python #

This post is strongly inspired by the Meta Programming Course by Trey Hunner ¹.

Introspection: I want to know about the technical properties of an object #

Python provides many built-in functions to introspect objects like does it have a certain attribute hasattr, what is the type of the object type, what is the name of the class __name__, or is it callable callable. I would like to highlight Annotated Properties, because they not only enable reading meta information about a property, but also a way to add meta information to a class property.

An easy, somewhat common form of Metaprogramming, given meta information about a class property, that are stored there using the Annotated Property. Using the __annotations__ attribute of a class, you can get the annotation of a class, and build therefore code based on how the other code is written or types.

from typing import Annotated, get_args


class Magic:
    def __init__(self, is_dark: bool = False):
        self.is_dark = is_dark


class Wizard:
    wand: Annotated[str, "magic", Magic(True)] = "Magic Staff"


merlina = Wizard()
annotation = merlina.__annotations__.get("wand", None)

print(annotation)

if any(isinstance(magic, Magic) for magic in get_args(annotation)):
    print("Merlina can use magic!")

typing.Annotated[str, 'magic', <__main__.Magic object at 0x1006bffd0>]
Merlina can use magic!

Decorators: I want to add functionality to a function #

Decorators are meta programmatic tools that diverge from the expected flow of the code. They are used to wrap around a function (or class). A decorator is a function that takes another function as an argument and returns a new function that adds some kind of behaviors. ^{1 2}

The wrapper function should be able to receive the same arguments as the original function, so it needs to hand down its *args, **kwargs to the original function. It should also return the value of the original function.

def make_magic(func):
    def magic_wrapper(*args, **kwargs):
        print("✨Magic started ✨")
        value = func(*args, **kwargs)
        print("✨Magic ended ✨")
        return value

    return magic_wrapper


def abracadabra(input: str) -> None:
    print("Abracadabra " + input + "!")


abracadabra = make_magic(abracadabra)
abracadabra("Hoc")

✨Magic started ✨
Abracadabra Hoc!
✨Magic ended ✨

Now the original abracadabra function is overwritten by the magic_wrapper function which contains the original function. Python provides a syntactic sugar to make this easier, using the @ symbol to decorate the function instead of abracadabra = make_magic(abracadabra).

@make_magic
def abracadabra(input: str) -> None:
    print("Abracadabra " + input + "!")


abracadabra("Hic")

✨Magic started ✨
Abracadabra Hic!
✨Magic ended ✨

Decorators can be very versatile! They can be used to add functionality to a function, allow some meta information added to the function, add a parameter, or (e.g. a decorator to skip a test) even to replace the function itself.

Mind the scopes, variables defined in the decorator are not available in the wrapper function, but Python allows adding variables to

def make_magic(func):
    def magic_wrapper(*args, **kwargs):
        print("✨Magic started ✨")
        value = func(*args, **kwargs)
        magic_wrapper.mana -= len(str(args) + str(kwargs)) - 6
        print("✨Magic ended ✨Mana left: " + str(magic_wrapper.mana))
        return value

    # Mind the scopes! A variable defined in the decorator is not available in the wrapper function.
    # Also avoid adding variables to the wrapped function (func) itself to avoid unexpected side effects. 
    magic_wrapper.mana = 100
    return magic_wrapper


@make_magic
def abracadabra(input: str) -> None:
    """  A magic function that prints a message. """
    print("Abracadabra " + input + "!")


print(abracadabra("Mundus"))
print(abracadabra("est"))
print(abracadabra("magia"))

✨Magic started ✨
Abracadabra Mundus!
✨Magic ended ✨Mana left: 93
✨Magic started ✨
Abracadabra est!
✨Magic ended ✨Mana left: 89
✨Magic started ✨
Abracadabra magia!
✨Magic ended ✨Mana left: 83

There is one problem with this approach: What looks like the original function is in fact a wrapper function. The information of identity, DocStrings, etc is lost.

print(help(abracadabra))

Help on function magic_wrapper in module __main__:

magic_wrapper(*args, **kwargs)

To avoid this, Python provides a functools.wraps decorator. This decorator is used to update the wrapper function with all the attributes of the original function, including its name, docstring, and module. Internally it adds the original function to the __wrapped__ property of the wrapper function.

from functools import wraps


def make_magic(func):
    @wraps(func)
    def magic_wrapper(*args, **kwargs):
        print("✨Magic started ✨")
        value = func(*args, **kwargs)
        print("✨Magic ended ✨")
        return value

    return magic_wrapper


@make_magic
def abracadabra(input: str) -> None:
    """  A magic function that prints a message. """
    print("Abracadabra " + input + "!")


print(help(abracadabra))

Help on function abracadabra in module __main__:

abracadabra(input: str) -> None
A magic function that prints a message.

Decorators in Practice: Most frameworks use decorators to add functionality to functions or classes, even many libraries. For example Celery ³ uses decorators to define tasks. Well written decorators are easy to find a project by searching for @wraps in the code, so feel free to find out how your project uses decorators.

Class Decorator work the same way, but instead of a function, a class is passed to the decorator, and a class is returned. A common used Class Decorator is @dataclass, which returns the same class as was passed in, but with dunder methods added.

Descriptors: I want to control how a class attribute is accessed or modified #

Sometimes it is handy to have an attribute that is not just a simple value, but calculated based on another value, like the radius of a circle and its area. Also it would be nice if the attribute could be set and the related value would also update. Such a behavior can be implemented using a descriptor.

Descriptors are objects that define how attributes are accessed or modified. They are defined by a class that implements either of the three methods __get__, __set__, or __delete__ ⁴. A descriptor that implements only the __get__ method is called a non-data descriptor, while a descriptor that implements more is called a data descriptor. To understand how descriptors work, it is important to understand how Python looks up attributes.

To understand how Python looks up attributes on an object the __dict__ attribute of the object is used. This attribute is a dictionary that contains the properties of the object, or if used on the type of the object, the class properties. Instance properties override class properties.

class Potion:
    milliliter: int = 50
    color: str = "yellow"

    def __init__(self, color: str):
        self.color = color


invisibility_potion = Potion("magenta")
print("Instance Dict")
print(invisibility_potion.__dict__)
print("Class Dict")
print(Potion.__dict__)

Instance Dict
{'color': 'magenta'}
Class Dict
{'__module__': '__main__', '__annotations__': {'milliliter': <class 'int'>, 'color': <class 'str'>}, 'milliliter': 50, 'color': 'yellow', '__init__': <function Potion.__init__ at 0x105751f30>, '__dict__': <attribute '__dict__' of 'Potion' objects>, '__weakref__': <attribute '__weakref__' of 'Potion' objects>, '__doc__': None}

In Python the logic of attribute lookup is implemented in C for performance reasons (after all, this code will run on every attribute access). The Code can be found in the CPython repository in a method called _PyObject_GenericGetAttrWithDict. But a pure Python implementation is provided in the Python docs⁴.

def object_getattribute(instance: object, attribute: str):
    """ Emulate PyObject_GenericGetAttr() in Objects/object.c """
    instances_type = type(instance)
    type_dict_at_attribute = getattr(instances_type, '__dict__', {})[attribute]

    # NonData Descriptors are defined by implementing __get__ method
    has_non_data_descriptor = hasattr(type_dict_at_attribute, '__get__')
    # Data Descriptors are NonData Descriptors that implement __set__ or __delete__ method
    has_data_descriptor = False
    if has_non_data_descriptor:
        if hasattr(type_dict_at_attribute, '__set__') or hasattr(type_dict_at_attribute, '__delete__'):
            has_data_descriptor = True

    # if there is a Data Descriptor, return the result of its __get__ method
    if has_data_descriptor:
        return type_dict_at_attribute.__get__(instance, instances_type)

    # If the attribute is defined in the instance's __dict__, return it
    if attribute in instance.__dict__:
        return instance.__dict__[attribute]

    # If there is a non-data descriptor, return the result of its __get__ method
    if has_non_data_descriptor:
        return type_dict_at_attribute.__get__(instance, instances_type)

    # If the attribute is defined in the class's __dict__,
    if attribute in instances_type.__dict__:
        return instances_type.__dict__[attribute]
    raise AttributeError(attribute)

Python Descriptors in Practice Descriptors are commonly used in Python to implement properties using the decorator @property, which disguises the setter and getter methods into methods of the class.

class Potion(object):
    milliliters = 50
    usage_size = 10

    def use(self) -> None:
        if self.milliliters > 0:
            self.milliliters -= self.usage_size
        else:
            raise ValueError("Potion is empty")

    @property
    def usages(self) -> int:
        return int(self.milliliters // self.usage_size)

    @usages.setter
    def usages(self, value: int) -> None:
        self.milliliters = value * self.usage_size


invisibility_potion = Potion()
print(f"I made 🧪 Potion with {invisibility_potion.milliliters}ml")
invisibility_potion.use()
print(f"After using the potion, I have {invisibility_potion.milliliters}ml left")
invisibility_potion.usages = 20
print(f"After refilling the potion, I have {invisibility_potion.milliliters}ml left")

I made 🧪 Potion with 50ml
After using the potion, I have 40ml left
After refilling the potion, I have 200ml left

An example of a data descriptor that abstracts logic away would be Djangos ForeignKey ⁵. While the developer who wants to use the framework does not need to know how the ForeignKey works, they can use it like property, but Django uses the descriptor to implement the retrival or storage of the related object in the database.

MetaClasses: I want to create a new Type and control how classes are created #

All Types in Python are(Meta)Classes that inherit from type, but all instances of a Type are Classes.

class Wizard:
    pass


print("A wizard instance is: " + str(type(Wizard())))
print("Wizard class is a: " + str(type(Wizard)))

A wizard instance is: <class '__main__.Wizard'>
Wizard class is a: <class 'type'>

Having a MeterClass allows to control how classes are created, for example if they are Singletons ⁶. The usage of methods like __new__ and __init__ can be overwritten to control the creation of many classes.

class OrderOfWizards(type):
    """
    The Maiar of the Order of Wizards may be found in many places,
    but wherever one class of Maiar is found, it will be always the same instance.
    """
    wizards: dict[str, "OrderOfWizards"] = {}

    def __call__(wizard_class, *args, **kwargs):
        if str(wizard_class) not in wizard_class.wizards:
            wizard_class.wizards[str(wizard_class)] = super(OrderOfWizards, wizard_class).__call__(*args, **kwargs)
        return wizard_class.wizards[str(wizard_class)]


class Gandalf(metaclass=OrderOfWizards):
    color: str = "grey"


gandalf = Gandalf()
print(f"There he is, Gandalf the {gandalf.color}!")
gandalf.color = "white"
gandalf = Gandalf()
print(f"There he is, Gandalf the {gandalf.color}!")

There he is, Gandalf the grey!
There he is, Gandalf the white!

Metaclasses in practice: Django uses metaclasses to control models. The Model class that is extended by all models has the BaseModel as MetaClass⁷. This class is used to handle Model metadata, like the database table or the migration status.

Conclusion #

All these tools are powerful, and looking how frameworks use them to provide a magically smooth experience for the developer can be inspiring. But it is important to understand, that many of these tools are not easy to understand without knowing the underlying mechanics. They are hard to debug, it’s not easy to follow the flow of code, and when not implemented correctly, they can lead to unexpected behavior.

If you think about using them, ask yourself if you just want to use the magic like a dark magic apprentice, or if you explicitly want to hide complexity from the developer who will use your code.

On this very post, Happy Understanding :)

Changing Language #

I have been coding in PHP since 2018 professionally, and after 6 years I am now switching to Python.

Coding in one language does mean more than knowing that syntax. Mastering a language meaning understanding pitfalls, how some parts of the language work internally, what tools the language provides for meta programming; it includes knowing the best practices, the accepted and unacceptable hacks, which style will be understood by the next developer and what is too “we don’t to that in our language”. Examples of what I mean by that are excessive use of pass-by-references in PHP or “Panic-driven” GoLang Code.

The language influences the architecture, the usage of patterns, and the tools - and often underestimated: the projects. Maybe there is a PHP Developer out there who never had to implement a Web Shop, but I guess they are as rare as Python Developers who never implemented a Data Science project. Different languages have different ecosystems, different specializations. Python has a strong Data Science and Machine Learning community, historically because of its beginner-friendliness, to now owning established libraries for data manipulation and interpretation. While PHP still dominates the web, historically started as template engine, growing whatever Wordpress is now, and powering many old and new web projects.

The different levels of learning a new language #

First, how do you run the code, aka how do you get the “Hello World” on the screen? Is the common practice using a docker container or a virtualenv to control language version? How do you install packages? How do you run the tests? Which frameworks are common?

Supported by a nice coding AI, its about learning the syntax, the data structures, the libraries, and the tools. What can you do with the langauge, where are limits and the pit falls? How are tests written and how does debugging work?

At some point, it is unavoidable to include the human factor. How doe the developers of the language think, what are the common practices and patterns? How do they uses the given tools?

Python Data Structures #

Typing: Typing always conveys intention, and in Python this can be done even more verbose than in PHP. Python provides Generators, supports Union types, specialised Types like Literals, and Annotated Types. Python Datastructures are so diverse and can convey so many intentions: Is the content ordered like a list? May it not contain duplicates like a set? Is it immutable like a tuple? Is it handled lazy like a generator? Does it have defaults like a DefaultDict? Is it designed to be stored in a database like a flag? ¹

Table infos from ²

Lists are the most common data structure as they support many operations that also make them easy to use stacks or queues. Also they support the awesome list comprehensions to create lists almost in mathematical notation.

[snake for snake in snakes if snake.size > 2]

As mutable data structures, they also cause one of the weirdest pitfalls: Python uses pass by reference, even on function parameter defaults. The Python compiler will go through a function parameter with the default value [] and will create a pointer to a list. If the function is called multiple times, the same pointer is used, potentially reusing a filled list. Avoid this by using None as default value or use immutable data structures like tuples. ³

Magic Methods #

In Python there are magic methods that allow you to overload operators and define what happens with the Class on casting, or how Classes can behave like a list or a dictionary - “Dunder Methods” (because “Doubler Underscore”) Be careful with them - if it is not doubtlessly clear what the code does, the developer can not easily click on the operator to check the behavior like with a non-magic function, eventually tracing down inheritances to find the magic method to understand the code.

Nevertheless, they are a powerful tool to make the code more readable and to convey intention.

class Snake:
    latin_name: str = "Serpens"
    size: int

    def __init__(self, size: int):
        """Constructor"""
        self.size = abs(size)

    def __str__(self) -> str:
        """String representation"""
        return ">-@" + "=" * self.size + "---"

    def __add__(self, other) -> "Snake":
        """Add two snakes together with a + operator"""
        if not isinstance(other, Snake):
            raise TypeError("Can only add Snake to Snake")
        return Snake(self.size + other.size)

    def __eq__(self, other):
        """Check if two snakes are equal"""
        if not isinstance(other, Snake):
            raise TypeError("Can only compare Snake to Snake")
        return self.size == other.size

    def __call__(self, other) -> tuple["Snake", "Snake"]:
        """Call the instance like a function"""
        half_size = int(self.size / 2)
        return Snake(half_size), Snake(half_size)

    def __iter__(self):
        """Iterate over the snake like it is a list"""
        for segment in range(self.size):
            yield Snake(1)

    def __getitem__(self, index: int) -> "Snake":
        """Get a segment of the snake like it is a dictionary"""
        if index < 0 or index >= self.size:
            raise IndexError("Index out of range")
        return Snake(1)

These are only a few examples of the magic methods that Python provides. Much more are listed here by Trey Hunner ⁴.

Typing #

Python allows much more Typing options than PHP, while both support Type unions, None type, and casting (and magic methods to cast), Python also has some special things.

Type Alias Because Python Types are somewhat classes, you can create Type Aliases.

Hydra = list[Snake]
snakes: Hydra = [Snake(1), Snake(2), Snake(3)]

Type Hinted Functions Python supports Typed Functions, which I feel like a blessing, as it seems to be “Pythonic” to hand over functions as parameters to make functions work in different use cases.

def feed(snake: Snake, food_count: int) -> Snake:
    snake.size += 1
    return snake


def sleep(snake: Snake, days: int) -> Snake:
    snake.size -= 1
    return snake


snake_care_methods: list[Callable[[Snake, int], Snake]] = [feed, sleep]

Generics A feature PHP misses, Generics allow to define that a Type will stay consistent, for example that a function that sorts a list of one type will also return a list of the same type, independent of the type.

Annotated Python even allows adding more meta information to a Type, although this is often not checked by the type checker, but it can be used with ValueRange or other self implemented classes, to give contexts like “Is this Percentage a float between 0 and 100, or 0 and 1?”, or “Does country refer to a country code or a country name?”.

from typing import Annotated


class Snake:
    size: Annotated[int, "Should be nice",] = 3


severus = Snake()
print(severus.__annotations__)

>> > {'size': typing.Annotated[int, 'Should be nice']}

Pythonic Culture #

Code should be natural to interact with, which also means that if follows cultural conventions (which starts with naming, but extends up to patterns). It means that it uses magic with caution, and is braced for misuse - which is easiest if the developer is well-informed about how the language can be misused or what magic is common knowledge. ¹

In Python there is so much possible. There are many conventions on how to solve common things, and while the fun part about switching languages is sometimes challenging those conventions, there is also a lot to learn. When to use dataclasses, when Attr, and when Pydentic ⁵? Why to Python developers so rarely use Dependency Injection, and how else do I solve Inverting Control Problems ⁶? When to use Flags? Why is nobody using match case?

What Pythonic is and what not is still something I am learning, but while many learnings in a new language can be discovered in self-study; this is something that requires working in one project with other experienced developers.

Conclusion: Stay Curious #

Looking back, it wasn’t a big step, but in between it felt like one. In between, I can contribute with my experiences of a different language and its patterns to architecture decisions, while at the same time asking questions why my for loop doesn’t work.

I am looking forward to share my experiences with learning Python.

Happy coding :)

Queues #

Queues are a way to manage asynchronous communication between different parts of a system.

Message queues are like a mailbox where messages are stored by one service, and independently / asynchronously picked up by (another or the same) service for processing.

Queues come with great benefits:

• Decoupling of services of different technologies or even programming languages
• Control of the flow of messages and therefore the system performance
• Increase loose coupling and scalability

AMQP #

AMQP 0-9-1 (Advanced Message Queuing Protocol) is a messaging protocol that enables conforming client applications to communicate with conforming messaging middleware brokers, or

A defined set of messaging capabilities called the “Advanced Message Queuing Protocol Model” (AMQ model). The AMQ model consists of a set of components that route and store messages within the broker service, plus a set of rules for wiring these components together.