whtsky's blog

Make (my) git fetch & pull 100x faster on monorepos

Sat, 06 Sep 2025 00:00:00 +0000

I work on a massive git monorepo where hundreds of developers push changes daily.
The repo is huge, and most commits happen on branches I don’t even care about. Unsurprisingly, Git doesn’t perform well at this scale.

Why `git fetch` is slow by default

By default, running git fetch downloads everything from the remote, even branches you’ll never touch.
Instead, if you only fetch the branch you’re working on, e.g. git fetch origin main, Git only pulls objects related to main, skipping everything else.

On my machine, this simple change made fetches about 100x faster, just by downloading less.

But what about `git pull`?

Here’s the problem:
git pull (which is really just git fetch + git rebase/merge) will always fetch all branches by default. That means even if you only care about your current branch, git pull stays slow.

The fix? Run git pull origin main.
This fetches only what you need and makes pulls significantly faster.

Making it less tedious

Of course, typing git pull origin main every single time is painful.
My solution is to create a Git alias that automatically pulls only the current branch (credit to this Stack Overflow post):

1
2
3

# Add this to your ~/.gitconfig
[alias]
    p = !git pull origin $(git branch --show-current)

Now, instead of git pull, just run git p, and enjoy faster pulls without thinking about it.

Takeaway

If you’re stuck with a giant monorepo, you don’t have to suffer every time you run git pull.
By narrowing fetches to just the branch you’re on and automating it with an alias, you can get massive speedups with almost no downside.

Decorator type gymnastics in Python

Fri, 28 May 2021 00:00:00 +0000

You need to type hints your decorator

Say you have a simple decorator for adding logging before calling a function.

import logging

def add_log(f):
    def wrapper(*args, **kwargs):
        logging.info("Called f!")
        return f(*args, **kwargs)
    return wrapper


@add_log
def two_sum(a, b):
    return a + b

One day you decide to add type hints for this module -- it's easy to add type hints for two_sum

1
2

def two_sum(a: int, b: int) -> int:
    return a + b

But you need to add type hints for your decorator (add_log in this case) too, or you'll get Anyed wrapped function. mypy's reveal_type can be used for verifying this.

import logging

def add_log(f):
    def wrapper(*args, **kwargs): # type: ignore
        logging.info("Called f!")
        return f(*args, **kwargs)
    return wrapper


def two_sum(a: int, b: int) -> int:
    return a + b

reveal_type(two_sum)  # Revealed type is 'def (a: builtins.int, b: builtins.int) -> builtins.int'
reveal_type(add_log(two_sum))  # Revealed type is 'Any'

Simple type hints for simple decorators

Let's adding type hints for this simple decorator. For a simple decorator which doesn't modify the functions' arguments and return (like add_log above ), TypeVar should do the job pretty well.

from typing import TypeVar, Callable, cast
import logging

TCallable = TypeVar("TCallable", bound=Callable)

def add_log(f: TCallable) -> TCallable:
    def wrapper(*args, **kwargs): # type: ignore
        logging.info("Called f!")
        return f(*args, **kwargs)
    return cast(TCallable, wrapper)


@add_log
def two_sum(a: int, b: int) -> int:
    return a + b


reveal_type(two_sum)  # Revealed type is 'def (a: builtins.int, b: builtins.int) -> builtins.int'
reveal_type(add_log(two_sum))  # Revealed type is 'def (a: builtins.int, b: builtins.int) -> builtins.int'

Hard type hints for hard decorators

But, what if you want to modify the arguments and/or return value? Well, there's no easy way to type arguments, but at least you can type return value correctly

from typing import TypeVar, Callable, Awaitable, cast
R = TypeVar('R')
def sync_to_async(f: Callable[..., R]) -> Callable[..., Awaitable[R]]:
    async def wrapper(*args, **kwargs): # type: ignore
        return f(*args, **kwargs)
    return cast(Callable[..., Awaitable[R]], wrapper)

@sync_to_async
def two_sum(a: int, b: int) -> int:
    return a + b

reveal_type(two_sum)  # Revealed type is 'def (*Any, **Any) -> typing.Awaitable[builtins.int*]'
reveal_type(two_sum(2, "3"))  # Revealed type is 'typing.Awaitable[builtins.int*]

The dark way for typing arguments & return values

You can do some type gymnastics.. by generating numbers of TypeVars and using overload.

from typing import TypeVar, Callable, Awaitable, overload

A = TypeVar('A')
B = TypeVar('B')
C = TypeVar('C')
D = TypeVar('D')
E = TypeVar('E')
RV = TypeVar('RV')
@overload
def sync_to_async(f: Callable[[A], RV]) -> Callable[[A], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B], RV]) -> Callable[[A, B], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B, C], RV]) -> Callable[[A, B, C], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B, C, D], RV]) -> Callable[[A, B, C, D], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B, C, D, E], RV]) -> Callable[[A, B, C, D, E], Awaitable[RV]]:
    ...
def sync_to_async(f):
    async def wrapper(*args, **kwargs):
        return f(*args, **kwargs)
    return wrapper

@sync_to_async
def two_sum(a: int, b: int) -> int:
    return a + b

@sync_to_async
def do_log(content: str) -> None:
    print(content)

reveal_type(two_sum)  # Revealed type is 'def (builtins.int*, builtins.int*) -> typing.Awaitable[builtins.int*]'
reveal_type(do_log)  # Revealed type is 'def (builtins.str*) -> typing.Awaitable[None]'

If you insist to go this way, here's the code snippet I used for generating code above:

gymnastics = 5

for i in range(gymnastics):
    char = chr(i + 65)
    print(f"{char} = TypeVar('{char}')")

print("RV = TypeVar('RV')")

for i in range(gymnastics):
    chars = [chr(n + 65) for n in range(i + 1)]
    args = ", ".join(chars)
    print(f"""@overload
def sync_to_async(f: Callable[[{args}], RV]) -> Callable[[{args}], Awaitable[RV]]:
    ...""")

The Future: PEP-612

PEP-612 defines ParamSpec and Concatenate. They can make type hinting decorators pretty easy:

from typing import Concatenate, ParamSpec

P = ParamSpec('P')
R = TypeVar('R')

def with_context(f: Callable[Concatenate[Context, P], R]) -> Callable[P, R]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> R:
        return f(context, *args, **kwargs)
    return inner

@with_context
def request(context: Context) -> int:
    return 42

The sad thing is PEP-612 is not widely supported, as of now mypy does not fully support it.

Fin

May the type be with you.

Don't forget `py.typed` for your typed Python package

Wed, 17 Feb 2021 00:00:00 +0000

Intro

pixelmatch-py, a Python library I maintained for comparing images, received a Pull Request to add type hints about 10 months ago. After merging this PR & releasing a new version, I thought the library's users will magically get their type working.

Until today, I was reading cryptography's Changelog and a line got my attention:

Added a py.typed file so that mypy will know to use our type annotations.

After reading PEP-561 and mypy documentation I'm sure that I didn't publish the package right: I should include a py.typed file, or the type checker won't use the type hints provided by the package.

Adding `py.typed`

It's faily simple to include this file: just touch a py.typed file in your package directory and include it in your distribution.
I'm using poetry, ~~so I added~~

packages = [
  {include = "pixelmatch"},
  {include = "pixelmatch/py.typed"},
]

~~under the [tool.poetry] section of pyproject.toml.~~

Update: As pointed out by Jürgen Gmach, poetry includes all files in your packages (including py.typed file!) by default, so there's no need to include the file manually.

If you're using setup.py, you can add package_data to setup call:

1
2
3

setup(
    package_data={"pixelmatch": ["py.typed"]},
)

Release a new version for your package then type informations from your packages should works.

Fin

If you're a Python package maintainer, be sure to include py.typed file for your typed package!

Using Celery with RabbitMQ's Lazy Queue

Wed, 15 Apr 2020 00:00:00 +0000

Celery & RabbitMQ basics

Skip this part if you know what Celery & RabbitMQ is.

Celery is a task queue system in Python. Celery can help you run something in the background, schedule cronjobs and distribute workloads across multiple servers.

RabbitMQ is a message broker. A message broker is a program to help you send messages. Celery needs to use a message broker to send & receive messages, and they highly recommend using RabbitMQ.

What is Lazy Queue?

When you publish a message into a RabbitMQ's regular queue, the message is stored into the memory so they can be delivered to consumers pretty fast. RabbitMQ will try to keep as much as messages it can until it hit a memory threshold. When it happens, RabbitMQ will save messages to disk("page out"), which will block the queue and prevent publishers from publishing any new messages.

As you may notice, if you have a lot of messages to publish, a regular queue will

use a lot of memory, depending on your message size and number
sometimes stop receiving new messages ( when paging out ), make the performance less predictive.

A lazy queue will try to save the message to disk as early as possible, therefore minimize the memory usage ( use constant space in most cases ) with the cost of throughput time, disk I/O and CPU usage.

Should I use Lazy Queue?

I'd always use lazy queues unless there are some really high-performance requirements. Lazy queues use significantly less memory and can hold a lot more messages than regular queues. CloudAMQP recommends to enable lazy queues to get predictable performance..

But before you do, I highly recommend you to carefully read the Caveats and Limitations section of lazy queues' documentation, and do some benchmarks yourself.

How to Celery with RabbitMQ's Lazy Queue

Specify queue arguments in Celery

Say you have a celery app

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_routes = ([
    ('account.tasks.*', {
        'queue': 'account'
    }),
    ('server.tasks.*', {
        'queue': 'server'
    })
])

You can specify queue_arguments to make queues lazy.

from celery import Celery
from kombu import Queue

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_queues = [
    Queue('account', queue_arguments={'x-queue-mode': 'lazy'}),
    Queue('server', queue_arguments={'x-queue-mode': 'lazy'}),
]

app.conf.task_routes = ([
    ('account.tasks.*', {
        'queue': 'account'
    }),
    ('server.tasks.*', {
        'queue': 'server'
    })
])

Or, if you haven't specified any task routes

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

In this case, Celery will route them into a default queue called celery by default. You need to change the default queue into lazy mode.

from celery import Celery
from kombu import Queue

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_queues = [
    Queue('celery', queue_arguments={'x-queue-mode': 'lazy'}),
]


@app.task
def add(x, y):
    return x + y

If the queue is already created, you may encounter a PRECONDITION_FAILED error saying the queue was declared with inequivalent arg 'x-queue-mode'. You should consider deleting the queue ( WILL LOSE ALL MESSAGES INSIDE IT ) or changing queue mode via policy.

Changing Queue Mode using RabbitMQ's policy

You can also change queues' mode using RabbitMQ's policy. This allows you to change an existing queue's mode without removing & recreating it. You can set a policy in shell using rabbitmqctl:

`1`	`rabbitmqctl set_policy Lazy "^celery$" '{"queue-mode":"default"}' --apply-to queues`

The above command changes the queue celery into lazy mode.

I prefer to specify queue arguments in Celery since it can be version controlled and don't require extra operations on the production server.

Enjoy.

Fixing Django's "populate() isn't reentrant" by printing traceback

Sun, 12 Apr 2020 00:00:00 +0000

Note: This article is based on Django 2.2.11

I was maintaining a huge Django app. Everything looked fine until I tried to run python manage.py -h:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "venv/lib/python3.6/site-packages/django/apps/registry.py", line 83, in populate
    raise RuntimeError("populate() isn't reentrant")
RuntimeError: populate() isn't reentrant

Hmmm, okay.

The exception was from execute_from_command_line in manage.py, which is generated by Django thus in great chance not the root cause of the exception.
I googled populate() isn't reentrant but the results weren't helpful enough to me.

Let's dig into Django's source code then. Followed the traceback from exception we can locate the populate function

# An RLock prevents other threads from entering this section. The
# compare and set operation below is atomic.
if self.loading:
    # Prevent reentrant calls to avoid running AppConfig.ready()
    # methods twice.
    raise RuntimeError("populate() isn't reentrant")
self.loading = True

Looks like the populate got called twice. Since execute_from_command_line is expected, let's record the traceback from the other call and print it out.

Modify venv/lib/python3.6/site-packages/django/apps/registry.py:

# An RLock prevents other threads from entering this section. The
# compare and set operation below is atomic.
if self.loading:
    # Prevent reentrant calls to avoid running AppConfig.ready()
    # methods twice.
    raise RuntimeError("populate() isn't reentrant.\n" + "\n".join(self.prev_traceback))
self.loading = True
import traceback
self.prev_traceback = traceback.format_stack()

Rerun python manage.py -h, and we can see the nice traceback.

  File "server/__init__.py", line 1, in <module>
    from .celery import app as celery_app

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "server/celery.py", line 11, in <module>
    django.setup()

  File "venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)

  File "venv/lib/python3.6/site-packages/django/apps/registry.py", line 85, in populate
    self.prev_traceback = traceback.format_stack()

Looks like someone called django.setup() in our celery module.

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'balisong_server.settings')
django.setup()

app = Celery('balisong')

After talking with the code author and looking into celery's document, I'm definite that the django.setup() here should be removed.
And removing this solves the problem.

Lessons learned: when in doubt, just print traceback.