Grouping by categories is quite common in web applications, from movies, courses or any other resource that presents a hierarchical relationship to another object. In Django there are different ways to model these relationships. Probably, the first that will come to your mind will be to create a category object, and then relate it by means of a ForeignKey with a subcategory.&#xA;<h2 id="one-subcategory-or-level-per-model">One subcategory or level per model</h2>&#xA;What I meant by one category or level per model is something like this:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"># app/models.py&#xA;from django.db import models&#xA;&#xA;class Category(models.Model):&#xA; name = models.CharField(max_length=256)&#xA; # other properties&#xA;&#xA;class SubCategory(models.Model):&#xA; name = models.CharField(max_length=256)&#xA; # other properties&#xA; category = models.ForeignKey(Category, related_name=&#34;subcategories&#34;, blank=True, null=True, on_delete=models.CASCADE)&#xA;</code></pre></div>This approach to the hierarchy problem in Django looks good at first glance. The resulting structure will look something like this:&#xA;<a hx-boost="false" href="https://res.cloudinary.com/dwrscezd2/image/upload/v1745886724/coffee-bytes/category-subcategory_mtkzpy.png">&#xA;<figure class="md-local-image"><img src="https://res.cloudinary.com/dwrscezd2/image/upload/v1745886724/coffee-bytes/category-subcategory_mtkzpy.png"&#xA; loading="lazy"&#xA; alt="Subcategory model schema with ForeignKey made Category in Django">&#xA;</figure>&#xA;</a>&#xA;<h3 id="the-problem-of-using-a-model-by-category">The problem of using a model by category</h3>&#xA;This scheme will work in situations where the hierarchies do not nest very deep, but what if those subcategories have subcategories?&#xA;Imagine a horror movie category, with a ghost subcategory that, in turn, has a home ghost subcategory and this, in turn, a subcategory of type of ending.&#xA;Well, we add a SubSubCategory class, don&rsquo;t we? But&hellip; what if those SubSubCategories have in turn subcategories. See what I&rsquo;m trying to get at?&#xA;<a hx-boost="false" href="https://res.cloudinary.com/dwrscezd2/image/upload/v1745886985/coffee-bytes/category-subcategory-problem_bx9ymd.webp">&#xA;<figure class="md-local-image"><img src="https://res.cloudinary.com/dwrscezd2/image/upload/v1745886985/coffee-bytes/category-subcategory-problem_bx9ymd.webp"&#xA; loading="lazy"&#xA; alt="Infinite subcategories problem scheme">&#xA;</figure>&#xA;</a>&#xA;Every time you need to create a new subcategory you will have to create a new model in the models.py file of your application. And not only that, but a new table that probably only has a few records. Is there a better approach to the problem? The &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/django/why-should-you-use-django-framework/">versatile Django Framework&rsquo;s ORM</a>&#xA; offers a pretty clean solution.&#xA;&#xA;<h2 id="foreignkey-to-the-same-model-in-django">ForeignKey to the same model in Django</h2>&#xA;To simplify the problem of categories in Django, we create a single model, with a property of type ForeignKey or foreign key pointing to the same object; that is, to self.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"># app/models.py&#xA;from django.db import models&#xA;&#xA;class Category(models.Model):&#xA; name = models.CharField(max_length=256)&#xA; parent = models.ForeignKey(&#xA; &#34;self&#34;,&#xA; related_name=&#34;subcategories&#34;,&#xA; on_delete=models.CASCADE,&#xA; blank=True,&#xA; null=True,&#xA; )&#xA;</code></pre></div>In this way we will have a structure similar to a graph, where each node points to another.&#xA;<a hx-boost="false" href="https://res.cloudinary.com/dwrscezd2/image/upload/v1745887246/coffee-bytes/foreign-key-to-self-django_uvvogs.webp">&#xA;<figure class="md-local-image"><img src="https://res.cloudinary.com/dwrscezd2/image/upload/v1745887246/coffee-bytes/foreign-key-to-self-django_uvvogs.webp"&#xA; loading="lazy"&#xA; alt="Schematic diagram of how ForeignKey made self(the same model) works in Django">&#xA;</figure>&#xA;</a>&#xA;This new arrangement allows us to create as many subcategories as we want, without the need to create new models. To do so, we simply assign the parent property to the Category class instance we want it to belong to.&#xA;<h3 id="accessing-the-same-model-in-django">Accessing the same model in Django</h3>&#xA;See how the ForeignKey does self in practice:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">from my_app.models import Category&#xA;parent = Category.objects.create(name=&#34;Programming languages&#34;)&#xA;subcategory = Category.objects.create(name=&#34;Python&#34;)&#xA;subcategory.parent = parent&#xA;# save it in the database&#xA;subcategory.save()&#xA;</code></pre></div>I explain what happens. First we create a main category, and a subcategory. Later, we assign the property parent, of this last one, to the main category. And ready, we save.&#xA;Finally, let&rsquo;s create a sub-subcategory for our subcategory&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">subsubcategory = Category.objects.create(name=&#34;Django&#34;)&#xA;subsubcategory.parent = subcategory&#xA;subsubcategory.save()&#xA;</code></pre></div>Notice how we have created another sub-subcategory (called Django), which comes from the same model we used in the other two. To assign it to a category, we match its parent property to the subcategory we created (named Python).&#xA;As you have already seen, a single model allows us to add as many subcategories as we want.&#xA;<h3 id="access-to-subcategories">Access to subcategories</h3>&#xA;If we examine the parent category, which we created previously, we will notice that it has a list of subcategories, among which our subcategory (Call Python) is already included.&#xA;We can access it as we would any other many-to-one relationship.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">parent.subcategories.all()&#xA;&lt;QuerySet [&lt;Category: Category object (2)&gt;]&gt;&#xA;parent.subcategories.all()[0].name&#xA;&#39;Python&#39;&#xA;</code></pre></div><h3 id="go-from-subcategories-to-categories">Go from subcategories to categories</h3>&#xA;On the other hand, if we want to go &ldquo;in reverse&rdquo; from the most nested category to the least nested, we simply go through it; we access the parent, and then the parent of the next instance, and so on as many nestings as we need.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">subcategory.name&#xA;&#39;Django&#39;&#xA;subcategory.parent.name&#xA;&#39;Python&#39;&#xA;subcategory.parent.parent.name&#xA;&#39;Programming languages&#39;&#xA;</code></pre></div><h2 id="other-resources">Other resources</h2>&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://docs.djangoproject.com/en/4.0/ref/models/fields/" rel="noopener" target="_blank">Official documentation on the ForeignKey to self</a>&#xA;</li>&#xA;</ul>&#xA;

Grouping by categories is quite common in web applications, from movies, courses or any other resource that presents a hierarchical relationship to another object. In Django there are different ways to model these relationships. Probably, the first that will come to your mind will be to create a category object, and then relate it by means of a ForeignKey with a subcategory.&#xA;<h2 id="one-subcategory-or-level-per-model">One subcategory or level per model</h2>&#xA;What I meant by one category or level per model is something like this:

Categories in Django using ForeignKey to self

Let&rsquo;s do a Debounce vs Throttle. Debounce and throttle are &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/python/design-patterns-in-software/">design patterns</a>&#xA; used to limit the execution of functions, generally they are used to restrict the amount of times an event is fired: click, scroll, resize or other events. This patterns are not exclusive to Javascript and can be used in any language or can be used as high level abstractions; in a previous post I explained how to use throttle to &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/software-architecture/throttling-on-nginx/">limit the number of requests received by the nginx server</a>&#xA;.

Debounce and Throttle Interactive Explanation

In this post I explain three methods to extend or customize Django&rsquo;s User model, without having to rewrite it from scratch, and keeping all &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/django/why-should-you-use-django-framework/">Django&rsquo;s user management features</a>&#xA;&#xA;But, before we start, let&rsquo;s see where Django&rsquo;s User model comes from.&#xA;<h2 id="where-does-the-django-user-model-come-from">Where does the Django User model come from?</h2>&#xA;Django&rsquo;s User model inherits from AbstractUser which, in turn, inherits from the AbstractBaseUser class.&#xA;<div class="mermaid">graph TD;&#xA; AbstractBaseUser-->AbstractUser;&#xA; AbstractUser-->User;&#xA;</div>&#xA; &#xA; &#xA; If you look at the Django source code, you will see that the User model you normally use has virtually no functionality of its own, but inherits all of its functionality from AbstractUser.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/django/how-to-customize-the-user-model-in-django/images/UserDjango.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/django/how-to-customize-the-user-model-in-django/images/UserDjango.png"&#xA; loading="lazy"&#xA; alt="Screenshot of Django User model code"><figcaption>&#xA; Screenshot of Django version 4.0 code&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;Now that we know the above, we can use the AbstractUser and AbstractBaseUser classes to create our custom User models.&#xA;&#xA;<h2 id="extend-django-user-model-inheriting-from-djangos-abstractuser-subclass">Extend Django user model inheriting from Django&rsquo;s AbstractUser subclass</h2>&#xA;This method is probably the most popular method for extending Django&rsquo;s User model. This is because it retains almost all the functionality of the original User model.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"># users/models.py&#xA;from django.contrib.auth.models import AbstractUser&#xA;from django.db import models&#xA;&#xA;class CustomUser(AbstractUser):&#xA; # Other properties defined by you&#xA; credits = models.PositiveIntegerField(verbose_name=&#39;credits&#39;,&#xA; default=0, &#xA; blank=True)&#xA;</code></pre></div>After creating a new class that inherits from AbstractUser, we need to tell Django that we want to use this new model instead of the default user model.&#xA;We set this behavior in our configuration file.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"># settings.py&#xA;AUTH_USER_MODEL = &#39;users.CustomUser&#39;&#xA;</code></pre></div><h3 id="extend-django-user-model-using-the-custom-model-in-djangos-account-views">Extend Django user model using the custom model in Django&rsquo;s account views</h3>&#xA;If we want to use the Django template system to automatically generate a registration form, we will need to tell Django to use the new user model, for this we inherit a new form from the UserCreationForm class, and pass it our custom model, which we can with the get_user_model method.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">from django.contrib.auth import get_user_model&#xA;from django.contrib.auth.forms import UserCreationForm&#xA;&#xA;User = get_user_model()&#xA;&#xA;class RegisterFormForCustomUser(UserCreationForm):&#xA; class Meta:&#xA; model = User&#xA; fields = [&#39;username&#39;, &#39;email&#39;]&#xA;</code></pre></div>And that&rsquo;s it, we can use it exactly as if we were using the User model included in Django.&#xA;<h3 id="django-admin-does-not-hashed-passwords">django admin does not hashed passwords</h3>&#xA;When we use a custom user model, we need to tell Django to handle passwords with the default user functionality&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">from django.contrib import admin&#xA;from django.contrib.auth.admin import UserAdmin&#xA;from django.contrib.auth import get_user_model&#xA;&#xA;user = get_user_model()&#xA;&#xA;class CustomUserAdmin(UserAdmin):&#xA; pass&#xA;&#xA;admin.site.register(user, CustomUserAdmin)&#xA;</code></pre></div>Now the admin panel will behave exactly as it would with the default Django user.&#xA;<h3 id="what-does-abstractuser-look-like-internally">What does AbstractUser look like internally?</h3>&#xA;Notice how the AbstractUser class inherits from _AbstractBaseUse_r and has multiple fields available to profile a user. Also, it cannot be instantiated directly, as it is an abstract class.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/django/how-to-customize-the-user-model-in-django/images/AbstractUserDjango-1.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/django/how-to-customize-the-user-model-in-django/images/AbstractUserDjango-1.png"&#xA; loading="lazy"&#xA; alt="Django AbstractUser class source code"><figcaption>&#xA; Screenshot of Django version 4.0 AbstractUser code&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;Let&rsquo;s move on to the second method.&#xA;<h3 id="how-to-change-the-user-field-for-authentication-in-django">How to change the user field for authentication in Django?</h3>&#xA;If you look at the code above, there is an uppercase property called USERNAME_FIELD, there you can specify another field to work as the user.&#xA;As you don&rsquo;t want two users to identify themselves in the same way that field has to be marked as unique. Besides that you have to modify the object manager, the code is a bit long so I won&rsquo;t put it here&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">class CustomUser(AbstractUser):&#xA; custom_id = models.CharField(max_length=40, unique=True)&#xA; # ...&#xA; USERNAME_FIELD = &#39;custom_id&#39;&#xA;</code></pre></div><h2 id="extend-django-user-model-inheriting-from-subclass-abstractbaseuser">Extend Django user model inheriting from subclass AbstractBaseUser</h2>&#xA;This class, as you can see in the previous image, is the base class used to create the AbstractUser. Its operation is the minimum and it only has 3 fields:&#xA;<ul>&#xA;<li>password</li>&#xA;<li>last_login</li>&#xA;<li>is_active</li>&#xA;</ul>&#xA;It only has the authentication function. And you have to indicate which field will be used as username, to authenticate the user.&#xA;This method is usually used to fully customize the User model or when we need almost no extra fields.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"># users/models.py&#xA;from django.contrib.auth.base_user import AbstractBaseUser&#xA;from django.db import models&#xA;&#xA;class CustomUser(AbstractBaseUser):&#xA; email = models.EmailField(verbose_name=&#39;emails&#39;, unique=True, max_length=255)&#xA; credits = models.PositiveIntegerField(verbose_name=&#39;credits&#39;,&#xA; default=0, &#xA; blank=True)&#xA; USERNAME_FIELD=&#39;email&#39;&#xA; REQUIRED_FIELDS = []&#xA;</code></pre></div>Remember to tell Django to use your custom model instead of the default one.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"># settings.py&#xA;AUTH_USER_MODEL = &#39;users.CustomUser&#39;&#xA;</code></pre></div><h3 id="what-does-abstractbaseuser-look-like-internally">What does AbstractBaseUser look like internally?</h3>&#xA;The following image is a direct screenshot of the Django code in version 4.0&#xA;As you can see, it only has the 3 fields mentioned, it inherits directly from models.Model and its Meta class tells Python that it is an abstract model; you cannot create instances directly from it.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/django/how-to-customize-the-user-model-in-django/images/AbstractBaseUserDjango-1.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/django/how-to-customize-the-user-model-in-django/images/AbstractBaseUserDjango-1.png"&#xA; loading="lazy"&#xA; alt="Django AbstractBaseUser class source code"><figcaption>&#xA; Screenshot of Django AbstractBaseUser version 4.0&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;Now let&rsquo;s look at the third way to extend Django&rsquo;s User model.&#xA;<h2 id="extend-django-user-model-creating-a-profile-for-model-user">Extend Django user model creating a profile for model User</h2>&#xA;Another way to extend the user model is to create another model that serves as a container for the extra fields and then relate it by an OneToOneField field to the model that the Django configuration receives by default.&#xA;This approach is ideal if we are the creator of a package that needs to customize the User model of the project to work, but without modifying it directly.&#xA;It is also useful when we need several types of users or different profiles, with different fields between them.&#xA;To create a profile in this way it is enough to declare a field that relates our new model to the User model, by means of an OneToOneField.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">from django.conf import settings&#xA;class Profile(models.Model):&#xA; # other fields&#xA; user = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)&#xA;</code></pre></div>And to access our user, we access the field that relates it to the model we created.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">user = User.objects.get(username=&#39;user&#39;)&#xA;user.Profile&#xA;</code></pre></div><h2 id="other-resources">Other resources</h2>&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://github.com/django/django/tree/main/django/contrib/auth" rel="noopener" target="_blank">Original source code of the Django User model</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://docs.djangoproject.com/en/4.0/topics/auth/customizing/" rel="noopener" target="_blank">Django User model documentation</a>&#xA;</li>&#xA;</ul>&#xA;

In this post I explain three methods to extend or customize Django&rsquo;s User model, without having to rewrite it from scratch, and keeping all &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/django/why-should-you-use-django-framework/">Django&rsquo;s user management features</a>&#xA;&#xA;But, before we start, let&rsquo;s see where Django&rsquo;s User model comes from.&#xA;<h2 id="where-does-the-django-user-model-come-from">Where does the Django User model come from?</h2>&#xA;Django&rsquo;s User model inherits from AbstractUser which, in turn, inherits from the AbstractBaseUser class.&#xA;<div class="mermaid">graph TD;&#xA; AbstractBaseUser-->AbstractUser;&#xA; AbstractUser-->User;&#xA;</div>&#xA; &#xA; &#xA; If you look at the Django source code, you will see that the User model you normally use has virtually no functionality of its own, but inherits all of its functionality from AbstractUser.

How to customize the User model in Django?

Django&rsquo;s select_related and prefetch_related methods are used to reduce the number of queries made to the database. This translates into response time for each view. In addition, using these methods is one of the &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/software-architecture/how-to-scale-a-django-app-to-serve-one-million-users/">actions to implement to improve the performance of a Django application</a>&#xA;&#xA;Just consider that there are more &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/opinion/dont-obsess-about-your-web-application-performance/">important things to optimize other than your app&rsquo;s performance</a>&#xA;, but if you insist, dive into annotate and aggregate, and be careful with the nested subqueries of annotate because &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/django/fix-slow-queries-in-django-when-using-annotate-and-subqueries/">they can make your django queries go really slow</a>&#xA;&#xA;<h2 id="differences-between-django-select_related-and-prefetch_related-summarized">Differences between django select_related and prefetch_related summarized</h2>&#xA;<table>&#xA; <thead>&#xA; <tr>&#xA; <th></th>&#xA; <th>select_related</th>&#xA; <th>prefetch_related</th>&#xA; </tr>&#xA; </thead>&#xA; <tbody>&#xA; <tr>&#xA; <td>Relationship</td>&#xA; <td>Foreign key or One to One</td>&#xA; <td>Many to Many</td>&#xA; </tr>&#xA; <tr>&#xA; <td>Number of queries</td>&#xA; <td>1</td>&#xA; <td>2</td>&#xA; </tr>&#xA; <tr>&#xA; <td>Union of objects</td>&#xA; <td>Directly by SQL</td>&#xA; <td>Using Python</td>&#xA; </tr>&#xA; </tbody>&#xA;</table>&#xA;&#xA;<h2 id="django-select_related">django select_related</h2>&#xA;The select_related method is used to follow a relationship of type ForeignKey or OneToOneField to the respective objects it points to and obtain them..&#xA;When using select_related we will have a longer query, however, the advantage is that it will no longer be necessary to access the database again to obtain the objects of the related model.&#xA;<a hx-boost="false" href="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o3xcx7ifog76559fy4lo.png">&#xA;<figure class="md-local-image"><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o3xcx7ifog76559fy4lo.png"&#xA; loading="lazy"&#xA; alt="Simplified diagram of how Django select_related works">&#xA;</figure>&#xA;</a>&#xA;Simplified diagram of how select_related works&#xA;Consider this example:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">from django.db import models&#xA;&#xA;class Main(models.Model):&#xA; name = models.CharField(max_length=256)&#xA;&#xA;class Derivative(models.Model):&#xA; name = models.CharField(max_length=256)&#xA; main = models.ForeignKey(&#xA; &#34;Main&#34;, related_name=&#34;derivatives&#34;, on_delete=models.CASCADE&#xA; )&#xA;</code></pre></div>If we try to access the object pointed to by the Foreign Key relationship, a new database query will be generated. select_related avoids that extra query for each object.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-html" data-lang="html">{% for object in queryset %}&#xA; &lt;p&gt;{{object.name}}&lt;/p&gt;&#xA; &lt;small&gt;{{object.main.name}}&lt;/small&gt;&#xA;{% endfor %}&#xA;</code></pre></div>For example, if we have three Derived objects related to a single main object:&#xA;<ul>&#xA;<li>A main query that retrieves all objects Derivative</li>&#xA;<li>Three queries, exactly the same, one for each time we access the main object from the Derived object.</li>&#xA;</ul>&#xA;<h3 id="use-select_related-in-a-query">Use select_related in a query</h3>&#xA;To use select_related we call it from our query, passing it the name of the field that corresponds to our relationship with the other model.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">Derivative.objects.select_related(&#34;main&#34;)&#xA;</code></pre></div><h3 id="internal-operation-of-select_related">Internal operation of select_related</h3>&#xA;How select_related works internally, select_related replaces multiple queries being performed by a single INNER JOIN at the database level:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql">SELECT my_app_derivative.id,&#xA; my_app_derivative.name,&#xA; my_app_derivative.main_id&#xA; FROM my_app_derivative&#xA;&#xA;SELECT my_app_main.id,&#xA; my_app_main.name&#xA; FROM my_app_main&#xA; WHERE my_app_main.id = &#39;1&#39;&#xA;&#xA;SELECT my_app_main.id,&#xA; my_app_main.name&#xA; FROM my_app_main&#xA; WHERE my_app_main.id = &#39;1&#39;&#xA;&#xA;SELECT my_app_main.id,&#xA; my_app_main.name&#xA; FROM my_app_main&#xA; WHERE my_app_main.id = &#39;1&#39;&#xA;</code></pre></div>This reduces multiple SQL queries to a single, longer query.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql">SELECT my_app_derivative.id,&#xA; my_app_derivative.name,&#xA; my_app_derivative.main_id,&#xA; my_app_main.id,&#xA; my_app_main.name&#xA; FROM my_app_derivative&#xA; INNER JOIN my_app_main&#xA; ON (my_app_derivative.main_id = my_app_main.id)&#xA;</code></pre></div><h2 id="django-prefetch_related">django prefetch_related</h2>&#xA;If the select_related method retrieves a single object from a single relationship field, the prefetch_related method is used when we have a multiple relationship with another model, i.e. a relationship of type ManyToMany or a reverse ForeignKey.&#xA;<a hx-boost="false" href="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jgyuop69qpmie1b0cgzj.png#?">&#xA;<figure class="md-local-image"><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jgyuop69qpmie1b0cgzj.png#?"&#xA; loading="lazy"&#xA; alt="Simplified diagram of how Django prefetch_related works">&#xA;</figure>&#xA;</a>&#xA;Simplified diagram of how prefetch_related works&#xA;Consider this example, note the ManyToManyField field towards the Main model.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">from django.db import models&#xA;&#xA;class Main(models.Model):&#xA; name = models.CharField(max_length=256)&#xA;&#xA;class ManyToManyModel(models.Model):&#xA; name = models.CharField(max_length=256)&#xA; ManyToManyRel = models.ManyToManyField(&#34;Main&#34;, related_name=&#34;multiples&#34;)&#xA;</code></pre></div>If we access the field that represents the multiple relation of our object, without using prefetch_related, we will be impacting the database with a new query.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-html" data-lang="html">{% for object in queryset %}&#xA; &lt;p&gt;{{object.name}}&lt;/p&gt;&#xA; {% for main in object.ManyToManyRel.all %}&#xA; &lt;!-- New query each iteration --&gt;&#xA; &lt;p&gt;&lt;small&gt;{{main.name}}&lt;/small&gt;&lt;/p&gt;&#xA; {% endfor %}&#xA;{% endfor %}&#xA;</code></pre></div><h3 id="use-prefetch_related-a-query">Use prefetch_related a query</h3>&#xA;To use the prefetch_related method call it at the end of our query, choosing the field that represents the many-to-many relationship in our object.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python">queryset = ManyToManyModel.objects.prefetch_related(&#34;ManyToManyRel&#34;)&#xA;</code></pre></div><h3 id="inner-workings-of-prefetch_related">Inner workings of prefetch_related</h3>&#xA;How does prefecth_related work internally? The prefetch_related method replaces the multiple SQL queries by only 2 SQL queries: one for the main query and the other for the related objects, then it will join the data using Python.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql">SELECT my_app_main.id,&#xA; my_app_main.name&#xA; FROM my_app_main&#xA; INNER JOIN my_app_manytomanyrel_main&#xA; ON (my_app_main.id = my_app_manytomanyrel_main.main_id)&#xA; WHERE my_app_manytomanyrel_main.manytomanyrel_id = &#39;1&#39;&#xA;&#xA;...&#xA;</code></pre></div>The multiple queries above are reduced to only 2 SQL queries.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql">SELECT my_app_manytomanyrel.id,&#xA; my_app_manytomanyrel.name&#xA; FROM my_app_manytomanyrel&#xA;&#xA;SELECT (my_app_manytomanyrel_main.manytomanyrel_id) AS *prefetch_related*val_manytomanyrel_id,&#xA; my_app_main.id,&#xA; my_app_main.name&#xA; FROM my_app_main&#xA; INNER JOIN my_app_manytomanyrel_main&#xA; ON (my_app_main.id = my_app_manytomanyrel_main.main_id)&#xA; WHERE my_app_manytomanyrel_main.manytomanyrel_id IN (&#39;1&#39;, &#39;2&#39;, &#39;3&#39;, &#39;4&#39;)&#xA;</code></pre></div><h2 id="other-related-resources">Other related resources</h2>&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://stackoverflow.com/questions/31237042/whats-the-difference-between-select-related-and-prefetch-related-in-django-orm" rel="noopener" target="_blank">¿What&rsquo;s the difference between select_related and prefetch_related in django IRM?</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://buildatscale.tech/select_related-vs-prefetch_related/" rel="noopener" target="_blank">Select related vs Prefecth related</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://docs.djangoproject.com/en/dev/ref/models/querysets/" rel="noopener" target="_blank">API Queryset reference</a>&#xA;</li>&#xA;</ul>&#xA;

Django&rsquo;s select_related and prefetch_related methods are used to reduce the number of queries made to the database. This translates into response time for each view. In addition, using these methods is one of the &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/software-architecture/how-to-scale-a-django-app-to-serve-one-million-users/">actions to implement to improve the performance of a Django application</a>&#xA;&#xA;Just consider that there are more &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/opinion/dont-obsess-about-your-web-application-performance/">important things to optimize other than your app&rsquo;s performance</a>&#xA;, but if you insist, dive into annotate and aggregate, and be careful with the nested subqueries of annotate because &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/django/fix-slow-queries-in-django-when-using-annotate-and-subqueries/">they can make your django queries go really slow</a>&#xA;

Differences between Django select_related and prefetch_related

When we use Typescript with React and we want to &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/react/what-types-to-use-for-react-components-with-children/">pass a children as prop to one of our components</a>&#xA;, we need to indicate the type. Generally we use the type React.FC, which is short for React.FunctionComponent. With this the Typescript message warning us of a children with type any will disappear.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">const Component: React.FC = ({ children }) =&gt; {&#xA; return (&lt;div&gt;{children}&lt;/div&gt;)&#xA;}&#xA;</code></pre></div>In addition to allowing us to work with children, React.FC also causes an error if we try to return undefined from our component.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">const Component: React.FC = ({ children }) =&gt; {&#xA; // Type &#39;() =&gt; undefined&#39; can&#39;t be assigned to type &#39;FC&lt;{}&gt;&#39;.&#xA; return undefined&#xA;}&#xA;&#xA;export default Component&#xA;</code></pre></div>As you can see, React.FC is quite comfortable to use, but some people do not agree with its use.&#xA;<h2 id="reactfc-more-disadvantages-than-advantages">React.FC, more disadvantages than advantages?</h2>&#xA;What&rsquo;s the problem? Well, some developers claim that React.FC can bring more disadvantages than advantages, there is even a discussion on github (link at the end), in which it is debated whether it is convenient to remove one of the examples in the documentation that uses it.&#xA;The user who started this discussion believes that the fact that React.FC is so popular is because its presence in the documentation positions it as the default way to handle React components with Typescript.&#xA;Here are some of the reasons why React.FC may have more disadvantages than advantages.&#xA;<h3 id="reactfc-does-not-warn-us-about-unused-children">React.FC does not warn us about unused children</h3>&#xA;React.FC is not always the most explicit way to tell typescript that a component receives children as part of its props.&#xA;Imagine that we pass a children to the component, but we don&rsquo;t use it.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">import Component from &#39;./Component&#39;;&#xA;&#xA;function App() {&#xA; return (&#xA; &lt;Component&gt;I&#39;m the children component&lt;/Component&gt;&#xA; );&#xA;}&#xA;&#xA;export default App;&#xA;</code></pre></div>Our component does receive the children as a prop, but the React.FC type appeases Typescript and prevents it from returning an error, even if we are not using it.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">const Component: React.FC = () =&gt; {&#xA; return (&#xA; &lt;div&gt;I&#39;m not the children which receives the component&lt;/div&gt;&#xA; )&#xA;}&#xA;&#xA;export default Component&#xA;</code></pre></div>Or, for an opposite case; imagine that we do not want our component to receive a children as one of its props, however, as we are using React.FC we will not get any error.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-javascript" data-lang="javascript">import Component from &#39;./Component&#39;;&#xA;&#xA;function App() {&#xA; return (&#xA; &lt;Component&gt;&lt;div&gt;I shouldn&#39;t be here&lt;/div&gt;&lt;/Component&gt;&#xA; );&#xA;}&#xA;&#xA;export default App;&#xA;</code></pre></div><h3 id="subcomponent-pattern-in-react-gets-more-complicated-using-reactfc">Subcomponent pattern in React gets more complicated using React.FC</h3>&#xA;The &#xA;&#xA;<a class="markdown-link" href="https://medium.com/@kunukn_95852/react-components-with-namespace-f3d169feaf91" rel="noopener" target="_blank">component pattern as namespace</a>&#xA; is fairly simple to create without using React.FC, but with React.FC it can get quite complicated.&#xA;Don&rsquo;t know what it is? Think of a pattern that allows you to group components within a certain parent, which functions as a namespace for our child components; similar to how the C++ namespace std would work.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">&lt;Namespace&gt;&#xA; &lt;Namespace.Component /&gt;&#xA;&lt;/Namespace&gt;&#xA;</code></pre></div>In its simplest form, omitting React.FC would look something like this:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">const Namespace = (props: NamespaceProps) =&gt; {/* ... */}&#xA;Namespace.Component = (props: ComponentProps) =&gt; { /*...*/ }&#xA;</code></pre></div>But if we choose to use React.FC the code would become more complicated and the readability would decrease a bit.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">const Namespace: React.FC&lt;NamespaceProps&gt; &amp; { Component: React.FC&lt;ComponentProps&gt; } = (props) =&gt; {/* ... */ }&#xA;Namespace.Component = (props) =&gt; { /*...*/ }&#xA;</code></pre></div>&#xA;<h2 id="what-can-we-use-instead-of-reactfc">What can we use instead of React.FC?</h2>&#xA;The core of the criticism is that React.FC adds the children implicitly, going against the explicit nature of typescript.&#xA;Just like using &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-programming-language-introduction-to-variables-and-data-types/">the Go programming language</a>&#xA; for its compiler.&#xA;<a hx-boost="false" href="https://res.cloudinary.com/dwrscezd2/image/upload/v1741885083/rust-meme-typescript_xa6ajl.webp">&#xA;<figure class="md-local-image"><img src="https://res.cloudinary.com/dwrscezd2/image/upload/v1741885083/rust-meme-typescript_xa6ajl.webp"&#xA; loading="lazy"&#xA; alt="Typescript decided to use Go for its compiler instead of Rust, making some Rust developers furious"><figcaption>&#xA; Typescript decided to use Go for its compiler instead of Rust, making some Rust developers furious&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;However, the discussion I am talking about is just that, a discussion, it is not written in stone, you may consider that it is better to sacrifice a bit of readability in favor of convenience or, on the contrary, you may believe that it is important to be explicit when working with typescript.&#xA;If this is the case, consider that you can always declare the children as a prop explicitly, just as you would any other prop. And in the same way, you can declare the return value of your component as a JSX type element.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">interface propsWithChildren {&#xA; children: React.ReactNode&#xA;}&#xA;&#xA;const Component = ({ children }: propsWithChildren): JSX.Element =&gt; {&#xA; return (&#xA; &lt;div&gt;{children}&lt;/div&gt;&#xA; )&#xA;}&#xA;&#xA;export default Component&#xA;</code></pre></div>Typescript should not warn you about any errors with this approach to the problem.&#xA;<h2 id="other-resources-on-react-fc-usage">Other resources on React FC usage</h2>&#xA;Below are links to the original discussion and some posts on this topic.&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://github.com/facebook/create-react-app/pull/8177" rel="noopener" target="_blank">Eliminate React.FC from typescript template</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://medium.com/raccoons-group/why-you-probably-shouldnt-use-react-fc-to-type-your-react-components-37ca1243dd13" rel="noopener" target="_blank">Why you probably shouldn&rsquo;t use react fc to type your react components</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://www.harrymt.com/blog/2020/05/20/react-typescript-react-fc.html" rel="noopener" target="_blank">React Typescript&rsquo; React.FC</a>&#xA;</li>&#xA;</ul>&#xA;

When we use Typescript with React and we want to &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/react/what-types-to-use-for-react-components-with-children/">pass a children as prop to one of our components</a>&#xA;, we need to indicate the type. Generally we use the type React.FC, which is short for React.FunctionComponent. With this the Typescript message warning us of a children with type any will disappear.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-jsx" data-lang="jsx">const Component: React.FC = ({ children }) =&gt; {&#xA; return (&lt;div&gt;{children}&lt;/div&gt;)&#xA;}&#xA;</code></pre></div>In addition to allowing us to work with children, React.FC also causes an error if we try to return undefined from our component.

Why using React.FC could be a bad practice?

In addition to &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/">unit test testing and coverage measurement in go</a>&#xA;, this programming language is capable of profiling the efficiency of the code by analyzing it in a very detailed way. This is quite useful to find bottlenecks or very expensive parts of the code, which are called numerous times or whose performance can be improved.&#xA;<h2 id="how-does-go-profiling-work-internally-in-gnulinux">How does Go profiling work internally in GNU/Linux?</h2>&#xA;GNU/Linux, more specifically GNU, has an alarm signal called &#xA;&#xA;<a class="markdown-link" href="https://www.gnu.org/software/libc/manual/html_node/Alarm-Signals.html" rel="noopener" target="_blank">SIGPROF</a>&#xA;, this signal warns when a timer finishes measuring CPU usage and interrupts code execution.&#xA;In Go profiling, the SIGPROF signal is programmed to be called every 10 ms. Each time this signal is invoked, the current instruction of the &#xA;&#xA;<a class="markdown-link" href="https://es.wikipedia.org/wiki/Contador_de_programa" rel="noopener" target="_blank">program counter (PC)</a>&#xA; is examined and traced back to the sequence of instructions that called it. The result of the above process is a report of the elements in the execution stack, known as &#xA;&#xA;<a class="markdown-link" href="https://es.wikipedia.org/wiki/Stack_trace" rel="noopener" target="_blank">stack trace</a>&#xA;.&#xA;The profiling process will slow down the execution of the code, as it is interrupted every 10ms to see what is being executed. As you probably already deduced, if a function is found multiple times in the stack traces that are generated after each SIGPROF signal, it means that it has been running for a long time.&#xA;At the end of the profiler the go pprof tool organizes the data so that it can be represented in a more user-friendly way.&#xA;&#xA;<h2 id="cpu-profiling-in-go">CPU profiling in go</h2>&#xA;For this entry I will use the classic fibonacci recursion to demonstrate the profiling capabilities of go. I am using go version go1.15.15 linux/amd64.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func Fibonacci(n int) int {&#xA; if n &lt;= 1 {&#xA; &#x9;return n&#xA; }&#xA; return (Fibonacci(n-1) + Fibonacci(n-2))&#xA;}&#xA;</code></pre></div>For testing in go, just as I explained in the previous post, we will use an array of structs to handle the different cases.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func TestFibonacci(t *testing.T) {&#xA; tables := []struct {&#xA; &#x9;n int&#xA; &#x9;fibo int&#xA; }{&#xA; &#x9;{0, 0},&#xA; &#x9;{1, 1},&#xA; &#x9;{2, 1},&#xA; &#x9;{15, 610},&#xA; &#x9;{17, 1597},&#xA; &#x9;{40, 102334155},&#xA; }&#xA;&#xA; for _, table := range tables {&#xA; &#x9;result := Fibonacci(table.n)&#xA; &#x9;if result != table.fibo {&#xA; &#x9;&#x9;t.Errorf(&#34;Fibonacci incorrecta, esperabamos %d, pero obtubimos %d&#34;, table.fibo, result)&#xA; &#x9;}&#xA; }&#xA;}&#xA;</code></pre></div><h3 id="obtaining-profiling-information">Obtaining profiling information</h3>&#xA;As we did to view the &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/">coverage in go testing</a>&#xA;, we use a flag to create a file with the profiling information in binary, this file cannot be viewed directly.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">go test -cpuprofile=cpu.out&#xA;&#xA;PASS&#xA;ok _/home/eduardo/Programacion/goTesting/testing 0.813s&#xA;</code></pre></div>But it can be used by other tools to visualize it in a more human way.&#xA;<h3 id="display-profiling-results">Display profiling results</h3>&#xA;To see the cpu usage summary we use tool pprof passing as argument the file containing the profiling data.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">go tool pprof cpu.out&#xA;&#xA;File: testing.test&#xA;Type: cpu&#xA;Time: Feb 10, 2022 at 1:06pm (CST)&#xA;Duration: 802.18ms, Total samples = 690ms (86.02%)&#xA;Entering interactive mode (type &#34;help&#34; for commands, &#34;o&#34; for options)&#xA;</code></pre></div>After executing the above command we will be inside a terminal. If we execute top we will see the behavior of our code.&#xA;See how Fibonacci occupies almost all of the time used.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">(pprof) top&#xA;&#xA;File: testing.test&#xA;Type: cpu&#xA;Time: Feb 10, 2022 at 1:06pm (CST)&#xA;Duration: 802.18ms, Total samples = 690ms (86.02%)&#xA;Entering interactive mode (type &#34;help&#34; for commands, &#34;o&#34; for options)&#xA;(pprof) top&#xA;Showing nodes accounting for 690ms, 100% of 690ms total&#xA; flat flat% sum% cum cum%&#xA; 680ms 98.55% 98.55% 680ms 98.55% _/home/eduardo/Programacion/goTesting/testing.Fibonacci&#xA; 10ms 1.45% 100% 10ms 1.45% runtime.epollwait&#xA; 0 0% 100% 680ms 98.55% _/home/eduardo/Programacion/goTesting/testing.TestFibonacci&#xA; 0 0% 100% 10ms 1.45% runtime.findrunnable&#xA; 0 0% 100% 10ms 1.45% runtime.mcall&#xA; 0 0% 100% 10ms 1.45% runtime.netpoll&#xA; 0 0% 100% 10ms 1.45% runtime.park_m&#xA; 0 0% 100% 10ms 1.45% runtime.schedule&#xA; 0 0% 100% 680ms 98.55% testing.tRunner&#xA;</code></pre></div>Within the pprof terminal it is possible to inspect the average execution time of each line of a function, using:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">list &lt;nombre_funcion&gt;&#xA;</code></pre></div>After executing the command a list will be generated where we can see each function broken down line by line, along with its impact.&#xA;You have probably noticed that most of the time is consumed by the recursive part of Fibonacci.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-profiling-or-basic-profiling-of-cpu-usage/images/GoListProfile.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-profiling-or-basic-profiling-of-cpu-usage/images/GoListProfile.png"&#xA; loading="lazy"&#xA; alt="Results of the list profiling command in go showing the impact of each line of code.">&#xA;</figure>&#xA;</a>&#xA;<h3 id="export-of-results">Export of results</h3>&#xA;In addition to console results, we can also visualize the results, in a more understandable way using the web command, which creates a small schematic accessible from the browser.&#xA;Each box represents an individual function and the lines indicate the order in which the functions call each other.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-profiling-or-basic-profiling-of-cpu-usage/images/GoWebProfile.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-profiling-or-basic-profiling-of-cpu-usage/images/GoWebProfile.png"&#xA; loading="lazy"&#xA; alt="Result of the profiling web command in go. The fibonacci function is shown in large and red">&#xA;</figure>&#xA;</a>&#xA;As an optional parameter we can pass the name of a function and go will filter the results.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">(pprof) web &lt;funcion&gt;&#xA;</code></pre></div><a hx-boost="false" href="https://coffeebytes.dev/en/go/go-profiling-or-basic-profiling-of-cpu-usage/images/GoWebProfileFunction.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-profiling-or-basic-profiling-of-cpu-usage/images/GoWebProfileFunction.png"&#xA; loading="lazy"&#xA; alt="Web command allows to isolate results by function">&#xA;</figure>&#xA;</a>&#xA;The schematic generated by the web command can be exported to a pdf with the pdf command.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">(pprof) pdf&#xA;</code></pre></div><h2 id="other-profiling-resources">Other profiling resources</h2>&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://go.dev/blog/pprof" rel="noopener" target="_blank">Go&rsquo;s official documentation</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://www.instana.com/blog/go-profiler-internals/" rel="noopener" target="_blank">Go profiler internals</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://www.youtube.com/watch?v=nok0aYiGiYA" rel="noopener" target="_blank">GopherCon 2019: Dave Cheney - Two programs in go, three profiling techniques</a>&#xA;</li>&#xA;</ul>&#xA;

In addition to &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/">unit test testing and coverage measurement in go</a>&#xA;, this programming language is capable of profiling the efficiency of the code by analyzing it in a very detailed way. This is quite useful to find bottlenecks or very expensive parts of the code, which are called numerous times or whose performance can be improved.&#xA;<h2 id="how-does-go-profiling-work-internally-in-gnulinux">How does Go profiling work internally in GNU/Linux?</h2>&#xA;GNU/Linux, more specifically GNU, has an alarm signal called &#xA;&#xA;<a class="markdown-link" href="https://www.gnu.org/software/libc/manual/html_node/Alarm-Signals.html" rel="noopener" target="_blank">SIGPROF</a>&#xA;, this signal warns when a timer finishes measuring CPU usage and interrupts code execution.

Go: profiling or basic profiling of CPU usage

Go already has a testing module in its standard library that is ready for our use, we just need to import it and use it.&#xA;&#xA;&#xA;&#xA; <img width="60" height="60" src="https://res.cloudinary.com/dwrscezd2/image/upload/v1717959563/Go_gopher_favicon_uzxa20.svg" alt="">&#xA; <a target="_blank" rel=”nofollow” href="https://coffeebytes.dev/en/pages/go-programming-language-tutorial/">&#xA; Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it</a>&#xA;&#xA;&#xA;<h2 id="testing-preparation-in-go">Testing preparation in go</h2>&#xA;For the tests to be carried out we need:&#xA;A file ending in _test.go * A file ending in _test.go * Run the command go test.&#xA;<ul>&#xA;<li>Run the go test command</li>&#xA;</ul>&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash">├── go.mod&#xA;├── main.go&#xA;└── testing&#xA; ├── main.go&#xA; └── main_test.go&#xA;&#xA;1 directory, 5 files&#xA;</code></pre></div>Consider that, if you are going to &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-package-import-and-module-management/">assign a name to your package</a>&#xA;, you should never name it testing. Why? If you do, go will confuse its testing package with yours, returning those incorrect results.&#xA;To create the tests, inside the testing/main_test.go file, we need a function that receives as argument our testing package with the destructuring character.&#xA;We will compare the result using an if, or whatever we want and, if the test fails, we will call the Errorf method of the testing module.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import &#34;testing&#34;&#xA;&#xA;func TestDivision(t *testing.T) {&#xA; total := Division(10, 2)&#xA; if total != 5 {&#xA; t.Errorf(&#34;División incorrecta, obtuvimos %d pero se esperaba %d&#34;, total, 5)&#xA; }&#xA;}&#xA;</code></pre></div>It is not necessary that the functions to be tested are inside the testing file, in this case I placed them inside testing/main.go.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;func Division(a int, b int) int {&#xA; return a / b&#xA;}&#xA;</code></pre></div>&#xA;<h2 id="execute-tests">Execute tests</h2>&#xA;To run the tests we need to find ourselves inside the directory where our files ending in _test.go are located and run the go test command. If the test passes we will get the PASS message.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">cd testing/&#xA;go test&#xA;&#xA;PASS&#xA;ok main/testing 0.001s&#xA;</code></pre></div>On the other hand, if the tests fail, the word FAIL will be printed on the screen:&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">--- FAIL: TestDivision (0.00s)&#xA; main_test.go:14: División incorrecta, obtuvimos 12 pero se esperaba 5&#xA;FAIL&#xA;exit status 1&#xA;FAIL main/testing 0.001s&#xA;</code></pre></div><h2 id="case-management-with-tables">Case management with tables</h2>&#xA;In the above example we used one function to test one case, however, if we needed to test multiple cases, we would need a function for each test, quite tedious, right?&#xA;To avoid filling up with functions, developers use an array composed of structs, where each struct represents a case to be tested. You can think of the array of structs as a table, where each row is a case and each column is a data type to be tested.&#xA;In this case, each struct in our array consists of three integers; the first two represent the arguments, while the last one is the result.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">tables := []struct {&#xA; &#x9;x int&#xA; &#x9;y int&#xA; &#x9;r int&#xA; }{&#xA; &#x9;{100, 10, 10}, // 100 / 10 = 10&#xA; &#x9;{200, 20, 10}, // 200 / 20 = 10&#xA; &#x9;{300, 30, 10},&#xA; &#x9;{1000, 100, 10},&#xA; }&#xA;</code></pre></div>I&rsquo;m sure you&rsquo;ve noticed that we&rsquo;re not covering division by zero, but leave it at that for now.&#xA;Now that we have our array of structs, we will iterate over each of its elements using &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-slices-and-arrays-basic-characteristics-and-most-common-uses/">go&rsquo;s range function</a>&#xA;. This way we will cover each case.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">for _, table := range tables {&#xA; &#x9;total := Division(table.x, table.y)&#xA; &#x9;if total != table.r {&#xA; &#x9;&#x9;t.Errorf(&#34;División de %d entre %d incorrecta, obtuvimos: %d, pero el resultado es: %d.&#34;, table.x, table.y, total, table.r)&#xA; &#x9;}&#xA; }&#xA;</code></pre></div>If everything went well, we will pass all the tests.&#xA;<h2 id="coverage">Coverage</h2>&#xA;Coverage is already part of the code in go, so we do not need external libraries. If you don&rsquo;t know what Coverage is, think of it as the percentage of your code that is tested. If all your code goes through the tests you will have a coverage of 100%, if only half of it goes through the tests the coverage will be 50%. Previously I talked about coverage in my entry &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/python/unittest-python-are-python-tests-worthwhile/">unittest in Python</a>&#xA;&#xA;To calculate the coverage, simply add the -cover flag to the go test command.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash">go test -cover&#xA;&#xA;PASS&#xA;coverage: 100.0% of statements&#xA;ok _/home/eduardo/Programacion/goTesting/testing 0.002s&#xA;</code></pre></div>As our function is tiny, we obtain a result of 100%, without breaking down, from coverage&#xA;<h3 id="export-coverage-results">Export coverage results</h3>&#xA;We can send all the raw data from our coverage test to an external file with the -coverprofile flag.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash">go test -coverprofile=coverage.out&#xA;&#xA;mode: set&#xA;/home/eduardo/Programacion/goTesting/testing/main.go:3.33,5.2 1 1&#xA;</code></pre></div>This file, named coverage.out, which was generated, is a file containing raw data and will be needed to visualize the results in a more detailed way.&#xA;<h3 id="viewing-results-with-go-tool">Viewing results with go tool</h3>&#xA;To summarize in a more readable way the information of the file containing our coverage test, we will use the tool command, accompanied by the -func flag, followed by the file name. This will return a broken down coverage result.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash">go tool cover -func=coverage.out&#xA;&#xA;/home/eduardo/Programacion/goTesting/testing/main.go:3: Division 100.0%&#xA;total: (statements) 100.0%&#xA;</code></pre></div>Go also allows us to visualize the coverage in HTML format, with colors, directly in our browser. For this we use the -html option, followed by the file with the coverage data.&#xA;When the command is executed, a browser tab will open and show the tested results in green and the untested results in red.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash">go tool cover -html=coverage.out&#xA;</code></pre></div><a hx-boost="false" href="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/images/Captura-de-pantalla-de-2022-02-09-12-35-57.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/images/Captura-de-pantalla-de-2022-02-09-12-35-57.png"&#xA; loading="lazy"&#xA; alt="Coverage en go">&#xA;</figure>&#xA;</a>&#xA;Full html coverage in go&#xA;If we decide to modify our function to handle the division by zero cases, and run the coverage tests again, we will get a different scheme than before. Now a section of code not covered by the tests appears in red and our coverage dropped to 50%.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/images/coverage-en-go.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-basic-testing-and-coverage/images/coverage-en-go.png"&#xA; loading="lazy"&#xA; alt="Screenshot of coverage en go">&#xA;</figure>&#xA;</a>&#xA;Incomplete html coverage in go&#xA;This is the end of this super short entry about go testing. For the next entry I will talk a little bit about profiling and I will finish the basic go entries to write again about Python.&#xA;<h2 id="other-resources-on-testing">Other resources on testing</h2>&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://blog.friendsofgo.tech/posts/empezando-con-los-tests-automatizados-en-go/" rel="noopener" target="_blank">Started with Friends of go automated tests</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://pkg.go.dev/testing" rel="noopener" target="_blank">Official Go documentation</a>&#xA;</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://github.com/stretchr/testify#assert-package" rel="noopener" target="_blank">Testify, to emulate Javascript testing syntax</a>&#xA;</li>&#xA;</ul>&#xA;

Go already has a testing module in its standard library that is ready for our use, we just need to import it and use it.&#xA;&#xA;&#xA;&#xA; <img width="60" height="60" src="https://res.cloudinary.com/dwrscezd2/image/upload/v1717959563/Go_gopher_favicon_uzxa20.svg" alt="">&#xA; <a target="_blank" rel=”nofollow” href="https://coffeebytes.dev/en/pages/go-programming-language-tutorial/">&#xA; Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it</a>&#xA;&#xA;&#xA;<h2 id="testing-preparation-in-go">Testing preparation in go</h2>&#xA;For the tests to be carried out we need:&#xA;A file ending in _test.go * A file ending in _test.go * Run the command go test.

Go: basic testing and coverage

In past posts I talked a bit about &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/">goroutines, deadlocks and channels</a>&#xA;. But there is another quite interesting issue about goroutines that stands out when we use asynchrony and there are many functions accessing data at the same time and. Multiple functions reading and writing the same information can lead to chaotic situations where very strange things can happen.&#xA;&#xA;&#xA;&#xA; <img width="60" height="60" src="https://res.cloudinary.com/dwrscezd2/image/upload/v1717959563/Go_gopher_favicon_uzxa20.svg" alt="">&#xA; <a target="_blank" rel=”nofollow” href="https://coffeebytes.dev/en/pages/go-programming-language-tutorial/">&#xA; Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it</a>&#xA;&#xA;&#xA;<h2 id="what-is-a-race-condition">What is a race condition?</h2>&#xA;If you already know what a race condition is, you can skip to the next section. If you do not, keep reading.&#xA;A race condition occurs when two threads have access to a shared variable at the same time. Let me give you a more detailed example.&#xA;Imagine it&rsquo;s a special day on your dubious content website and you&rsquo;re going to give away 20 tokens to your users for every new user they refer. Then you decide to use goroutines to implement the functionality. Everything goes smoothly, until one of your customers contacts you about an alleged bug.&#xA;DarkLord69 claims to have referred two people, however he complains that his tokens have not increased by 40, but by 20.&#xA;You probably think you&rsquo;ve already messed up in the code, but everything looks good, so what could have gone wrong? After reviewing your code you realize that the problem is in the goroutines.&#xA;It happens that a goroutine read the amount of DarkLord69&rsquo;s credits: 120, and, almost at the same time, another goroutine read those same credits.&#xA;The first goroutine said, &ldquo;I have 120 credits, if I add 20, the total of new credits is 140&rdquo;.&#xA;While the second goroutine, almost at the same time as the first one, said: &ldquo;I read that there were 120 credits, if I add 20, the total of new credits will be 140&rdquo;.&#xA;Both goroutines are setting the total credits at 140, because they read, almost at the same time, that there were 120.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-race-conditions-on-goroutines-and-mutexes/images/race-conditions-go.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-race-conditions-on-goroutines-and-mutexes/images/race-conditions-go.png"&#xA; loading="lazy"&#xA; alt="Example of a race condition in go"><figcaption>&#xA; Example of a race condition in go.&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;As you may already know, DarkLord69 will not be joyful, but if you give him some extra tokens as compensation there should be no problem. But what if instead of tokens it had been money, or perhaps some other more precious commodity?&#xA;Now let&rsquo;s go to an example made in code.&#xA;If you don&rsquo;t understand what the waitgroup does, I wrote a post where I explain &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-use-of-channels-to-communicate-goroutines/">goroutines, channels and waitgroups</a>&#xA; that you can refer to. For now, just keep in mind that they wait for all goroutines to finish before proceeding with the code execution.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import (&#xA; &#34;fmt&#34;&#xA; &#34;sync&#34;&#xA;)&#xA;&#xA;var tokens int = 120&#xA;&#xA;func main() {&#xA; var wg sync.WaitGroup&#xA; for i := 0; i &lt; 10; i++ {&#xA; &#x9;wg.Add(1)&#xA; &#x9;go func() {&#xA; &#x9;&#x9;tokens_leidos := tokens&#xA; &#x9;&#x9;tokens_leidos += 20&#xA; &#x9;&#x9;tokens = tokens_leidos&#xA; &#x9;&#x9;wg.Done()&#xA; &#x9;}()&#xA; }&#xA; wg.Wait()&#xA; fmt.Println(tokens)&#xA;}&#xA;</code></pre></div>The important part is that, within a goroutine, we read the variable content tokens, then increment it by 20 and then assign that result to tokens again. But no problem, if we run the code we will get the correct result: 320 (120 tokens + One increment of 20 tokens for each of the 10 goroutines).&#xA;Everything is perfect, isn&rsquo;t it? No, the code is so small and the process happens so fast that you don&rsquo;t notice the problem.&#xA;If we add a small waiting time in the goroutines (which can be caused by a database access or any other process), we will face the problem head on.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import (&#xA; &#34;fmt&#34;&#xA; &#34;sync&#34;&#xA; &#34;time&#34;&#xA;)&#xA;&#xA;var tokens int = 120&#xA;&#xA;func main() {&#xA; var wg sync.WaitGroup&#xA; for i := 0; i &lt; 10; i++ {&#xA; &#x9;wg.Add(1)&#xA; &#x9;go func() {&#xA; &#x9;&#x9;tokens_leidos := tokens&#xA; &#x9;&#x9;time.Sleep(1 * time.Millisecond) // Tiempo de espera&#xA; &#x9;&#x9;tokens_leidos += 20&#xA; &#x9;&#x9;tokens = tokens_leidos&#xA; &#x9;&#x9;wg.Done()&#xA; &#x9;}()&#xA; }&#xA; wg.Wait()&#xA; fmt.Println(tokens)&#xA;}&#xA;</code></pre></div>After executing the code, instead of the correct result, I get 140 with 1 millisecond and 200 with one microsecond.&#xA;&#xA;<h2 id="detect-race-conditions-with-race">Detect race conditions with &ndash;race</h2>&#xA;To help us identify these problems, Go has a flag to detect race conditions, if we run the compilation with the &ndash;race flag we will see that go warns us that our code has race conditions.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">go build --race main.go&#xA;./main.go&#xA;==================&#xA;WARNING: DATA RACE&#xA;Read at 0x0000005fe430 by goroutine 8:&#xA;# ...&#xA;</code></pre></div><h2 id="preventing-race-conditions-with-mutex">Preventing race conditions with Mutex</h2>&#xA;There is an object called Mutex (mutual exclusion) that will ensure that our code does not access a variable until we tell it to, preventing race conditions from occurring.&#xA;Think of a mutex as a lock, which will block access to our code by other goroutines until we release it.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">var lock sync.Mutex&#xA;lock.Lock()&#xA;// Todo lo que está aquí está bloqueado para el resto de las goroutines&#xA;lock.Unlock()&#xA;</code></pre></div><a hx-boost="false" href="https://coffeebytes.dev/en/go/go-race-conditions-on-goroutines-and-mutexes/images/mutex-lock-en-go.png">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-race-conditions-on-goroutines-and-mutexes/images/mutex-lock-en-go.png"&#xA; loading="lazy"&#xA; alt="The Lock method of Mutex blocks code access, while the Unlock method releases it"><figcaption>&#xA; How Mutex works in go&#xA; </figcaption>&#xA;</figure>&#xA;</a>.&#xA;If we protect the reading and writing of the tokens variable, our code should work perfectly. Even if we compile with the &ndash;race flag and run it, Go will no longer return a warning.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import (&#xA; &#34;fmt&#34;&#xA; &#34;sync&#34;&#xA; &#34;time&#34;&#xA;)&#xA;&#xA;var tokens int = 120&#xA;&#xA;func main() {&#xA; var wg sync.WaitGroup&#xA; var lock sync.Mutex&#xA; for i := 0; i &lt; 10; i++ {&#xA; &#x9;wg.Add(1)&#xA; &#x9;go func() {&#xA; &#x9;&#x9;lock.Lock()&#xA; &#x9;&#x9;// Todo lo que está aquí está bloqueado para el resto de las goroutines&#xA; &#x9;&#x9;tokens_leidos := tokens&#xA; &#x9;&#x9;time.Sleep(1 * time.Millisecond)&#xA; &#x9;&#x9;tokens_leidos += 20&#xA; &#x9;&#x9;tokens = tokens_leidos&#xA; &#x9;&#x9;lock.Unlock()&#xA; &#x9;&#x9;wg.Done()&#xA; &#x9;}()&#xA; }&#xA; wg.Wait()&#xA; fmt.Println(tokens)&#xA;}&#xA;</code></pre></div><h2 id="read-and-write-mutex">Read and write mutex</h2>&#xA;In addition to the above mutex, Go has a lock called RWMutex, which allows, at any given time, only a single goroutine to write or multiple readers to read.&#xA;RWmutex works by activating the lock when there is a process writing to the lock, during which time the contents of the lock cannot be read or written to. But when one process is reading, other processes can read as well.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">var lock sync.RWMutex&#xA;lock.RLock()&#xA;// Todo lo que está aquí está bloqueado para el resto de las goroutines&#xA;lock.RUnlock()&#xA;</code></pre></div><h2 id="useful-mutex-resources">Useful mutex resources</h2>&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://cloudxlab.com/blog/race-condition-and-deadlock/" rel="noopener" target="_blank">Race conditions</a>&#xA; (en inglés)</li>&#xA;</ul>&#xA;

In past posts I talked a bit about &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/">goroutines, deadlocks and channels</a>&#xA;. But there is another quite interesting issue about goroutines that stands out when we use asynchrony and there are many functions accessing data at the same time and. Multiple functions reading and writing the same information can lead to chaotic situations where very strange things can happen.&#xA;&#xA;&#xA;&#xA; <img width="60" height="60" src="https://res.cloudinary.com/dwrscezd2/image/upload/v1717959563/Go_gopher_favicon_uzxa20.svg" alt="">&#xA; <a target="_blank" rel=”nofollow” href="https://coffeebytes.dev/en/pages/go-programming-language-tutorial/">&#xA; Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it</a>&#xA;

Go: race conditions on goroutines and mutexes

When working with channels there is a quite common error that occurs when you are not familiar with the concepts, the error is &ldquo;fatal error: all goroutines are asleep - deadlock!&rdquo;. The first time I saw this error I was perplexed and, although I knew how to fix it, I didn&rsquo;t understand why it happened, so in this post I explain why it happens as I would have liked to have read it at the time.&#xA;&#xA;&#xA;&#xA; <img width="60" height="60" src="https://res.cloudinary.com/dwrscezd2/image/upload/v1717959563/Go_gopher_favicon_uzxa20.svg" alt="">&#xA; <a target="_blank" rel=”nofollow” href="https://coffeebytes.dev/en/pages/go-programming-language-tutorial/">&#xA; Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it</a>&#xA;&#xA;&#xA;<h2 id="why-does-the-error-fatal-error-all-goroutines-are-asleep---deadlock-happen">Why does the error fatal error: all goroutines are asleep - deadlock! happen?</h2>&#xA;This error occurs when:&#xA;<ul>&#xA;<li>A channel sends information, but no channel is there to receive it.</li>&#xA;<li>There is a channel that receives information, but not channel that sends it.</li>&#xA;<li>When we are not inside a goroutine other than the one from the main function.</li>&#xA;</ul>&#xA;In any of these cases the goroutines are &ldquo;waiting&rdquo; either to send or receive information, so we can say that we are &ldquo;stuck&rdquo; and that is when we receive the fatal error: all goroutines are asleep - deadlock!&#xA;&#xA;<h2 id="blocking-operations">Blocking operations</h2>&#xA;In go, operations that send or receive channel values are blocking inside their own goroutine (remember that the main function is a goroutine), i.e., they keep code execution waiting:&#xA;<ul>&#xA;<li>If an operation receives information from a channel, it will be blocked until it receives it.</li>&#xA;<li>If an operation sends information to a channel, it will be blocked until the sent information is received.</li>&#xA;</ul>&#xA;Knowing the two previous situations we will have two main cases:&#xA;<h3 id="blocking-or-deadlock-due-to-lack-of-sender">Blocking or deadlock due to lack of sender</h3>&#xA;What if an operation is waiting to receive information from a channel, but that channel is never going to send anything?&#xA;It happens when there is a channel that receives information, but not one that sends it.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/images/deadlock-sender-go.jpg">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/images/deadlock-sender-go.jpg"&#xA; loading="lazy"&#xA; alt="Deadlock due to lack of input channel"><figcaption>&#xA; There is no operation that sends data through a channel.&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import (&#xA; &#34;fmt&#34;&#xA;)&#xA;&#xA;func main() {&#xA;&#xA; c := make(chan string)&#xA; fmt.Println(&lt;-c)&#xA;}&#xA;// fatal error: all goroutines are asleep - deadlock!&#xA;</code></pre></div><h3 id="blocking-or-deadlock-for-lack-of-recipient">Blocking or deadlock for lack of recipient</h3>&#xA;What if an operation sends information to a channel but there is no longer any other goroutine receiving information from that channel?&#xA;It happens when there is a channel that sends information, but not one that receives it.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/images/deadlock-receiver-go.jpg">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/images/deadlock-receiver-go.jpg"&#xA; loading="lazy"&#xA; alt="Deadlock due to lack of output channel"><figcaption>&#xA; In a deaclock due to lack of output channel there is no operation that receives text through a channel.&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;func main() {&#xA;&#xA; c := make(chan string)&#xA; c &lt;- &#34;text&#34;&#xA;}&#xA;// fatal error: all goroutines are asleep - deadlock!&#xA;</code></pre></div><h3 id="blocking-by-absence-of-goroutine">Blocking by absence of goroutine</h3>&#xA;There is a third case: if there is an operation that writes to and reads from a channel, as in the example below, but it is not inside another goroutine (different from the main goroutine) that reads the value, we will get a deadlock error.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import &#34;fmt&#34;&#xA;&#xA;func main() {&#xA; var channel = make(chan string)&#xA; channel &lt;- &#34;forty two&#34;&#xA; fmt.Println(&lt;-channel) // This should be in other goroutine&#xA; fmt.Println(&#34;Finished&#34;)&#xA;}&#xA;// fatal error: all goroutines are asleep - deadlock!&#xA;</code></pre></div>In the three previous cases the program would be waiting and, as you know, it makes no sense to keep the program waiting for something that will never happen, so the execution is aborted and the fatal error &ldquo;fatal error: all goroutines are asleep - deadlock!&rdquo; is generated.&#xA;The above situation is referred to as a deadlock.&#xA;<h2 id="deadlocks-or-dead-spots">Deadlocks or dead spots</h2>&#xA;In a deadlock there is a goroutine waiting to read or write to a channel, however there is no goroutine running because they are waiting for each other; they are in a deadlock from which no progress can be made.&#xA;<h3 id="prevent-deadlocks-on-go-with-goroutines">Prevent deadlocks on go with goroutines</h3>&#xA;In the &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-use-of-channels-to-communicate-goroutines/">introduction to channels in go post</a>&#xA;, I told you that the default capacity of a channel is 0, this causes that we cannot store data in the channels by default. If we try to store a data in a channel, we will get an error from the compiler, because there is no other goroutine to receive the value immediately.&#xA;To prevent a deadlock, we can immediately use the channel data by creating a goroutine that uses the channel value.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import &#34;fmt&#34;&#xA;&#xA;func main() {&#xA; var channel = make(chan int)&#xA; go func(channel chan int) {&#xA; channel &lt;- 42&#xA; }(channel)&#xA; fmt.Println(&lt;-channel)&#xA; fmt.Println(&#34;Finished&#34;)&#xA;}&#xA;// No ocurre el error de deadlock&#xA;</code></pre></div>In this case I have created a goroutine with an anonymous function that reads from the channel.&#xA;<h2 id="how-to-prevent-deadlocks-and-all-goroutines-are-asleep-error-in-go-using-buffered-channels">How to prevent deadlocks and all goroutines are asleep error in go using buffered channels</h2>&#xA;If a channel does not have buffer, the value remains &ldquo;retained&rdquo; until it is received, blocking the execution while it happens, causing the deadlock or deadlock if no one receives it.&#xA;On the other hand, if the channel is buffered, then execution is blocked until the value has been copied to the buffer, so we will not get an error, even if no goroutine receives it.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/images/deadlocks-prevencion-go-1.jpg">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/images/deadlocks-prevencion-go-1.jpg"&#xA; loading="lazy"&#xA; alt="Schematic of deadlock prevention in go"><figcaption>&#xA; In a buffered channel it does not matter if the value is not read because it has already been copied to the buffer.&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import &#34;fmt&#34;&#xA;&#xA;func main() {&#xA; var channel = make(chan int, 1)&#xA; channel &lt;- 42&#xA; fmt.Println(&lt;-channel)&#xA; fmt.Println(&#34;Finished&#34;)&#xA;}&#xA;// No deadlock error&#xA;</code></pre></div><h2 id="more-resources-on-deadlocks-in-go">More resources on deadlocks in go</h2>&#xA;To conclude the article, I&rsquo;ll share with you some interesting resources about deadlocks in Go that I consider worth mentioning.&#xA;<ul>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://gist.github.com/YumaInaura/8d52e73dac7dc361745bf568c3c4ba37" rel="noopener" target="_blank">Golang - Understanding channel, buffer, blocks, deadlocks and gorotuines</a>&#xA;.</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://stackoverflow.com/questions/61759204/why-a-go-routine-block-on-channel-is-considered-as-deadlock" rel="noopener" target="_blank">Why a goroutine block on channel is considered as deadlock?</a>&#xA;.</li>&#xA;<li>&#xA;&#xA;<a class="markdown-link" href="https://go.dev/doc/effective_go#channels" rel="noopener" target="_blank">Effective go, channels</a>&#xA;</li>&#xA;</ul>&#xA;

When working with channels there is a quite common error that occurs when you are not familiar with the concepts, the error is &ldquo;fatal error: all goroutines are asleep - deadlock!&rdquo;. The first time I saw this error I was perplexed and, although I knew how to fix it, I didn&rsquo;t understand why it happened, so in this post I explain why it happens as I would have liked to have read it at the time.

Go: channels, understanding the goroutines deadlocks

So far I have explained how to run a goroutine, execute code concurrently with the goroutines and wait for them to finish executing but our goroutines can&rsquo;t do anything else, they can&rsquo;t cooperate with each other to speed up the processes.&#xA;Imagine you have a web scrapper that gets data from the internet concurrently; we get the data with goroutines and process it with goroutines. do we have to wait for all the goroutines to finish to use it? Ideally, the goroutines should communicate with each other the data and continue processing.&#xA;&#xA;&#xA;&#xA; <img width="60" height="60" src="https://res.cloudinary.com/dwrscezd2/image/upload/v1717959563/Go_gopher_favicon_uzxa20.svg" alt="">&#xA; <a target="_blank" rel=”nofollow” href="https://coffeebytes.dev/en/pages/go-programming-language-tutorial/">&#xA; Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it</a>&#xA;&#xA;&#xA;<h2 id="communicating-goroutines-with-channels">Communicating goroutines with channels</h2>&#xA;Channels are &ldquo;conduits&rdquo; that accept a single type of data. Through these channels we &ldquo;introduce&rdquo; information that we can later &ldquo;take out&rdquo;.&#xA;The goroutines can send data to the channels and also read data from them, thus communicating with each other.&#xA;<a hx-boost="false" href="https://coffeebytes.dev/en/go/go-use-of-channels-to-communicate-goroutines/images/channels-en-go.jpg">&#xA;<figure class="md-local-image"><img src="https://coffeebytes.dev/en/go/go-use-of-channels-to-communicate-goroutines/images/channels-en-go.jpg"&#xA; loading="lazy"&#xA; alt="Schematic diagram of how a channel works in go"><figcaption>&#xA; Basic diagram of how channels work in Go&#xA; </figcaption>&#xA;</figure>&#xA;</a>&#xA;A channel in go is declared with make and the word chan, which refers to the word channel.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">c := make(chan string)&#xA;</code></pre></div>Channels are used to communicate goroutines, as in the &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/software-architecture/worker-pool-design-pattern-explanation/">worker-pool design pattern</a>&#xA;&#xA;<h3 id="channels-or-buffered-channels">Channels or buffered channels</h3>&#xA;The make function allows you to pass as an extra argument the limit amount of simultaneous data to be handled by that channel. **This is known as a buffered channel.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">c := make(chan string, 1)&#xA;</code></pre></div><h3 id="channels-or-unbuffered-channels">Channels or unbuffered channels</h3>&#xA;If we do not specify a buffer size the channel:&#xA;<ul>&#xA;<li>It will block the sender until the recipient is ready to receive the information.</li>&#xA;<li>Block the recipient until the sender is ready to send the information</li>&#xA;</ul>&#xA;Or to put it another way, if you try to send data to an unbuffered channel, the operation will block until the data is received. If you try to receive data it will block until the data is sent.&#xA;These channels are useful when you want to ensure that the sender and recipient are in sync and that the information is transmitted immediately, but they can lead to deadlocks if the sender and recipient are not in sync.&#xA;<h3 id="accessing-a-channel-in-a-function">Accessing a channel in a function</h3>&#xA;When we want to refer to the channel as an argument of a function, it is necessary to indicate the data type of the channel.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func say(text string, c chan string) {}&#xA;</code></pre></div>The data type of a channel can also be one defined using a struct.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func say(text string, c chan MiStruct) {}&#xA;</code></pre></div><h3 id="putting-data-into-a-channel">Putting data into a channel</h3>&#xA;To indicate data input through the channel, we use the character pair &lt;-&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func say(text string, c chan string) {&#xA; c &lt;- text&#xA;}&#xA;</code></pre></div><h3 id="pulling-data-from-a-channel">Pulling data from a channel</h3>&#xA;To obtain the channel response we reverse the order between the channel and the &lt;- symbol.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">fmt.Println(&lt;-c)&#xA;</code></pre></div><h3 id="entering-and-retrieving-data-from-a-channel-in-go">Entering and retrieving data from a channel in go</h3>&#xA;The complete process of entering and extracting data from a channel in go would look something like this:&#xA;Create a channel of a data type using make.&#xA;2. We introduce a data (in this case string) to the channel using a&#xA;goroutine&#xA;3. Extract the text from the channel and use it (in this case just print it).&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">package main&#xA;&#xA;import &#34;fmt&#34;&#xA;&#xA;func escribirEnCanal(texto string, c chan string) {&#xA; c &lt;- texto&#xA;}&#xA;&#xA;func main() {&#xA; c := make(chan string)&#xA; go escribirEnCanal(&#34;Dato de un canal&#34;, c)&#xA; fmt.Println(&lt;-c)&#xA;}&#xA;</code></pre></div><h3 id="input-and-output-channels">Input and output channels</h3>&#xA;There are channels that receive information and channels that output information, by default a channel is bidirectional, but we can declare input and output channels.&#xA;To identify them, observe the flow of the arrow around the word chan; one enters (or goes to) chan and the other leaves chan.&#xA;This is an input channel.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func say(text string, c chan&lt;- string) {}&#xA;</code></pre></div>While this is an output channel.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">func say(text string, c &lt;-chan string) {}&#xA;</code></pre></div>It is important to define the type of channel because, with bidirectional channels, we run the risk of causing a &#xA;&#xA;<a class="markdown-link" href="https://coffeebytes.dev/en/go/go-channels-understanding-the-goroutines-deadlocks/">deadlock in our go program</a>&#xA;.&#xA;&#xA;<h2 id="channel-capacity">Channel capacity</h2>&#xA;Remember I told you that the make function could set the maximum number of data a channel can work with? Well, it is possible to retrieve that information using the len function.&#xA;The len function tells us how much data there is in a channel, while cap returns the maximum capacity, respectively.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">c := make(chan string, 3)&#xA;c &lt;- &#34;dato1&#34;&#xA;c &lt;- &#34;dato2&#34;&#xA;fmt.Println(len(c), cap(c))&#xA;// 2 3&#xA;</code></pre></div>In the above example, the channel has two slots occupied, but has a total capacity of 3, even though not all of its slots are occupied.&#xA;<h2 id="close-a-channel-in-go">Close a channel in go</h2>&#xA;If we want to disable a channel, even if it has available capacity to store more data, the close function allows us to do so.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">c :=make(chan string, 3) &#xA;c &lt;- &#34;dato1&#34; &#xA;c &lt;- &#34;dato2&#34; &#xA;close(c)&#xA;c &lt;- &#34;dato3&#34;&#xA;//panic: send on closed channel&#xA;</code></pre></div><h2 id="iterate-on-a-channel">Iterate on a channel</h2>&#xA;Range is ideal for iterating over channel data. However, it is important to note that, there is no certainty about what data we will receive since the channel content can come from multiple goroutines.&#xA;<div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go">c := make(chan string, 3)&#xA;c &lt;- &#34;dato1&#34;&#xA;c &lt;- &#34;dato2&#34;&#xA;close(c)&#xA;for message := range c {&#xA; fmt.Println(message)&#xA;}&#xA;</code></pre></div>With that you know the basic information about the channels and how you can use them to communicate goroutines.&#xA;

So far I have explained how to run a goroutine, execute code concurrently with the goroutines and wait for them to finish executing but our goroutines can&rsquo;t do anything else, they can&rsquo;t cooperate with each other to speed up the processes.&#xA;Imagine you have a web scrapper that gets data from the internet concurrently; we get the data with goroutines and process it with goroutines. do we have to wait for all the goroutines to finish to use it? Ideally, the goroutines should communicate with each other the data and continue processing.

Eduardo Zepeda's latests posts