<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>Regression toward the meaning</title>
<link>https://blog.horaceg.xyz/</link>
<atom:link href="https://blog.horaceg.xyz/index.xml" rel="self" type="application/rss+xml"/>
<description>Horace Guy&#39;s blog about random things</description>
<generator>quarto-1.9.37</generator>
<lastBuildDate>Fri, 21 Nov 2025 00:00:00 GMT</lastBuildDate>
<item>
  <title>A real estate investment calculator</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/housing/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/housing/monthly_payments.png" class="img-fluid figure-img"></p>
<figcaption>Monthly Payments: loan &amp; interest</figcaption>
</figure>
</div>
<p>As a young-ish professional, I’ve been wondering whether I should buy my primary residence or keep renting. With this thought, I’ve been driven to answer this question in a rational manner.</p>
<p>You can see the results of this investigation on this website: <a href="https://linemo.horaceg.xyz/">linemo</a>.</p>
<section id="loan-interest-rates" class="level2">
<h2 class="anchored" data-anchor-id="loan-interest-rates">Loan interest rates</h2>
<p>It all started when I wanted to know how much I would pay in interest for a loan. Because let’s face it, the interest rate is super <em>opaque</em> about how much interest I would pay over the life of the loan. One might, naively, think that a 4% interest rate means that I would pay 4% of the principal in interest over the life of the loan (In a loan, the principal is the original amount of money you borrow, not including any interest or fees).</p>
<p>However, this is not the case. The interest rate is applied to the remaining balance of the loan, every month. This means that the interest you pay decreases over time ; but the amount of interest you pay over the life of the loan is way higher than 4% of the principal.</p>
<p>So I’ve built a simple <a href="https://linemo.horaceg.xyz/loan">loan simulator</a> to answer this question.</p>
<section id="case-study-contracting-a-loan" class="level3">
<h3 class="anchored" data-anchor-id="case-study-contracting-a-loan">Case study : contracting a loan</h3>
<p>Let’s assume that I pay every month 1,500€ over 20 years. The amount of the loan is 266,000€.</p>
<p>According to the current interest rates around 3.2%, I would pay 94 000€ in interest over the life of the loan ; or around 35% of the principal. Note that this is way more than the 4% of the principal that one may naively think.</p>
<p><a href="https://linemo.horaceg.xyz/loan?mode=mp&amp;monthly=1500&amp;loan=165&amp;duration=20&amp;rate=3.2">Here is the case study</a>.</p>
<p>Now that we understand how a loan works, let’s move on to the next question: should I buy a home, to live in, or rent it ?</p>
</section>
</section>
<section id="buy-or-rent" class="level2">
<h2 class="anchored" data-anchor-id="buy-or-rent">Buy or rent?</h2>
<p>You generally have 2 options:</p>
<ul>
<li>Buy a home (to live in)</li>
<li>Rent a home</li>
</ul>
<p>The question is: which option is better for you in the short, medium, and long run ?</p>
<p>Surprisingly, the answer is not obvious. At a first glance, it seems that buying a home is always better: instead of <strong>throwing money away</strong> by renting, you are <strong>investing</strong> in your own home. However, this is not always the case, mostly because of the costs associated with buying a home: <strong>agency fees</strong> and <strong>closing costs</strong>.</p>
<p>Agency fees are typically “paid by the seller” to the agency that helps them sell their home. Closing costs are the costs associated with the purchase of a home, such as notary fees, registration fees, etc. When summing up both costs, it is possible that the total cost of buying a home is higher than the total cost of renting a home. This means that for every transaction, you are <strong>losing money</strong> by buying a home. Now, it <em>could</em> be worth it <em>if you stay long enough</em>. How long ? This is the question that the <a href="https://linemo.horaceg.xyz/buy-vs-rent">buy or rent</a> calculator answers.</p>
<section id="case-study-renting-or-buying-an-apartment-in-france---fall-2025" class="level3">
<h3 class="anchored" data-anchor-id="case-study-renting-or-buying-an-apartment-in-france---fall-2025">Case study : renting or buying an apartment in France - fall 2025</h3>
<p><a href="https://linemo.horaceg.xyz/buy-vs-rent?mode=mp&amp;monthly=1500&amp;loan=309&amp;duration=25&amp;rate=3.2&amp;rent=1200&amp;rentInc=1&amp;savingsRoi=6&amp;tax=30&amp;down=15&amp;appreciation=1&amp;closing=7.7&amp;agency=5&amp;extras=50&amp;propertyTax=1000&amp;maintenance=150">Here is a realistic scenario</a>.</p>
<p>As you can see, the conclusion is: <em>Wait 15.2 years for buying to overtake renting</em>. So in this scenario, buying is better than renting ONLY if you stay in the apartment for more than 15.2 years.</p>
<p>Of course, there are many assumptions here that may not be true, especially since they are based on future values. For instance: the rent increase, the evolution of real estate prices, the Return on investment (ROI) of the savings, the tax rate, etc.</p>
<p>However, this calculator let’s you play with the assumptions to see how the result changes.</p>
</section>
</section>
<section id="the-math" class="level2">
<h2 class="anchored" data-anchor-id="the-math">The math</h2>
<p><a href="https://linemo.horaceg.xyz/how-it-works">Here is the math behind the calculator</a>. In a nutshell, thanks to arithmetico-geometric series, we can compute all the interest payments, and the total amount of interest paid over the life of the loan.</p>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>All in all, this calculator is a tool to help you make decisions. It is not perfect, but it can be a useful tool to help:</p>
<ul>
<li>Understand how a loan works, and how many interest you will pay</li>
<li>Understand the costs associated with buying / renting a home</li>
<li>Understand the impact of the assumptions on the result</li>
<li>Make a decision based on the results</li>
</ul>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2025,
  author = {Guy, Horace},
  title = {A Real Estate Investment Calculator},
  date = {2025-11-21},
  url = {https://blog.horaceg.xyz/posts/housing/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2025" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2025. <span>“A Real Estate Investment Calculator.”</span>
November 21. <a href="https://blog.horaceg.xyz/posts/housing/">https://blog.horaceg.xyz/posts/housing/</a>.
</div></div></section></div> ]]></description>
  <category>real estate</category>
  <category>investment</category>
  <category>javascript</category>
  <category>programming</category>
  <guid>https://blog.horaceg.xyz/posts/housing/</guid>
  <pubDate>Fri, 21 Nov 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>An interactive introduction to probabilistic programming</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/intro-ppl/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/intro-ppl/badnormal.png" class="img-fluid figure-img"></p>
<figcaption>Prior, posterior and likelihood</figcaption>
</figure>
</div>
<div class="container">
<iframe id="iframe" scrolling="no" src="https://intro-ppl.pages.dev/">
</iframe>
</div>
<style>
    .container {
        display: flex;
        width: 100%; 
        height: 100%; 
        flex-direction: column; 
        overflow: auto;
    }
    iframe {
        flex-grow: 1; 
        border: none; 
        margin: 0; 
        padding: 0; 
        display: block;
        height: 800vh; 
    }
</style>



<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2025,
  author = {Guy, Horace},
  title = {An Interactive Introduction to Probabilistic Programming},
  date = {2025-01-06},
  url = {https://blog.horaceg.xyz/posts/intro-ppl/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2025" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2025. <span>“An Interactive Introduction to Probabilistic
Programming.”</span> January 6. <a href="https://blog.horaceg.xyz/posts/intro-ppl/">https://blog.horaceg.xyz/posts/intro-ppl/</a>.
</div></div></section></div> ]]></description>
  <category>machine learning</category>
  <category>programming</category>
  <category>probabilistic</category>
  <category>bayesian</category>
  <category>inference</category>
  <guid>https://blog.horaceg.xyz/posts/intro-ppl/</guid>
  <pubDate>Mon, 06 Jan 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>An introduction to bayesian inference</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/bayes/</link>
  <description><![CDATA[ 





<p>I first learned about bayesian statistics in a job posting for an internship. At the time, I was about 20 years old and had never heard of it. The concept was so alien to me that I first read <em>bayesian <strong>interference</strong></em>. I figured it had something to do with optics, such as the Michelson interferometer that I (still) fear.</p>
<p>Fast-forward a small decade and I now consider myself a (secular) bayesian. What is bayesian statistics <em>really</em> about ?</p>
<section id="bayes-theorem" class="level2">
<h2 class="anchored" data-anchor-id="bayes-theorem">Bayes’ theorem</h2>
<p>In the beginning of every probability book, you will see Bayes’ thoerem for events proudly stated as a direct consequence of the definition of conditional probability and the product (chain) rule. I argue here that it is not really helpful to think of bayes this way. If you wonder what Bayes’ theorem is, you can look it up online.</p>
</section>
<section id="a-biased-coin" class="level2">
<h2 class="anchored" data-anchor-id="a-biased-coin">A biased coin</h2>
<p>When I was in high school, I remember that we saw in a book that if you toss a coin 3 times and it leads 3 heads, the probability that the next toss will lead head too is … 0.5 (50%). I was a bit surprised by the result, because I overlooked the crucial fact that we knew the coin was unbiased, i.e.&nbsp;that it always leads to heads with a probability of 0.5.</p>
<p>However, if you don’t know if the coin is biased or not, then your belief that the next coin toss is going to lead heads should surely be a little higher than tails. A relevant question you can ask yourself is: at what odds would you accept to bet on tails ? Assuming you want your expected gain to be positive of course.</p>
</section>
<section id="the-reverend-thomas-bayes" class="level2">
<h2 class="anchored" data-anchor-id="the-reverend-thomas-bayes">The reverend Thomas Bayes</h2>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/thomas-bayes.png" class="img-fluid figure-img"></p>
<figcaption>Only known portait of possibly Thomas Bayes. 18th century</figcaption>
</figure>
</div>
<p>Born c.&nbsp;1701, Thomas Bayes was a pioneer of statistics, and a minister of the Church. Coincidentally, these days some people are so <a href="https://www.inference.vc/the-secular-bayesian-using-belief-distributions-without-really-believing/">deep into the Bayesian doctrine</a> that we can consider them as being part of the <em>Bayesian church</em>.</p>
<p>In 1763 was published <em>An Essay towards solving a Problem in the Doctrine of Chances</em>, 2 years after Bayes’ death. In it, he states in the beginning the Bayes’ theorem as we know it. <a href="https://en.wikipedia.org/wiki/An_Essay_towards_solving_a_Problem_in_the_Doctrine_of_Chances#Outline">On wikipedia</a> we can read:</p>
<blockquote class="blockquote">
<p>it does not appear that Bayes emphasized or focused on this finding. Rather, he focused on the finding the solution to a much broader inferential problem:</p>
<p>“Given the number of times in which an unknown event has happened and failed [… Find] the chance that the probability of its happening in a single trial lies somewhere between any two degrees of probability that can be named.”</p>
</blockquote>
<p>Sounds like he was wondering about the coin toss problem !</p>
</section>
<section id="inverse-probability" class="level2">
<h2 class="anchored" data-anchor-id="inverse-probability">Inverse probability</h2>
<p>During the 18 &amp; 19th century, mathematicians such as Laplace and De Morgan refer to the <em>inverse probability</em> for the probability distribution of an unobserved variable. The term <em>Bayesian</em> was coined by Ronald Fisher in 1950, with the development of frequentism, to refer to the same thing.</p>
<p>Intuitively, statistics is about inverse problems : if you have some unobserved variables and possibly observed variables (often in the form of events), how can you assess the distribution of the unobserved variable ?</p>
</section>
<section id="the-inference-theorem" class="level2">
<h2 class="anchored" data-anchor-id="the-inference-theorem">The inference theorem</h2>
<p>In Bayesian inference, parameters are random and data is not.</p>
<p>Here is the formula that we care about in inferential bayesian statistics:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7Bposterior%7D%20%5Cpropto%20%5Cmathrm%7Bprior%7D%20%5Ctimes%20%5Cmathrm%7Blikelihood%7D"></p>
<p>The <img src="https://latex.codecogs.com/png.latex?%5Cpropto"> sign means proportional to, as such <img src="https://latex.codecogs.com/png.latex?a%20%5Cpropto%20b"> means <img src="https://latex.codecogs.com/png.latex?a%20=%20C%20%5Ctimes%20b"> for some <strong>constant</strong> <img src="https://latex.codecogs.com/png.latex?C">.</p>
<p>In more precise terms, let <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> be the unobserved random variable that we care about (weights &amp; biases in machine learning for instance).<br>
Let <img src="https://latex.codecogs.com/png.latex?p(%5Ctheta)"> denote the distribution of <img src="https://latex.codecogs.com/png.latex?%5Ctheta">. More rigorously, <img src="https://latex.codecogs.com/png.latex?p"> in this post loosely denotes either the probability of an event or the probability density (or probability mass function) of a random variable. When I write <em>distribution</em> it often means density of the random variable.</p>
<p>Let <img src="https://latex.codecogs.com/png.latex?D"> (for <em>Data</em>) denote our observed data, for instance <img src="https://latex.codecogs.com/png.latex?D%20=%20(%5Cmathbf%7Bx%7D,%20%5Cmathbf%7By%7D)%20=%20(x_i,%20y_i)_i"> for a regression or classification problem. Then the above formula can be written more precisely:</p>
<p><img src="https://latex.codecogs.com/png.latex?p(%5Ctheta%20%7C%20D)%20%5Cpropto%20p(%5Ctheta)%20%5Ctimes%20p(D%20%7C%20%5Ctheta)"></p>
<p>The <strong>prior distribution</strong> <img src="https://latex.codecogs.com/png.latex?p(%5Ctheta)"> is the probability distribution of the unobserved random variable (r.v.) of interest <em>a priori</em>, i.e.&nbsp;without having seen any data / observed related r.v. / events. It may reflect your knowledge about the world.</p>
<p>The <strong>likelihood</strong> <img src="https://latex.codecogs.com/png.latex?p(D%20%7C%20%5Ctheta)"> is the probability of the data conditioned on this unobserved random variable. This may sounds paradoxical: how can we condition on something we don’t observe ? Well, remember that this is a function of the unobserved random variable. As such, the likelihood is NOT a density, it doesn’t integrate to 1. We can put it another way: If we knew the unobserved variable, how likely would the data (events) we have observed be ?</p>
<p>The <strong>posterior distribution</strong> <img src="https://latex.codecogs.com/png.latex?p(%5Ctheta%20%7C%20D)"> is what we ultimately care about. It describes the distribution of the <em>unobserved</em> r.v. <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> condtioning on the data that we’ve collected, i.e.&nbsp;events that happened.</p>
<section id="what-to-make-of-this-formula" class="level3">
<h3 class="anchored" data-anchor-id="what-to-make-of-this-formula">What to make of this formula ?</h3>
<p>There are mainly 3 types of inference methods that we can leverage based on the above formula:</p>
<ul>
<li><strong>Maximum likelihood estimation (MLE)</strong>: we seek one point estimate of <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> based on observations and the likelihood</li>
<li><strong>Maximum a posteriori (MAP)</strong>: we seek one point estimate of <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> based on observations and prior knowledge - likelihood and prior</li>
<li><strong>Full posterior inference</strong>: we seek representative samples from the posterior distribution of <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> based on the inference formula.</li>
</ul>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/meme-inferences.png" class="img-fluid figure-img"></p>
<figcaption>Meme about MLE, MAP and posterior sampling</figcaption>
</figure>
</div>
</section>
<section id="maximum-likelihood" class="level3">
<h3 class="anchored" data-anchor-id="maximum-likelihood">Maximum Likelihood</h3>
<p>In this setting, we just optimize the likelihood to get the argmax. In this case, we ignore the prior and the posterior ; hence, we don’t really use the inference formula. Kevin Murphy refers to MLE and MAP as <strong>poor man’s Bayes</strong>.</p>
<p>This method is convenient because we can just optimize the function, with e.g.&nbsp;the gradient, and find the value of <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> that maximizes the likelihood function. We can either compute the gradient in closed form and see where it vanishes, or run an iterative algorithm such as (stochastic) gradient descent.</p>
<p>In practice, we dont maximize the likelihood function but rather we minimize the negative log-likelihood function (NLL) with the logarithm function:</p>
<p><img src="https://latex.codecogs.com/png.latex?NLL(%5Ctheta)%20:=%20-%20%5Clog%20p(D%20%7C%20%5Ctheta)"></p>
<p>Note that in the case of a gaussian likelihood, for a regression, the negative log-likelihood is then equal (up to a constant) to the mean-squared error loss function, widely used in machine learning. In the linear case, this is the <em>Ordinary Least Squares (OLS)</em> method.</p>
</section>
<section id="maximum-a-posteriori" class="level3">
<h3 class="anchored" data-anchor-id="maximum-a-posteriori">Maximum a posteriori</h3>
<p>A more precise inference technique is to find the argmax of the posterior, that is the point that maximizes the posterior distribution. We call this value the <strong>mode</strong> of the distribution.</p>
<p>As with the likelihood, we use the negative logarithm. Since <img src="https://latex.codecogs.com/png.latex?%5Clog%20ab%20=%20%5Clog%20a%20+%20%5Clog%20b">, we want to minimize the sum of the NLL and the negative log-prior.</p>
<p>Note that the MAP with a uniform prior is the same as MLE since in this case the log-prior is a constant with respect to <img src="https://latex.codecogs.com/png.latex?%5Ctheta">.</p>
<p>Intuitively, the negative log-prior is a regularization term when we assume a centered random variable.</p>
<p>In case of a centered Gaussian prior, we find exactly the <img src="https://latex.codecogs.com/png.latex?L%5E2"> regularization framework. If the likelihoods is also Gaussian as above, we find the Ridge regression. The regularization parameter of the Ridge is then equal to the variance of the prior.</p>
<p>If you chose instead a Laplace prior, you will find the <img src="https://latex.codecogs.com/png.latex?L%5E1"> regularization, i.e.&nbsp;the Lasso regression.</p>
<p>In our case, we are now not restricted to Lasso or Ridge, we can choose any prior for the MAP as long as we can take its gradient (assuming we are using an algorithmic differentiation framework such as JAX). We can also freely pick the likelihood we want. How nice ! We are now only constrained by imagination (and tractable probability distributions).</p>
<p>The Expectation-maximization (EM) algorithm can also be used to find the MLE or MAP, if for some reason you can’t have the gradient.</p>
<p>If we are true bayesians<img src="https://latex.codecogs.com/png.latex?%5E%7B%5Cmathrm%7BTM%7D%7D">, we need to sample from the posterior to find at least the posterior mean instead of the mode.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/meme-drake-mle-posterior-sampling.png" class="img-fluid figure-img"></p>
<figcaption>Meme about Maximum likelihood vs sampling from the posterior</figcaption>
</figure>
</div>
</section>
<section id="full-posterior-inference" class="level3">
<h3 class="anchored" data-anchor-id="full-posterior-inference">Full posterior inference</h3>
<p>Last but not least, posterior inference is about efficiently sampling the posterior distribution. We want to obtain a set of samples on the <strong>typical set</strong> of the posterior distribution. This step is usually computationally intensive, as we need to produce enough samples to represent the full probability distribution of the posterior, as opposed to just finding the mode with e.g.&nbsp;MAP.</p>
<section id="inference-techniques" class="level4">
<h4 class="anchored" data-anchor-id="inference-techniques">Inference techniques</h4>
<p>In the best situation, we have the (unnormalized) log-posterior function. That is, we have a computer function that computes the log-density of the posterior distribution for any <img src="https://latex.codecogs.com/png.latex?%5Ctheta">.</p>
<p>So we won, right ? We have the function !</p>
<p>Well, not yet. We still have to sample from this distribution inside the typical set.</p>
<section id="the-typical-set" class="level5">
<h5 class="anchored" data-anchor-id="the-typical-set">The typical set</h5>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/typical-set.png" class="img-fluid figure-img" width="600"></p>
<figcaption>In high dimensions a probability density, <img src="https://latex.codecogs.com/png.latex?%5Cpi(q)">, will concentrate around its mode, but the volume over which we integrate that density, <img src="https://latex.codecogs.com/png.latex?dq">, is much larger away from the mode. Contributions to any expectation are determined by the product of density and volume, <img src="https://latex.codecogs.com/png.latex?%5Cpi(q)%20dq">, which then concentrates in a nearly-singular neighborhood called the typical set (grey). <br> Figure and caption from <a href="https://arxiv.org/abs/1701.02434"><em>A Conceptual Introduction to Hamiltonian Monte Carlo</em>, Betancourt, 2017</a></figcaption>
</figure>
</div>
<p>The typical set is the region where we have <em>both</em> high density and high volume. In high dimensions, it is crucial to think about volume, in that regions with high density don’t necessarily have high volume. In practice, we find the typical set with the sampling process that explores the parameter space.</p>
<p>A popular algorithm is Markov Chain Monte Carlo (MCMC), where we sample from the posterior by exploring the space in an iterative fashion. Each sample define a state in a markov chain, i.e.&nbsp;a sequential discrete process that has no memory, and we explore the space according to some <em>step</em> function.</p>
<p>We can also use Variational Inference, where we approximate the posterior density with a known parameterized family of distributions such as Gaussians that we can optimize with e.g.&nbsp;stochastic gradient descent; in this case, it becomes stochastic variational inference (SVI). Note that the similarity between posterior and the variational distribution is often evaluated with the Kullback-Leibler divergence.</p>
<p>If you need something even more scalable, paying the price of quality, you can use the Laplace approximation <sup>1</sup>.</p>
<p>The detailed machinery of inference algorithms are out of scope for this blog post, but the workhorse of modern bayesian inference. Indeed, for many problems the inference using e.g.&nbsp;MCMC is very computationally intensive, so scalable techniques are highly desired.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/meme-bernie-asking-post-samples.png" class="img-fluid figure-img"></p>
<figcaption>Meme about asking for posterior samples</figcaption>
</figure>
</div>
</section>
</section>
</section>
</section>
<section id="back-to-the-repeated-coin-toss-problem" class="level2">
<h2 class="anchored" data-anchor-id="back-to-the-repeated-coin-toss-problem">Back to the repeated coin toss problem</h2>
<p>We’re going to solve the coin toss problem with Python and Numpyro, a probabilistic programming library based on JAX. We are going to explore full posterior inference with a MCMC algorithm (in our case the No U-Turn Sampler or NUTS).</p>
<section id="uniform-prior" class="level3">
<h3 class="anchored" data-anchor-id="uniform-prior">Uniform prior</h3>
<p>Let’s assume a uniform prior for the bias of the coin, that is <em>any</em> bias is equally likely. Note that you can make one side of a coin heavier, so that it is biased, but I doubt that we could have a coin that always leads heads.</p>
<p>Anyways, as a first step let’s try this model.</p>
<details>
<summary>
First, let’s import the packages we need
</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> numpy <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> np</span>
<span id="cb1-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> matplotlib.pyplot <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> plt</span>
<span id="cb1-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pandas <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> pd</span>
<span id="cb1-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> numpyro <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> ny</span>
<span id="cb1-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> numpyro.distributions <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> dist</span>
<span id="cb1-6"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> numpyro.infer <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> MCMC, NUTS, Predictive</span>
<span id="cb1-7"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> jax <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> random</span>
<span id="cb1-8"></span>
<span id="cb1-9">plt.style.use(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ggplot'</span>)</span>
<span id="cb1-10"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%</span>config InlineBackend.figure_format <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'svg'</span></span></code></pre></div></div>
</details>
<p><br></p>
<p>Let’s describe the data. Note that here 1 means heads, so that the observations <code>[1, 1, 1]</code> map to the realizations of our experiments.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1">tosses <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> np.asarray([<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>])</span>
<span id="cb2-2">nb_heads <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> (tosses <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>).<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">sum</span>()</span>
<span id="cb2-3">total_tosses <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> tosses.shape[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>]</span></code></pre></div></div>
<p>Then, let us define the model. As discussed above, we use a Uniform prior for the bias of the coin. Then, we use a binomial likelihood. The <code>nb_heads</code> parameter use as <code>obs=nb_heads</code> in the likelihood allows us to tell the model to condition on this observation for the inference. This parameter is optional because for prediction we don’t observe it.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> model(N, nb_heads<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>):</span>
<span id="cb3-2">    p <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ny.sample(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p_heads"</span>, dist.Uniform())</span>
<span id="cb3-3">    ny.sample(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nb_heads"</span>, dist.Binomial(N, probs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>p), obs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>nb_heads)</span></code></pre></div></div>
<p>Let’s check a prior predictive simulation, i.e.&nbsp;what we would predict without having seen any data. We simulate 1 toss with 10_000 samples from the prior distribution:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1">prior_predictive <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Predictive(model, num_samples<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10_000</span>)</span>
<span id="cb4-2">all_prior_samples <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> prior_predictive(random.PRNGKey(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>), N<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb4-3">plt.hist(all_prior_samples[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p_heads"</span>], bins<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>)</span>
<span id="cb4-4">plt.title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"(Beta) Prior probability distribution of the bias of the coin p(heads)"</span>)</span></code></pre></div></div>
<p><img src="https://blog.horaceg.xyz/posts/bayes/bayes_files/bayes_6_1.svg" class="img-fluid"></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb5-1">prior_preds <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> all_prior_samples[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nb_heads"</span>]</span>
<span id="cb5-2">prior_proba_next_heads <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> (prior_preds <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>).mean()</span>
<span id="cb5-3">plt.hist(prior_preds)</span>
<span id="cb5-4">plt.title(</span>
<span id="cb5-5">    <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Prior predictive probability of next toss to be Heads is </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>prior_proba_next_heads<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:.3f}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb5-6">)</span></code></pre></div></div>
<p><img src="https://blog.horaceg.xyz/posts/bayes/bayes_files/bayes_7_1.svg" class="img-fluid"></p>
<p>Nothing surprising, the prior probability of the next toss being heads is approximately 0.5, since we have a uniform prior i.e.&nbsp;all possible biases are equally likely.</p>
<p>Now let’s run the inference using a Markov Chain Monte Carlo (MCMC) algorithm to infer the posterior distribution of the bias of the coin. We also plot this distribution and its mean.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb6-1">mcmc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> MCMC(NUTS(model), num_warmup<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>, num_samples<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span>)</span>
<span id="cb6-2">mcmc.run(random.PRNGKey(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>), N<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>total_tosses, nb_heads<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>nb_heads)</span>
<span id="cb6-3">posterior_p_heads_samples <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> mcmc.get_samples()[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p_heads"</span>]</span>
<span id="cb6-4">plt.hist(posterior_p_heads_samples, bins<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>)</span>
<span id="cb6-5">plt.axvline(posterior_p_heads_samples.mean(), linestyle<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"--"</span>, color<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"mean"</span>)</span>
<span id="cb6-6">plt.legend()</span>
<span id="cb6-7">plt.title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Posterior distribution of the probability of Heads (bias of the coin)"</span>)</span>
<span id="cb6-8">mcmc.print_summary()</span></code></pre></div></div>
<pre><code>sample: 100%|██████████| 11000/11000 [00:02&lt;00:00, 5274.72it/s, 3 steps of size 1.05e+00. acc. prob=0.92]



                mean       std    median      5.0%     95.0%     n_eff     r_hat
   p_heads      0.67      0.15      0.68      0.43      0.91   3534.42      1.00

Number of divergences: 0</code></pre>
<p><img src="https://blog.horaceg.xyz/posts/bayes/bayes_files/bayes_9_2.svg" class="img-fluid"></p>
<p>We obtain predictions directly with the <code>Predictive</code> method. Note that since we simulate one toss, this is a Bernoulli experiment thus the predictive probability is exactly equal to the mean.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb8-1">predictive <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Predictive(model, mcmc.get_samples())</span>
<span id="cb8-2">preds <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> predictive(random.PRNGKey(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>), N<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nb_heads"</span>]</span>
<span id="cb8-3">proba_next_heads <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> (preds <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>).mean()</span>
<span id="cb8-4">plt.hist(preds)</span>
<span id="cb8-5">plt.title(</span>
<span id="cb8-6">    <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Posterior probability of next toss to lead to heads is approx. </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>proba_next_heads<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:.3f}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb8-7">)</span></code></pre></div></div>
<p><img src="https://blog.horaceg.xyz/posts/bayes/bayes_files/bayes_11_1.svg" class="img-fluid"></p>
</section>
<section id="a-more-realistic-beta-prior" class="level3">
<h3 class="anchored" data-anchor-id="a-more-realistic-beta-prior">A more realistic Beta prior</h3>
<p>Let’s modify the model with a <img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7BBeta%7D"> prior:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb9-1"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> model(N, nb_heads<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>):</span>
<span id="cb9-2">    p <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ny.sample(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p_heads"</span>, dist.Beta(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>))</span>
<span id="cb9-3">    ny.sample(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nb_heads"</span>, dist.Binomial(N, probs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>p), obs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>nb_heads)</span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/beta-prior/bayes_6_1.svg" class="img-fluid figure-img"></p>
<figcaption>Beta prior</figcaption>
</figure>
</div>
<p>The prior predictive is almost the same since the prior is centered in 1. However, the posterior distribution is very different:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/beta-prior/bayes_9_2.svg" class="img-fluid figure-img"></p>
<figcaption>Beta posterior</figcaption>
</figure>
</div>
<p>Same as before, since we only simulate one toss, the probability of the next toss being Heads is the mean of the posterior distribution:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/bayes/beta-prior/bayes_11_1.svg" class="img-fluid figure-img"></p>
<figcaption>Posterior probability of heads under a beta prior</figcaption>
</figure>
</div>
</section>
</section>
<section id="betting-on-the-outcome" class="level2">
<h2 class="anchored" data-anchor-id="betting-on-the-outcome">Betting on the outcome</h2>
<p>As discovered by Bruno De Finetti, our belief about the world should translate into betting odds. In other words, what are the <strong>fair</strong> odds that you should accept for a given outcome ? The fair decimal odds for a given event of probability <img src="https://latex.codecogs.com/png.latex?p"> can be obtained by taking <img src="https://latex.codecogs.com/png.latex?odds%20=%20%7B1%20%5Cover%20p%7D">.</p>
<section id="betting-on-the-next-coin-toss" class="level3">
<h3 class="anchored" data-anchor-id="betting-on-the-next-coin-toss">Betting on the next coin toss</h3>
<p>In the case of a uniform prior, the probability of the next coin toss leading to Heads is <img src="https://latex.codecogs.com/png.latex?0.8">, hence odds of <img src="https://latex.codecogs.com/png.latex?%7B5%20%5Cover%204%7D%20=%201.25">.</p>
<p>However, with a <img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7BBeta%7D"> prior, we have a probability of <img src="https://latex.codecogs.com/png.latex?0.68">, hence odds of <img src="https://latex.codecogs.com/png.latex?1.47">. Way higher, for the same set of observations !</p>
<p>So, which is the right odds ? <strong>Neither</strong>. Because there are many scenarios, that is many possible biases, that could lead to the sequence <em>Heads - Heads - Heads</em> as the first three tosses. As Michael Betancourt would say, <em>embracing uncertainty with probability</em> is what makes us bayesian. Hence, not focusing on the mean of the posterior distribution of <code>p_heads</code> but rather checking the whole extent of the density. This gives us plenty of information, from which we can draw credible intervals if we want.</p>
<p>As Georges Box famously said:</p>
<blockquote class="blockquote">
<p>All models are wrong, but some are useful.</p>
</blockquote>
<p>We could even argue that the <em>true</em> probability of an event doesn’t exist, it is merely a way to measure our incomplete information about the world in a specific context. In the real world, there is no probability, only events that happen or don’t happen. But what about quantum physics ? Well, we don’t deal with such a small scale here so I argue it’s irrelevant. We got a bit sidetracked, and to be honest this stuff is over my head.</p>
<p>Note: <a href="../../posts/betting-theory#fairness">I wrote a bit more about how to translate probabilities to bets</a> if you are interested in this stuff.</p>
</section>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>If by now you are not at least a little bit Bayesian, you must be deep into the frequentist cult! Joke aside, I hope that the mysteries of bayesian inference are now a bit less obscure, and that you will produce lots of great models with probabilistic programming.</p>
<hr>
<p><em>Many thanks to Théo Stoskopf, Théo Dumont, Haixuan Xavier Tao, Ai-Jun Tai for their precious feedback on early drafts of this post</em></p>


</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>See e.g.&nbsp;<a href="https://github.com/aleximmer/Laplace">https://github.com/aleximmer/Laplace</a>↩︎</p></li>
</ol>
</section><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2024,
  author = {Guy, Horace},
  title = {An Introduction to Bayesian Inference},
  date = {2024-04-24},
  url = {https://blog.horaceg.xyz/posts/bayes/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2024" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2024. <span>“An Introduction to Bayesian Inference.”</span>
April 24. <a href="https://blog.horaceg.xyz/posts/bayes/">https://blog.horaceg.xyz/posts/bayes/</a>.
</div></div></section></div> ]]></description>
  <category>probability</category>
  <category>bayes</category>
  <category>inference</category>
  <category>statistics</category>
  <guid>https://blog.horaceg.xyz/posts/bayes/</guid>
  <pubDate>Wed, 24 Apr 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Understanding Variational Auto-Encoders (VAE)</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/vae/vae.html</link>
  <description><![CDATA[ 





<p>Disclaimer: this post is heavily based on <em>Introduction to variational auto-encoders, Kingma &amp; Welling, 2019</em>[^intro-vae]</p>
<section id="introduction" class="level1">
<h1>Introduction</h1>
<p>As of April 2024, <a href="https://arxiv.org/abs/1312.6114">the VAE Paper</a> from 2013 has been cited more than 34 thousand times. What can we learn from the variational auto-encoder ?</p>
<p>The framework of VAE provides a principled method for jointly learning <em>deep latent-variable models</em> and corresponding inference models using stochastic gradient descent.</p>
<p>It has a wide array of application: - generative modelling - semi-supervised learning - representation learning</p>
<p>The VAE can be viewed as two coupled, but independently parameterized models: - the <em>encoder</em> or <em>recognition model</em> - the <em>decoder</em> or <em>generative model</em></p>
<p>The recognition model is the approximate inverse of the generative model according to Bayes rule.</p>
<p>The idea of training an inference network to “invert” a generative network, rather than running an optimization algorithm to infer the latent code, is called <em>amortized inference</em>.</p>
<section id="vae-vs-vanilla-ae" class="level2">
<h2 class="anchored" data-anchor-id="vae-vs-vanilla-ae">VAE vs vanilla AE</h2>
<p>The main advantage of a VAE over a deterministic autoencoder is that it defines a proper generative model that can create sensible-looking novel images by decoding prior samples <img src="https://latex.codecogs.com/png.latex?z%20%5Csim%20%5Cmathcal%7BN%7D(0,%20I)">. By contrast, an autoencoder only knows how to decode latent codes derived from the training set, so does poorly when fed random inputs.</p>
<p>The reason behind this is that the VAE embeds images into Gaussians in latent space, whereas the AE embeds images into points, which are like delta functions. The advantage of using a latent <em>distribution</em> is that it encourages <strong>local smoothness</strong> since a given image may map to multiple nearby places, depending on the stochastic sampling. By contrast, in an AE the latent space is <em>not smooth</em>, so images from different classes often en up next to each other.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: center;"><img src="https://blog.horaceg.xyz/posts/vae/vae-vs-ae-tsne-mnist.png" class="img-fluid" alt="tSNE projection of a 20-dimensional latent space for VAE and AE"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">tSNE projection of a 20-dimensional latent space for MNIST with (a) VAE (b) deterministic AE</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="key-ideas" class="level1">
<h1>Key ideas</h1>
<p>3 key ideas according to Bishop [^deep-bishop]:</p>
<ul>
<li><em>Evidence Lower Bound (ELBO)</em> to approximate likelihood function (leads to a close relationship to EM algorithm)</li>
<li><em>Amortized inference</em> in which a second model, the encoder network, is used to approximate the posterior distribution over latent variables in the E step, rather than evaluating the posterior distribution for each data point exactly</li>
<li>Making the training of the encoder model tractable using the <em>reparameterization trick</em></li>
</ul>
</section>
<section id="concept" class="level1">
<h1>Concept</h1>
<p><strong>Inference model / recognition model / encoder</strong>: <img src="https://latex.codecogs.com/png.latex?q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%7C%5Cmathbf%7Bx%7D)"> where <img src="https://latex.codecogs.com/png.latex?%5Cphi"> are the <em>variational parameters</em>.</p>
<p>We optimize these parameters such that:</p>
<p><img src="https://latex.codecogs.com/png.latex?q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%7C%5Cmathbf%7Bx%7D)%20%5Capprox%20p_%7B%5Cmathbf%7B%5Ctheta%7D%7D(%5Cmathbf%7Bz%7D%7C%5Cmathbf%7Bx%7D)"></p>
<p>the distribution <img src="https://latex.codecogs.com/png.latex?q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%7C%5Cmathbf%7Bx%7D)"> can be parameterized using deep neural networks. In this case, the variational parameters <img src="https://latex.codecogs.com/png.latex?%5Cphi"> include the weights and biases of the neural network. For example:</p>
<p><img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7B%5Cmu%7D,%20%5Clog%20%5Cmathbf%7B%5Csigma%7D)%20=%20%5Cmathrm%7BEncoderNeuralNet%7D_%7B%5Cphi%7D(%5Cmathbf%7Bx%7D)"></p>
<p><img src="https://latex.codecogs.com/png.latex?q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)%20=%20%5Cmathcal%7BN%7D(%5Cmathbf%7Bz%7D;%20%5Cmathbf%7B%5Cmu%7D,%20%5Cmathrm%7Bdiag%7D(%5Csigma))"></p>
<table class="caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: center;">The VAE learns stochastic mappings between observed x-space, whose empirical distribution is complicated, and a latent z-space whose distribution is simple (here spherical)</td>
</tr>
</tbody>
</table>
<section id="evidence-lower-bound-elbo" class="level2">
<h2 class="anchored" data-anchor-id="evidence-lower-bound-elbo">Evidence Lower Bound (ELBO)</h2>
<p><strong>Evidence lower bound / Variational lower bound</strong>.</p>
<p>We define the <img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7BELBO%7D"> as:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cmathcal%7BL_%7B%5Ctheta,%5Cphi%7D%7D(%5Cmathbf%7Bx%7D)%20=%20%5Cmathbb%7BE_%7Bq_%7B%5Cphi%7D(%5Cmathbf%7Bz%7D%7C%5Cmathbf%7Bx%7D)%7D%7D%5B%5Clog%20p_%7B%5Ctheta%7D(%5Cmathbf%7Bx%7D,%20%5Cmathbf%7Bz%7D)%20-%20%5Clog%20q_%7B%5Cphi%7D(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)%5D"></p>
<p>It’s possible to prove the following <em>key equation</em>:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Clog%20p_%7B%5Ctheta%7D(%5Cmathbf%7Bx%7D)%20=%20%5Cmathcal%7BL_%7B%5Ctheta,%5Cphi%7D%7D(%5Cmathbf%7Bx%7D)%20+%20D_%7BKL%7D(q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)%20%7C%7C%20p_%7B%5Cmathbf%7B%5Ctheta%7D%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D))"></p>
<p>The point is that the KL-divergence is always non-negative, so the ELBO is a lower bound on the log-likelihood of the data (marginal log-likelihood).</p>
<section id="two-for-one" class="level3">
<h3 class="anchored" data-anchor-id="two-for-one">Two for One</h3>
<p>Maximizing the ELBO will have two effects: - maximize the marginal likelihood <img src="https://latex.codecogs.com/png.latex?p_%7B%5Ctheta%7D(x)"> =&gt; the generative model will become better - It will minimize the KL divergence of the approximation <img src="https://latex.codecogs.com/png.latex?q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)"> from the true posterior <img src="https://latex.codecogs.com/png.latex?p_%7B%5Ctheta%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)">, so the approximation becomes better.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/vae/vae-tiger.png" class="img-fluid figure-img"></p>
<figcaption>Simple schematic of computational flow in a variational autoencoder.</figcaption>
</figure>
</div>
</section>
</section>
<section id="stochastic-gradient-based-optimization-of-the-elbo" class="level2">
<h2 class="anchored" data-anchor-id="stochastic-gradient-based-optimization-of-the-elbo">Stochastic gradient-based optimization of the ELBO</h2>
<p>The individual-datapoint ELBO, and its gradient is, in general, intractable. However, good unbiased estimators exist, as we will show, such that we can still perform minibatch SGD:</p>
<ul>
<li>Unbiased gradients of the ELBO with respect to (w.r.t.) the generative model parameters <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> are simple to obtain, with a Monte Carlo estimator.</li>
<li>However, unbiased gradients w.r.t. the <em>variational</em> paramters <img src="https://latex.codecogs.com/png.latex?%5Cphi"> are more difficult to obtain because the expectation is taken w.r.t. to a function of <img src="https://latex.codecogs.com/png.latex?%5Cphi">.</li>
</ul>
<p>For continuous latent variables, we can use a reparameterization trick to compute unbiased estimates of the double gradient.</p>
</section>
<section id="reparameterization-trick" class="level2">
<h2 class="anchored" data-anchor-id="reparameterization-trick">Reparameterization Trick</h2>
<p>We express the random variable <img src="https://latex.codecogs.com/png.latex?z%20%5Csim%20q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)"> as some differentiable and invertible transformation of another r.v. <img src="https://latex.codecogs.com/png.latex?%5Cepsilon">, given <img src="https://latex.codecogs.com/png.latex?x"> and <img src="https://latex.codecogs.com/png.latex?%5Cphi">:</p>
<p><img src="https://latex.codecogs.com/png.latex?z%20=%20g(%5Cepsilon,%20%5Cphi,%20%5Cmathbf%7Bx%7D)"></p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cepsilon%20%5Csim%20p(%5Cepsilon)"> is a noise sample.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: center;"><img src="https://blog.horaceg.xyz/posts/vae/vae-reparam.png" class="img-fluid" alt="Reparameterization trick"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Reparameterization trick to backpropagate gradients</td>
</tr>
</tbody>
</table>
<p>With the reparam trick, we can rewrite the ELBO as:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cmathcal%7BL_%7B%5Ctheta,%5Cphi%7D%7D(%5Cmathbf%7Bx%7D)%20=%20%5Cmathbb%7BE_%7Bp(%5Cepsilon)%7D%7D%5B%5Clog%20p_%7B%5Ctheta%7D(%5Cmathbf%7Bx%7D,%20%5Cmathbf%7Bz%7D)%20-%20%5Clog%20q_%7B%5Cphi%7D(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)%5D"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?z%20=%20g(%5Cepsilon,%20%5Cphi,%20%5Cmathbf%7Bx%7D)">.</p>
<p>We can then form a simple Monte-Carlo estimator <img src="https://latex.codecogs.com/png.latex?%5Ctilde%7B%5Cmathcal%7BL_%7B%5Ctheta,%20%5Cphi%7D%7D%7D(x)"> of the individual-datapoint ELBO where we use a single noise sample <img src="https://latex.codecogs.com/png.latex?%5Cepsilon"> from <img src="https://latex.codecogs.com/png.latex?p(%5Cepsilon)">:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cepsilon%20%5Csim%20p(%5Cepsilon)"> <img src="https://latex.codecogs.com/png.latex?z%20=%20g(%5Cphi,%20x,%20%5Cepsilon)"> <img src="https://latex.codecogs.com/png.latex?%5Ctilde%7B%5Cmathcal%7BL_%7B%5Ctheta,%20%5Cphi%7D%7D%7D(x)%20=%20%5Clog%20p_%7B%5Ctheta%7D(x,%20z)%20-%20%5Clog%20q_%7B%5Cphi%7D(z%7Cx)"></p>
<p>The two terms above correspond to the generative model (decoder) and the recognition model (encoder).</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: center;"><img src="https://blog.horaceg.xyz/posts/vae/vae-algo-elbo.png" class="img-fluid" alt="VAE Algorithm"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">The VAE algorithm - stochastic optimization of the ELBO</td>
</tr>
</tbody>
</table>
<section id="computing-log-q_phizx" class="level3">
<h3 class="anchored" data-anchor-id="computing-log-q_phizx">Computing <img src="https://latex.codecogs.com/png.latex?%5Clog%20q_%7B%5Cphi%7D(z%7Cx)"></h3>
<p>Computation of the estimator of the ELBO requires computation of the density $ q_{}(z|x)$, given a value of <img src="https://latex.codecogs.com/png.latex?x"> and given a value of <img src="https://latex.codecogs.com/png.latex?z"> or equivalently <img src="https://latex.codecogs.com/png.latex?%5Cepsilon">.</p>
<p>This log-density is a simple computation, as long as we choose the right transformation <img src="https://latex.codecogs.com/png.latex?g">.</p>
<p>As long as <img src="https://latex.codecogs.com/png.latex?g(.)"> is an invertible function, the densities of <img src="https://latex.codecogs.com/png.latex?%5Cepsilon"> and <img src="https://latex.codecogs.com/png.latex?z"> are related as:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Clog%20q_%7B%5Cphi%7D%20(z%7Cx)%20=%20%5Clog%20p(%5Cepsilon)%20-%20%5Clog%20d_%7B%5Cphi%7D(x,%20%5Cepsilon)"></p>
<p>Where the second term is the log of the absolute value of the determinant of the jacobian matrix <img src="https://latex.codecogs.com/png.latex?%5Cpartial%20z%20%5Cover%20%5Cpartial%20%5Cepsilon">.</p>
<p>This is the <em>log-determinant</em> of the transformation from <img src="https://latex.codecogs.com/png.latex?%5Cepsilon"> to <img src="https://latex.codecogs.com/png.latex?z">.</p>
<p>We will see that we can build very flexible transformations <img src="https://latex.codecogs.com/png.latex?g(.)"> for which the log-determinant is simple to compute, resulting in highly flexible inference models <img src="https://latex.codecogs.com/png.latex?q_%7B%5Cphi%7D(z%7Cx)">.</p>
</section>
</section>
<section id="factorized-gaussian-posteriors" class="level2">
<h2 class="anchored" data-anchor-id="factorized-gaussian-posteriors">Factorized Gaussian posteriors</h2>
<p><img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7B%5Cmu%7D,%20%5Clog%20%5Cmathbf%7B%5Csigma%7D)%20=%20%5Cmathrm%7BEncoderNeuralNet%7D_%7B%5Cphi%7D(%5Cmathbf%7Bx%7D)"></p>
<p><img src="https://latex.codecogs.com/png.latex?q_%7B%5Cmathbf%7B%5Cphi%7D%7D%20(%5Cmathbf%7Bz%7D%20%7C%20%5Cmathbf%7Bx%7D)%20=%20%5Cprod_i%20q_%7B%5Cphi%7D(z_i%20%7C%20%5Cmathbf%7Bx%7D)%20=%20%5Cprod_i%20%5Cmathcal%7BN%7D(z_i;%20%5Cmu_i,%20%5Csigma_i%5E2)"></p>
<p>After reparameterization, we can write instead:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cepsilon%20%5Csim%20%5Cmathcal%7BN%7D(0,%20%5Cmathbf%7BI%7D)"></p>
<p><img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7B%5Cmu%7D,%20%5Clog%20%5Cmathbf%7B%5Csigma%7D)%20=%20%5Cmathrm%7BEncoderNeuralNet%7D_%7B%5Cphi%7D(%5Cmathbf%7Bx%7D)"></p>
<p><img src="https://latex.codecogs.com/png.latex?z%20=%20%5Cmu%20+%20%5Csigma%20%5Codot%20%5Cepsilon%20"></p>
<p>The jacobian of the transformation from <img src="https://latex.codecogs.com/png.latex?%5Cepsilon"> to <img src="https://latex.codecogs.com/png.latex?z"> is:</p>
<p><img src="https://latex.codecogs.com/png.latex?%7B%5Cpartial%20z%20%5Cover%20%5Cpartial%20%5Cepsilon%7D%20=%20%5Cmathrm%7Bdiag%7D(%5Csigma)"></p>
<p>Thus the log-determinant is</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Clog%20d_%7B%5Cphi%7D(x,%20%5Cepsilon)%20=%20%5Csum_i%20%5Clog%20%5Csigma_i"></p>
<p>And the posterior density when <img src="https://latex.codecogs.com/png.latex?z%20=%20g(%5Cepsilon,%20%5Cphi,%20x)"> is:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Clog%20q_%7B%5Cphi%7D%20(z%7Cx)%20=%20%5Csum_i%20%5Clog%20%5Cmathcal%7BN%7D(%5Cepsilon_i;%200,%201)%20-%20%5Clog%20%5Csigma_i"></p>
</section>
<section id="estimation-of-the-marginal-likelihood" class="level2">
<h2 class="anchored" data-anchor-id="estimation-of-the-marginal-likelihood">Estimation of the Marginal Likelihood</h2>
<p>After training a VAE, we can estimate the probability of data under the model using <em>importance sampling</em>.</p>
<p>Since the marginal likelihood of a datapoint is:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Clog%20p_%7B%5Ctheta%7D(x)%20=%20%5Clog%20%5Cmathbb%7BE_%7Bq_%7B%5Cphi%7D(z%7Cx)%7D%7D%5B%7Bp_%7B%5Ctheta%7D(x,%20z)%20%5Cover%20q_%7B%5Cphi%7D(z%7Cx)%7D%5D"></p>
<p>Taking random samples <img src="https://latex.codecogs.com/png.latex?z%5E%7B(l)%7D%20%5Csim%20q_%7B%5Cphi%7D(z%7Cx)">, a Monte-Carlo estimator of this is:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Clog%20p_%7B%5Ctheta%7D(x)%20%5Capprox%20%5Clog%20%7B1%20%5Cover%20L%7D%20%5Csum_%7Bl=1%7D%5EL%20%7Bp_%7B%5Ctheta%7D%20(x,%20z%5E%7B(l)%7D)%20%5Cover%20q_%7B%5Cphi%7D(z%5E%7B(l)%7D%7Cx)%7D"></p>
<p>With a large <img src="https://latex.codecogs.com/png.latex?L">, the approximation becomes a better estimate of the marginal likelihood.</p>
</section>
<section id="follow-up-work" class="level2">
<h2 class="anchored" data-anchor-id="follow-up-work">Follow-up work</h2>
<p>Here are some important applications for deep generative models and VAE:</p>
<ul>
<li>Representation learning
<ul>
<li>data-efficient learning, such as semi-supervised learning</li>
<li>visualisation of data as low-dimensional manifolds</li>
</ul></li>
<li>Artificial creativity: plausible interpolation between data and extrapolation from data</li>
</ul>
<section id="representation-learning" class="level3">
<h3 class="anchored" data-anchor-id="representation-learning">Representation learning</h3>
<p>For supervised learning tasks, we want to learn a conditional distribution.: to predict the distribution over the possible values of a variable given the value of some other variable. One example is image classification. For this task, deep neural nets such as CNNs are very good with large amounts of data.</p>
<p>However, when the number of labaled examples is low, solutions with purely supervised approaches exhibit poor generalization to new data. In this case, generative models can be used as an effective type of regularization. One strategy is to optimize the classification model jointly with a VAE over the input variables, sharing parameters between the two. This procedure improves the data efficiency of the classification solution.</p>
<p>Example: &lt; 1% classification error on MNIST when trained with only 10 labeled images per class, i.e.&nbsp;99.8% of the labels in the training set removed. Some works even show that VAE-based semi-supervised learning can even do well when only a single sample per class is presented.</p>
</section>
<section id="understanding-of-data-and-artificial-creativity" class="level3">
<h3 class="anchored" data-anchor-id="understanding-of-data-and-artificial-creativity">Understanding of data and artificial creativity</h3>
<ul>
<li>Chemical design: learn a continuous representation (laten space) of discrete structures of molecules to do gradient-based optimization towards certain properties.</li>
</ul>
<table class="caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: center;">Chemical design - gradient-based optimization on latent space</td>
</tr>
</tbody>
</table>
<ul>
<li>Natural language synthesis: interpolate between sentences, imputation of missing words.</li>
<li>Image (re-)synthesis: modification of an image in latent space along a <em>smile vector</em></li>
</ul>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: center;"><img src="https://blog.horaceg.xyz/posts/vae/vae-smile-vector.png" class="img-fluid" alt="VAE Smile vector"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Modifications of an image along a <em>smile vector</em></td>
</tr>
</tbody>
</table>
</section>
</section>
</section>
<section id="conclusion" class="level1">
<h1>Conclusion</h1>
<p>Parameterizing the conditional distributions of directed probabilistic models with differentiable deep neural networks can make them very flexible.</p>
<p>The main contribution of VAEs is a framework for efficient and scalable gradient-based variational posterior inference and approximate maximum likelihood learning.</p>
</section>
<section id="references" class="level1">
<h1>References</h1>
<p>[^intro-vae] <a href="https://arxiv.org/abs/1906.02691">An Introduction to Variational Autoencoders, Kingma &amp; Welling (original VAE authors), 2019</a></p>
<p>[^deep-bishop] <a href="https://www.bishopbook.com/">Deep Learning - Foundations and concepts, Bishop &amp; Bishop, 2023, chapter 19.2 page 569 (slider 581)</a></p>
<p><a href="https://probml.github.io/pml-book/book1.html">Probabilistic Machine Learning: An introduction, Kevin P. Murphy, 2022</a></p>
<p><a href="https://probml.github.io/pml-book/book2.html">Probabilistic Machine Learning: Advanced Topics, Kevin P. Murphy, 2023</a></p>
<p><a href="https://sebastiancallh.github.io/post/vae-anatomy/">Sebastian Callh’s blog post</a></p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2024,
  author = {Guy, Horace},
  title = {Understanding {Variational} {Auto-Encoders} {(VAE)}},
  date = {2024-04-04},
  url = {https://blog.horaceg.xyz/posts/vae/vae.html},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2024" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2024. <span>“Understanding Variational Auto-Encoders
(VAE).”</span> April 4. <a href="https://blog.horaceg.xyz/posts/vae/vae.html">https://blog.horaceg.xyz/posts/vae/vae.html</a>.
</div></div></section></div> ]]></description>
  <category>machine learning</category>
  <category>probabilistic programming</category>
  <guid>https://blog.horaceg.xyz/posts/vae/vae.html</guid>
  <pubDate>Thu, 04 Apr 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Crafting compilers</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/compiler-1/</link>
  <description><![CDATA[ 





<p>TL; DR: My own <a href="https://github.com/horaceg/cse131">code is on github</a>.</p>
<section id="an-attempt-at-crafting-interpreters" class="level3">
<h3 class="anchored" data-anchor-id="an-attempt-at-crafting-interpreters">An attempt at crafting interpreters</h3>
<p>Due to the warm reviews, I tried first to read <em>Crafting Interpreters</em> and went though the first chapters, but I was quickly underwhelmed. Indeed I found the following:</p>
<ul>
<li>The language of the first part is Java, which I am not proficient in. I still managed to roughly translate the scanning part to python, but I am not familiar with Java so it adds a barrier for me.</li>
<li>The imperative, stateful style is hard to wrap one’s head around.</li>
<li>In order to have a working interpreter you have to go through the entire first part.</li>
<li>I want to learn assembly, and this book doesn’t have any.</li>
</ul>
<p>All in all, after the Scanning part I was unsatisfied and looked for something else.</p>
</section>
<section id="compiler-construction" class="level3">
<h3 class="anchored" data-anchor-id="compiler-construction"><em>Compiler Construction</em></h3>
<p>So I pivoted and instead I went with University of California San Diego’s <em>Compiler Construction</em> <a href="https://ucsd-cse131-f19.github.io/">course (codename CSE 131) of fall 2019</a>, taught by Joe Gibbs Politz. Incidentally, he is also part of the <a href="https://www.pyret.org/crew/">Pyret language crew</a>.</p>
<p>It checks almost all of my (digital) boxes:</p>
<ul>
<li>The code is in OCaml, which I know a bit</li>
<li>It goes all the way to Assembly</li>
<li>References the classic <em>Modern compiler implementation in ML</em> by Andrew W. Appel (also known as the Tiger book), which is on my to-do list after this course</li>
<li>The approach is incremental: at the end of each assignment, I have a fully functional compiler !</li>
</ul>
<p>The only minor drawback is that I am more interested in ARM than x86_64, because from what I’ve read x86 is messier than ARM.</p>
<p>I find it rewarding since now, after the first assignment, I have an end-to-end compiler for a calculator with variable. Of course, this is not a full-fledged language but I’m going to incrementally add features.</p>
<p>I just wish it were a book because I prefer this format to video lectures.</p>
</section>
<section id="modern-ocaml" class="level3">
<h3 class="anchored" data-anchor-id="modern-ocaml">Modern OCaml</h3>
<p>The course leverages basic features of the <code>OCaml</code> programming language and relies on makefiles for the build system. I tried to follow the style but unfortunately I didn’t get satisfactory intellisense in vscode without <code>dune</code>, the newish <code>OCaml</code> build system. So I converted everything to use <code>dune</code>.<br>
Also, for unit test the old <code>OUnit</code> is used, so I changed to leverage the more user-friendly and colorful <code>Alcotest</code>.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: center;"><img src="https://blog.horaceg.xyz/posts/compiler-1/alcotest.png" class="img-fluid" alt="terminal output from Alcotest"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;"><em>Look at this colorful and explicit output !</em></td>
</tr>
</tbody>
</table>
<section id="project-structure" class="level4">
<h4 class="anchored" data-anchor-id="project-structure">Project structure</h4>
<p>The structure is the following:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">~/dev/cse131/pa1</span> main<span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span></span>
<span id="cb1-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">❯</span> tree <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-C</span></span>
<span id="cb1-3"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">.</span></span>
<span id="cb1-4"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> bin</span>
<span id="cb1-5"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── dune</span>
<span id="cb1-6"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── main.c</span>
<span id="cb1-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   └── main.ml</span>
<span id="cb1-8"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> dune-project</span>
<span id="cb1-9"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> input</span>
<span id="cb1-10"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── 42.ana</span>
<span id="cb1-11"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── addnums.ana</span>
<span id="cb1-12"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── letdup.ana</span>
<span id="cb1-13"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── letdup_valid.ana</span>
<span id="cb1-14"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── letlet.ana</span>
<span id="cb1-15"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   └── sub1.ana</span>
<span id="cb1-16"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> lib</span>
<span id="cb1-17"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── asm.ml</span>
<span id="cb1-18"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── compile.ml</span>
<span id="cb1-19"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── compile_simple.ml</span>
<span id="cb1-20"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── dune</span>
<span id="cb1-21"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── expr.ml</span>
<span id="cb1-22"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   ├── parser.ml</span>
<span id="cb1-23"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">│</span>   └── runner.ml</span>
<span id="cb1-24"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> Makefile</span>
<span id="cb1-25"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> output</span>
<span id="cb1-26"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> pa1.opam</span>
<span id="cb1-27"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> README.md</span>
<span id="cb1-28"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">└──</span> test</span>
<span id="cb1-29">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> dune</span>
<span id="cb1-30">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">├──</span> myTests.ml</span>
<span id="cb1-31">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">└──</span> test_pa1.ml</span>
<span id="cb1-32"></span>
<span id="cb1-33"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">5</span> directories, 23 files</span></code></pre></div></div>
<p>My own <a href="https://github.com/horaceg/cse131">code is on github</a>.</p>
</section>
</section>
<section id="wrap-up" class="level3">
<h3 class="anchored" data-anchor-id="wrap-up">Wrap-up</h3>
<p>[edit: January 14 2024]</p>
<p>I stopped after the second assignment because the lectures started to get hairy and I got the feeling that everything wasn’t recorded. And also because of other constraints (life). Still, the mini-compiler should be functional.</p>
<p>All-in-all, I enjoyed working on this project.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2022,
  author = {Guy, Horace},
  title = {Crafting Compilers},
  date = {2022-10-23},
  url = {https://blog.horaceg.xyz/posts/compiler-1/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2022" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2022. <span>“Crafting Compilers.”</span> October 23. <a href="https://blog.horaceg.xyz/posts/compiler-1/">https://blog.horaceg.xyz/posts/compiler-1/</a>.
</div></div></section></div> ]]></description>
  <category>programming languages</category>
  <category>compiler</category>
  <category>assembly</category>
  <category>functional</category>
  <category>OCaml</category>
  <category>programming</category>
  <guid>https://blog.horaceg.xyz/posts/compiler-1/</guid>
  <pubDate>Sun, 23 Oct 2022 00:00:00 GMT</pubDate>
</item>
<item>
  <title>A probabilistic programming language from the future</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/ppl.html</link>
  <description><![CDATA[ 





<p>Lately I’ve been interested in probabilistic programming languages (PPL). I was wondering what would a PPL from the future look like ?</p>
<section id="state-of-the-art" class="level1">
<h1>State of the art</h1>
<section id="probabilistic-programming" class="level2">
<h2 class="anchored" data-anchor-id="probabilistic-programming">Probabilistic programming</h2>
<p>In a nutshell, probabilistic programming is about making computer reason about statistical bayesian inference. It refers to a domain-specific language, embedded in a host language or not, where the user declares a model of some kind, and plug-and-play samplers.</p>
<p>This ia called a probabilistic programming language, or <strong>PPL</strong>. The two most popular ones are <code>Stan</code> and <code>PyMC</code>.</p>
<p>The model is usually <em>generative</em>, i.e.&nbsp;a data-generating process that allows of the build a <em>log-probability</em>, that is a <em>log-likelihood</em> plus a <em>log-prior</em> (thanks to Bayes’ Theorem).</p>
<p>The <em>sampler</em> is an inference algorithm, whose goal is to find the <em>posterior distribution</em> of model parameters. It often belongs to the <strong>Markov Chain Monte Carlo (MCMC)</strong> family: Metropolis-Hastings, Gibbs, Hamiltonian Monte-Carlo, No-U-Turn Sampler etc.</p>
<p>So the <strong>model definition</strong> procedure is like this:</p>
<ul>
<li>Define <em>priors</em> for each model parameter. This is an a priori belief you have, such as <em>the <img src="https://latex.codecogs.com/png.latex?%5Calpha"> parameter is roughly between 5 and 10</em>, or <em>the <img src="https://latex.codecogs.com/png.latex?b"> parameters is greater than 0</em>, or <em>the <img src="https://latex.codecogs.com/png.latex?r"> parameter is between 0 and 1</em>.</li>
<li>Combine parameters with operations such as addition, multiplication, division, <img src="https://latex.codecogs.com/png.latex?%5Cexp">, <img src="https://latex.codecogs.com/png.latex?%5Clog">, <img src="https://latex.codecogs.com/png.latex?%5Csqrt."> etc. to generate a <em>synthetic observation</em>, or <em>prediction</em>.</li>
</ul>
<p>The <strong>inference</strong> of the <em>sampler</em> consists in updating the distribution of parameters towards a value that <em>minimizes</em> the difference between observations and prediction. The MCMC algorithm is fundamentally sequential. Indeed, it’s a Markov Chain so each state depends on the previous one. Which means it’s not embarissingly parallel.</p>
<p>However, often scientist launch <em>multiple chains</em> in parallel, one on each CPU core, in order to assess the converge of the algorithm and improve the posterior sampling.</p>
<!-- A nice property of Bayes' rule is that when you want to update your beliefs twice, you can do it once and then again with the updated belief as the new prior. -->
<section id="the-hamiltonian-monte-carlo-algorithm" class="level3">
<h3 class="anchored" data-anchor-id="the-hamiltonian-monte-carlo-algorithm">The Hamiltonian Monte-Carlo algorithm</h3>
<p>While MCMC has been increasingly popular in scientific domains for the past 50 years, a newish (2011) variant is all the rage right now: HMC for Hamiltonian Monte-Carlo, and its offspring <strong>NUTS</strong>. These are the workhorses of <em>Stan</em> or <em>PyMC</em>.</p>
<p>These algorithms are very efficient, although they require the <strong>gradient</strong> of the log-probability.</p>
</section>
</section>
<section id="automatic-differentiation" class="level2">
<h2 class="anchored" data-anchor-id="automatic-differentiation">Automatic differentiation</h2>
<p>To achieve this while having total flexibility over the model definition, one can leverage <strong>automatic differentiation</strong>. This means being able to <em>efficiently</em> compute the gradient of a function programmed in the PPL.</p>
<p>The most well-known automatic differentiation engines are included in tensor computation (and often deep learning) libraries. In Python, examples are Tensorflow, PyTorch, JAX, Aesara etc.</p>
<section id="enzyme" class="level3">
<h3 class="anchored" data-anchor-id="enzyme">Enzyme</h3>
<p>Enter Enzyme. It differentiates directly the LLVM intermediate representation (IR). This means that one can differentiate <em>any program</em> compiled with LLVM. This gives access to gradient of programs written in Fortran, C, C++, Rust, Julia, Swift…</p>
<p>Compared to python frameworks, this is very appealing: it is framework-agnostic ! Better, the programmer doesn’t even have to be aware that someone is going to differentiate their program ! Which means arbitrary program differentiation.</p>
</section>
</section>
<section id="compilers" class="level2">
<h2 class="anchored" data-anchor-id="compilers">Compilers</h2>
<p>To translate a program written in textual form, such as Stan-lang or Python, to machine code, one needs a compiler (or an interpreter, but we’ll say compiler here).</p>
<ul>
<li><p><code>Stanc</code>, the official compiler for the Stan language, transpiles to <code>C++</code>. In this low-level language, it leverages the Stan math library for automatic differentiation, linear algebra, ODE solver, probability distributions, etc. OpenCL is used to accelerate linear-algebra heavy workloads on general-purpose GPUs.</p></li>
<li><p><code>PyMC</code> uses the <code>Aesara</code> tensor library to define a computation graph, and perform symbolic and computational operations on it. As of now, Aesara has two backends: one with Numba, that compiles a subset of python Just-in-time with LLVM, and one with <code>JAX</code>, that compiles to <code>XLA</code> (<em>accelerated linear algebra</em>) just-in-time as well.</p></li>
<li><p><code>Numpyro</code> uses JAX directly as its tensor library.</p></li>
<li><p><code>Tensorflow-probability</code> targets XLA as well, piggybacking on tensorflow.</p></li>
</ul>
<p>XLA targets most CPUs, Nvidia GPUs (with CUDA), or TPUs. And when a model is compiled to XLA, it runs <em>very fast</em>. I’ve seen some of my predictions in <em>100 <img src="https://latex.codecogs.com/png.latex?%5Cmu">s</em> on CPU. Numba can be very fast as well on CPU.</p>
<p>However, in order to accelerate with XLA out of the CPU, one needs an Nvidia GPU. These are very expensive, power-hungry, and well it’s very monopolistic. Or a TPU, which is even worse since it’s impossible to own one.</p>
</section>
</section>
<section id="big-ideas" class="level1">
<h1>Big ideas</h1>
<p>In this section, I’m going to present the big ideas for a 2022+ scientific programming language such as a PPL.</p>
<section id="compiler" class="level2">
<h2 class="anchored" data-anchor-id="compiler">Compiler</h2>
<p>One can write a compiler in any language. However, some are a better fit for it. One classic language for compilers is <code>OCaml</code>. The ML language lends itself well to lex, tokenize, parse, and generate code. It is also well-known that the Ocaml compiler is very fast, and there is an OCaml bytecode to javascript compiler (<code>Js_of_OCaml</code>, or jsoo). So a compiler written in OCaml could run inside the browser.</p>
<p>Stan has recently (2020-ish ?) swapped its old C++ compiler to a brand new OCaml one: Stanc3.</p>
<p><code>Rust</code> is the other candidate. More fashionable, modern, is definitely lower-level. One could compile the compiler to WASM and run it inside a browser as well. This is also where the cool kids are, so lots of friendly libraries such as <code>chumsky</code> are just waiting for us to leverage them!</p>
<p>From <a href="https://matklad.github.io/2018/06/06/modern-parser-generator.html">https://matklad.github.io/2018/06/06/modern-parser-generator.html</a> :</p>
<blockquote class="blockquote">
<p>I think today it is the best language for writing compiler-like stuff (yes, better than OCaml!),</p>
</blockquote>
</section>
<section id="compiler-target" class="level2">
<h2 class="anchored" data-anchor-id="compiler-target">Compiler target</h2>
<p>Now, the question is: what kind of code would be generated by this compiler?</p>
<p>I think it could be nice to have two targets, loosely speaking: - CPU - GPU</p>
<p>Fortunately, web standards have emerged and we now have two perfectly suitable targets for both. Both of them are in the <em>write once, run anywhere</em> category, including the web!</p>
<section id="webassembly" class="level3">
<h3 class="anchored" data-anchor-id="webassembly">WebAssembly</h3>
<p>The main target is WebAssembly (WASM). The advantages is portability: run anywhere, web or native.</p>
<p>The compiler may leverage <code>binaryen</code> or just the <code>wasm-encode</code> Rust crate to compile the PPL to WASM. Then, <code>wastime</code> could be used to interpret or compile ahead-of-time (AOT) wasm to native code, or if needed the browser can directly execute the assembly file.</p>
<p>Now one conflicting requirement is that the open-source linear algebra libraries such as <code>BLAS</code> don’t currently support webassembly. However, the new hot BLAS-like framework <code>BLIS</code> can be compiled to wasm, see <a href="https://twitter.com/ethanhs/status/1381500487334162432">https://twitter.com/ethanhs/status/1381500487334162432</a></p>
<p>The slowdown compared to native code is a bit underwhelming though: &gt; - Native zen2 build (with SIMD kernels): 4.154655s &gt; - Native generic systemless build: 15.015203s &gt; - WASM generic systemless build: 70.516310s</p>
<p>So until someone creates an (SIMD) webassembly kernel for BLIS, it won’t be competitive with native.</p>
</section>
<section id="webgpu" class="level3">
<h3 class="anchored" data-anchor-id="webgpu">WebGPU</h3>
<p>One can write <em>shaders</em> in WGSL, the <strong>WebGPU</strong> language. It gives acess to general-purpose GPU (GPGPU) computing, on <em>any device</em>. This means my laptop’s integrated graphics card as well as the latest RTX 4090.</p>
<p>How to leverage the GPU in the case of a PPL?<br>
One classic possibility is to optimize for linear algebra, e.g.&nbsp;matric multiplication (GEMM). Another one would be to run thousands of somewhat short MCMC chains in parallel on the GPU.</p>
</section>
<section id="rust" class="level3">
<h3 class="anchored" data-anchor-id="rust">Rust</h3>
<p>This is where Rust must come into play. Indeed, Rust is the language with the best support for WebAssembly. And the best-in-class webgpu implementation, that also have native backends, is the <strong>wgpu-rs</strong> rust library.</p>
<p>Also, it must be very convenient to use a low-level language to manipulate… well, low-level code.</p>
<p>Now, in order to use HMC and NUTS, which are required today to be competitive with Stan or PyMC, one needs to take the gradients.</p>
</section>
</section>
<section id="automatic-differentiation-in-wasm" class="level2">
<h2 class="anchored" data-anchor-id="automatic-differentiation-in-wasm">Automatic differentiation in WASM</h2>
<p>Inspired by Enzyme and its autodiff on LLVM IR, we could imagine differentiating WASM code directly. I have no idea if this is possible, but why not?</p>
<p>The even more tricky part is to differentiate both the CPU (WASM) and GPU (WebGPU) backends.</p>
<p>A probably simpler task would be to design the language and the compiler such that it is easy to differentiate on e.g.&nbsp;the Compiler IR level, before the assembly code is actually generated.</p>
<p>A good reading of <a href="https://github.com/google-research/dex-lang/issues/454">https://github.com/google-research/dex-lang/issues/454</a> could help I guess.</p>
<p>The alternative is to target LLVM IR, that can emit WASM, and leverage Enzyme to compute the gradients. One would need to write a WebGPU LLVM backend for this.</p>
</section>
<section id="the-vision" class="level2">
<h2 class="anchored" data-anchor-id="the-vision">The vision</h2>
<p>So we have a PPL that runs either natively or entirely in the browser, on any CPU and any (i)GPU. The whole language is differentiable, which allows one to build efficient gradients and run fast algorithms requiring them such as NUTS. On any (i)GPU, it can run thousands of (short) chains in parallel, greatly improving the reliability of scientific analysis.</p>


</section>
</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2022,
  author = {Guy, Horace},
  title = {A Probabilistic Programming Language from the Future},
  date = {2022-10-08},
  url = {https://blog.horaceg.xyz/posts/ppl.html},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2022" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2022. <span>“A Probabilistic Programming Language from the
Future.”</span> October 8. <a href="https://blog.horaceg.xyz/posts/ppl.html">https://blog.horaceg.xyz/posts/ppl.html</a>.
</div></div></section></div> ]]></description>
  <category>probabilistic programming</category>
  <guid>https://blog.horaceg.xyz/posts/ppl.html</guid>
  <pubDate>Sat, 08 Oct 2022 00:00:00 GMT</pubDate>
</item>
<item>
  <title>The responsive markdown resume</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/resume/</link>
  <description><![CDATA[ 





<p>I decided to do my resume since I haven’t done it in a while. My old one was based on the latex template <em>moderncv</em>. But this time, I wanted to have one <em>for the web</em> that looks good on desktop and mobile, as well as paper (pdf).</p>
<p>TL;DR: <a href="https://resume.horaceg.xyz">resume.horaceg.xyz</a></p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/resume/header.png" class="img-fluid figure-img"></p>
<figcaption>The header of my resume</figcaption>
</figure>
</div>
<section id="inspiration" class="level2">
<h2 class="anchored" data-anchor-id="inspiration">Inspiration</h2>
<p>I saw a <a href="https://blog.chmd.fr/editing-a-cv-in-markdown-with-pandoc.html">blog post</a> by Christophe-Marie Duquesne that seemed to exactly fit the bill. It is even inspired by <em>moderncv</em> !</p>
<p>So I <del>copied</del> took insipiration from it and wrote my own experience. However, it didn’t display well on mobile since the left section overflows to the center one.</p>
<p>Since we are in 2022, mobile first and all, I modified it to look good on mobile as well as <em>desktop</em> and <em>pdf</em>.</p>
</section>
<section id="width-and-overflow" class="level2">
<h2 class="anchored" data-anchor-id="width-and-overflow">Width and overflow</h2>
<p>The first thing I had to modify was chaning the body <code>width</code> property to <code>max-width</code>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode css code-with-copy"><code class="sourceCode css"><span id="cb1-1">body {</span>
<span id="cb1-2">    ...</span>
<span id="cb1-3">    max-width<span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> 900px</span>
<span id="cb1-4">}</span></code></pre></div></div>
<p>Then, I had to manage the overflow of the left section on mobile:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode css code-with-copy"><code class="sourceCode css"><span id="cb2-1">dt {</span>
<span id="cb2-2">    ...</span>
<span id="cb2-3">    overflow-wrap<span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> break-word;</span>
<span id="cb2-4">}</span></code></pre></div></div>
<p>This fixed the first issue: now one could somewhat read it on mobile !</p>
</section>
<section id="collapsible-items" class="level2">
<h2 class="anchored" data-anchor-id="collapsible-items">Collapsible items</h2>
<p>Still, it was hardly readable since there’s a lot of text that spans many paragraphs for each item. So here I had the idea that I need to make the individual items collapsible.</p>
<p>Fortunately, there is exactly an HTML element to do this: <code>&lt;details&gt;</code> &amp; <code>&lt;summary&gt;</code> ! There is no specific markdown syntax to deal with these, so I had to put it as-is in the <code>.md</code> file (having tried and failed to write a custom pandoc filter).</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode markdown code-with-copy"><code class="sourceCode markdown"><span id="cb3-1">2019 - now</span>
<span id="cb3-2">:   <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">details</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> open</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">summary</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span>*Data scientist* in R&amp;D at **Deepki** (Paris, France)<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">summary</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb3-3">    </span>
<span id="cb3-4"><span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Data science improving energy efficiency of buildings.</span></span>
<span id="cb3-5"><span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ...</span></span>
<span id="cb3-6"><span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    &lt;/details&gt;</span></span></code></pre></div></div>
</section>
<section id="collapsed-on-mobile-by-default" class="level2">
<h2 class="anchored" data-anchor-id="collapsed-on-mobile-by-default">Collapsed on mobile by default</h2>
<p>This looks good, but there’s still an improvement possible. I want to show only the summary on mobile, and show the full, extended version on desktop.</p>
<p>This is where this bit of <code>javascript</code> helped:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode js code-with-copy"><code class="sourceCode javascript"><span id="cb4-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">window</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">addEventListener</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"load"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span> () {</span>
<span id="cb4-2">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">var</span> elements <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">document</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">getElementsByTagName</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"details"</span>)<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-3">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> (<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">let</span> e <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">of</span> elements) {</span>
<span id="cb4-4">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> (<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">window</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">innerWidth</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">500</span>) {</span>
<span id="cb4-5">      e<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">open</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">false</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-6">    } <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> {</span>
<span id="cb4-7">      e<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">open</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">true</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-8">    }</span>
<span id="cb4-9">  }</span>
<span id="cb4-10">})</span></code></pre></div></div>
<p>On the <code>load</code> event, this script toggles the <code>open</code> attribute of all the <code>details</code> elements based on the screen width.</p>
</section>
<section id="showing-a-preview-on-collapsed" class="level2">
<h2 class="anchored" data-anchor-id="showing-a-preview-on-collapsed">Showing a preview on collapsed</h2>
<p>There’s one thing left to be desired: since it does not look like a button, users won’t necessrily understand that collapsed sections are clickable. One could make it more obvious by styling the summary like a button, but I liked the clean simple design.</p>
<p>So I added this snippet of css:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode css code-with-copy"><code class="sourceCode css"><span id="cb5-1">summary<span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::after</span> {</span>
<span id="cb5-2">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">content</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\a</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">attr(</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">preview</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb5-3">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">white-space</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">pre</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb5-4">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">opacity</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb5-5">}</span>
<span id="cb5-6"></span>
<span id="cb5-7">details<span class="ex" style="color: null;
background-color: null;
font-style: inherit;">[</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">open</span><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> summary<span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::after</span> {</span>
<span id="cb5-8">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">content</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">none</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb5-9">}</span></code></pre></div></div>
<p>On collapsed, it shows the <code>preview</code> attribute of the <code>summary</code> element with half opacity, preceded by a newline <code>"\a"</code>.</p>
<p>Now it works but I still have to manually fill the preview attributes, which is boring and error-prone in case I change items.</p>
</section>
<section id="setting-the-element-automatically" class="level2">
<h2 class="anchored" data-anchor-id="setting-the-element-automatically">Setting the element automatically</h2>
<p><code>js, hl_lines=5-13 window.addEventListener("load", function () {   var elements = document.getElementsByTagName("details");   for (let e of elements) {       ...     e.children[0].setAttribute(       "preview",       e.outerHTML         .split("&lt;p&gt;")[1]         .split(" ")         .slice(0, 4)         .join(" ")         .replace("amp;", "") + "..."     );   } });</code></p>
<p>Now, the preview attribute of all details elements contains the first 4 words of the following section, plus “…” appended at the end.</p>
</section>
<section id="building-html-and-pdf-documents" class="level2">
<h2 class="anchored" data-anchor-id="building-html-and-pdf-documents">Building HTML and pdf documents</h2>
<p>To build in html, I need to include the javascript in the header of the html file. Since pandoc uses <code>wkhtmltopdf</code>, and the latter puts large margins everywhere by default, I need to specify the margins at zero for them to be handled by the css.</p>
<p>In <code>build.bash</code>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#! /bin/bash</span></span>
<span id="cb6-2"></span>
<span id="cb6-3"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">set</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-e</span></span>
<span id="cb6-4"></span>
<span id="cb6-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">date</span></span>
<span id="cb6-6"></span>
<span id="cb6-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">pandoc</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--from</span> markdown <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--to</span> html <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb6-8">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-H</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;(</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;script&gt;"</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> script.js<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">)</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;/script&gt;"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb6-9">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-c</span> style.css <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-o</span> resume.html resume.md</span>
<span id="cb6-10"></span>
<span id="cb6-11"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">pandoc</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--from</span> markdown <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--to</span> html <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb6-12">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-V</span> margin-top=0 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-V</span> margin-left=0 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-V</span> margin-right=0 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-V</span> margin-bottom=0 <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb6-13">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-V</span> papersize=letter <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb6-14">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-c</span> style.css <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-o</span> horace_guy.pdf resume.md</span></code></pre></div></div>
<p>When I edit the document, I use <code>entr</code> to watch for files in the folder and build on change. In <code>watch.bash</code>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#! /bin/bash</span></span>
<span id="cb7-2"></span>
<span id="cb7-3"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">set</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-e</span></span>
<span id="cb7-4"></span>
<span id="cb7-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{md</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">css</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">js}</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">entr</span> ./build.bash</span></code></pre></div></div>
<p>The dev workflow is now: <code>./watch.bash</code>.</p>
</section>
<section id="putting-it-online" class="level2">
<h2 class="anchored" data-anchor-id="putting-it-online">Putting it online</h2>
<p>Well, we need to put online an html file, a css file and a pdf file. I use <a href="https://sitejs.org/">sitejs</a> with <code>site push</code> to put it online on a $5/month Linode box.</p>
</section>
<section id="wrap-up" class="level2">
<h2 class="anchored" data-anchor-id="wrap-up">Wrap-up</h2>
<p>All in all, I am happy with this new resume. I need to focus more on the content now that I have the template. All input files are in the <a href="https://github.com/horaceg/resume">github repo</a></p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2022,
  author = {Guy, Horace},
  title = {The Responsive Markdown Resume},
  date = {2022-02-24},
  url = {https://blog.horaceg.xyz/posts/resume/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2022" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2022. <span>“The Responsive Markdown Resume.”</span>
February 24. <a href="https://blog.horaceg.xyz/posts/resume/">https://blog.horaceg.xyz/posts/resume/</a>.
</div></div></section></div> ]]></description>
  <category>HTML</category>
  <category>programming</category>
  <category>CSS</category>
  <category>javascript</category>
  <category>resume</category>
  <category>CV</category>
  <category>web</category>
  <guid>https://blog.horaceg.xyz/posts/resume/</guid>
  <pubDate>Thu, 24 Feb 2022 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Concurrency in Python with FastAPI</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/python-concurrency/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/python-concurrency/image.webp" class="img-fluid figure-img"></p>
<figcaption>A machine (server) handling multiple requests at once, illustrating concurrency. Include the Python logo prominently. Made with ChatGPT</figcaption>
</figure>
</div>
<p>I’ve struggled for a long time with concurrency and parallelism. Let’s dive in with the hot-cool-new <a href="https://asgi.readthedocs.io/en/latest/">ASGI</a> framework, <a href="https://fastapi.tiangolo.com/">FastAPI</a>. It is a concurrent framework, which means <code>asyncio</code>-friendly. Tiangolo, the author, claims that the performance is on par with Go and Node webservers. We’re going to see a glimpse of the reason (spoilers: concurrency).</p>
<p>First things first, let’s install FastAPI by following the <a href="https://fastapi.tiangolo.com/#installation">guide</a>.</p>
<section id="purely-io-bound-workloads" class="level2">
<h2 class="anchored" data-anchor-id="purely-io-bound-workloads">Purely IO-bound workloads</h2>
<p>We are going to simulate a pure IO operation, such as an waiting for a database to finish its operation. Let’s create the following <code>server.py</code> file:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## server.py</span></span>
<span id="cb1-2"></span>
<span id="cb1-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> time</span>
<span id="cb1-4"></span>
<span id="cb1-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> fastapi <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> FastAPI</span>
<span id="cb1-6"></span>
<span id="cb1-7">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> FastAPI()</span>
<span id="cb1-8"></span>
<span id="cb1-9"></span>
<span id="cb1-10"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@app.get</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/wait"</span>)</span>
<span id="cb1-11"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> wait():</span>
<span id="cb1-12">    duration <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.</span></span>
<span id="cb1-13">    time.sleep(duration)</span>
<span id="cb1-14">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"duration"</span>: duration}</span></code></pre></div></div>
<p>Run it with</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvicorn</span> server:app <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--reload</span></span></code></pre></div></div>
<p>You should see at <code>http://127.0.0.1:8000/wait</code> something like:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode json code-with-copy"><code class="sourceCode json"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"duration"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span></span></code></pre></div></div>
<p>Ok, it works. Now, let’s dive into the performance comparison. We could use <a href="https://en.wikipedia.org/wiki/ApacheBench">ApacheBench</a>, but here we are going to implement everything in python for the sake of clarity.</p>
<p>Let’s create a <code>client.py</code> file:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## client.py</span></span>
<span id="cb4-2"></span>
<span id="cb4-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> functools</span>
<span id="cb4-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> time</span>
<span id="cb4-5"></span>
<span id="cb4-6"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb4-7"></span>
<span id="cb4-8"></span>
<span id="cb4-9"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> timed(N, url, fn):</span>
<span id="cb4-10">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@functools.wraps</span>(fn)</span>
<span id="cb4-11">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> wrapper(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb4-12">        start <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> time.time()</span>
<span id="cb4-13">        res <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> fn(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs)</span>
<span id="cb4-14">        stop <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> time.time()</span>
<span id="cb4-15">        duration <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> stop <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> start</span>
<span id="cb4-16">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>N <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> duration<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:.2f}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> reqs / sec | </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>N<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> reqs | </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>url<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> | </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>fn<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb4-17">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> res</span>
<span id="cb4-18"></span>
<span id="cb4-19">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> wrapper</span>
<span id="cb4-20"></span>
<span id="cb4-21"></span>
<span id="cb4-22"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> get(url):</span>
<span id="cb4-23">    resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.get(url)</span>
<span id="cb4-24">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> resp.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span id="cb4-25">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> resp.json()</span>
<span id="cb4-26"></span>
<span id="cb4-27"></span>
<span id="cb4-28"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> sync_get_all(url, n):</span>
<span id="cb4-29">    l <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [get(url) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> _ <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">range</span>(n)]</span>
<span id="cb4-30">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> l</span>
<span id="cb4-31"></span>
<span id="cb4-32"></span>
<span id="cb4-33"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> run_bench(n, funcs, urls):</span>
<span id="cb4-34">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> url <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> urls:</span>
<span id="cb4-35">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> func <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> funcs:</span>
<span id="cb4-36">            timed(n, url, func)(url, n)</span>
<span id="cb4-37"></span>
<span id="cb4-38"></span>
<span id="cb4-39"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb4-40">    urls <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/wait"</span>]</span>
<span id="cb4-41">    funcs <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [sync_get_all]</span>
<span id="cb4-42">    run_bench(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, funcs, urls)</span></code></pre></div></div>
<p>Let’s run this:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">python</span> client.py</span></code></pre></div></div>
<pre><code>0.99 reqs / sec | 10 reqs | http://127.0.0.1:8000/wait | sync_get_all</code></pre>
<p>So far, we know that the overhead is sub-10 ms for ten requests, so less than 1ms per request. Cool!</p>
<section id="threadpools-client-side" class="level3">
<h3 class="anchored" data-anchor-id="threadpools-client-side">Threadpools client-side</h3>
<p>Now, we are going to simulate multiple simultaneous connections. This is usually a problem we want to have: the more users of our web API or app, the more simultaneous requests. The previous test wasn’t very realistic: users rarely browse sequentially, but rather appear simultaneously, forming bursts of activity.</p>
<p>We are going to implement concurrent requests using a <strong>threadpool</strong>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## client.py</span></span>
<span id="cb7-2"></span>
<span id="cb7-3">...</span>
<span id="cb7-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> concurrent.futures <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> ThreadPoolExecutor <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> Pool</span>
<span id="cb7-5">...</span>
<span id="cb7-6"></span>
<span id="cb7-7"></span>
<span id="cb7-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> thread_pool(url, n, limit<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>):</span>
<span id="cb7-9">    limit_ <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> limit <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">or</span> n</span>
<span id="cb7-10">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> Pool(max_workers<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>limit_) <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> pool:</span>
<span id="cb7-11">        result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> pool.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">map</span>(get, [url] <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> n)</span>
<span id="cb7-12">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> result</span>
<span id="cb7-13"></span>
<span id="cb7-14"></span>
<span id="cb7-15"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb7-16">    urls <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/wait"</span>]</span>
<span id="cb7-17">    run_bench(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, [sync_get_all, thread_pool], urls)</span></code></pre></div></div>
<p>We get:</p>
<pre><code>0.99 reqs / sec | 10 reqs | http://127.0.0.1:8000/wait | sync_get_all
9.56 reqs / sec | 10 reqs | http://127.0.0.1:8000/wait | thread_pool</code></pre>
<p>This looks 10x better! The overhead is of 44 ms for 10 requests, where does that come from?</p>
<p>Also, how come the server was able to answer asynchronously, since we only wrote synchronous (regular) Python code? There are no <code>async</code> nor <code>await</code>…</p>
<p>Well, this is how FastAPI works behind the scenes: it runs every synchronous request in a threadpool. So, we have threadpools both client-side and <strong>server-side</strong>!</p>
<p>Let’s lower the duration:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb9-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## server.py</span></span>
<span id="cb9-2"></span>
<span id="cb9-3">...</span>
<span id="cb9-4"></span>
<span id="cb9-5"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@app.get</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/wait"</span>)</span>
<span id="cb9-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> wait():</span>
<span id="cb9-7">    duration <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span></span>
<span id="cb9-8">    time.sleep(duration)</span>
<span id="cb9-9">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"duration"</span>: duration}</span></code></pre></div></div>
<p>Let’s also run the benchmark 100 times:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb10-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## client.py</span></span>
<span id="cb10-2"></span>
<span id="cb10-3">...</span>
<span id="cb10-4"></span>
<span id="cb10-5"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb10-6">    urls <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/wait"</span>]</span>
<span id="cb10-7">    run_bench(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, [sync_get_all, thread_pool], urls)</span></code></pre></div></div>
<pre><code>15.91 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | sync_get_all
196.06 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | thread_pool</code></pre>
<p>We can see there is some overhead on the server-side. Indeed, we should have <img src="https://latex.codecogs.com/png.latex?100%20/%200.05%20=%202000"> requests per second if everything worked without any friction.</p>
</section>
<section id="async-routes" class="level3">
<h3 class="anchored" data-anchor-id="async-routes"><code>async</code> routes</h3>
<p>There is another way to declare a route with FastAPI, using the <code>asyncio</code> keywords.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb12-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## server.py</span></span>
<span id="cb12-2"></span>
<span id="cb12-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> asyncio</span>
<span id="cb12-4">...</span>
<span id="cb12-5"></span>
<span id="cb12-6"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@app.get</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/asyncwait"</span>)</span>
<span id="cb12-7"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> asyncwait():</span>
<span id="cb12-8">    duration <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span></span>
<span id="cb12-9">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> asyncio.sleep(duration)</span>
<span id="cb12-10">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"duration"</span>: duration}</span></code></pre></div></div>
<p>Now just add this route to the client:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb13-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## client.py</span></span>
<span id="cb13-2"></span>
<span id="cb13-3"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb13-4">    urls <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/wait"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/asyncwait"</span>]</span>
<span id="cb13-5">    run_bench(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, [sync_get_all, thread_pool], urls)</span></code></pre></div></div>
<p>And run the benchmark:</p>
<pre><code>15.66 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | sync_get_all
195.41 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | thread_pool
15.52 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | sync_get_all
208.06 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | thread_pool</code></pre>
<p>We see a small improvement. But isn’t asyncio supposed to be very performant? And <a href="https://www.uvicorn.org/">Uvicorn</a> is based on <a href="https://github.com/MagicStack/uvloop">uvloop</a>, described as:</p>
<blockquote class="blockquote">
<p>Ultra fast asyncio event loop.</p>
</blockquote>
<p>Maybe the overhead comes from the client? Threadpools maybe?</p>
</section>
<section id="drinking-the-asyncio-kool-aid" class="level3">
<h3 class="anchored" data-anchor-id="drinking-the-asyncio-kool-aid">Drinking the <code>asyncio</code> kool-aid</h3>
<p>To check this, we’re going to implement a fully-asynchronous client. This is a bit more <em>involved</em>. Yes, this means <code>async</code>s and <code>await</code>s. I know you secretly enjoy these.</p>
<p>Just do <code>pip install aiohttp</code>, then:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb15-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## client.py</span></span>
<span id="cb15-2"></span>
<span id="cb15-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> asyncio</span>
<span id="cb15-4">...</span>
<span id="cb15-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> aiohttp</span>
<span id="cb15-6"></span>
<span id="cb15-7">...</span>
<span id="cb15-8"></span>
<span id="cb15-9"></span>
<span id="cb15-10"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> aget(session, url):</span>
<span id="cb15-11">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> session.get(url) <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> response:</span>
<span id="cb15-12">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> response.status <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span id="cb15-13">        json <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> response.json()</span>
<span id="cb15-14">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> json</span>
<span id="cb15-15"></span>
<span id="cb15-16"></span>
<span id="cb15-17"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> gather_limit(n_workers, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>tasks):</span>
<span id="cb15-18">    semaphore <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> asyncio.Semaphore(n_workers)</span>
<span id="cb15-19"></span>
<span id="cb15-20">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> sem_task(task):</span>
<span id="cb15-21">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> semaphore:</span>
<span id="cb15-22">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> task</span>
<span id="cb15-23"></span>
<span id="cb15-24">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> asyncio.gather(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>(sem_task(task) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> task <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> tasks))</span>
<span id="cb15-25"></span>
<span id="cb15-26"></span>
<span id="cb15-27"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> aget_all(url, n, n_workers<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>):</span>
<span id="cb15-28">    limit <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> n_workers <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">or</span> n</span>
<span id="cb15-29">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> aiohttp.ClientSession() <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> session:</span>
<span id="cb15-30">        result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> gather_limit(limit, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>[aget(session, url) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> _ <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">range</span>(n)])</span>
<span id="cb15-31">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> result</span>
<span id="cb15-32"></span>
<span id="cb15-33"></span>
<span id="cb15-34"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> async_main(url, n):</span>
<span id="cb15-35">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> asyncio.run(aget_all(url, n))</span></code></pre></div></div>
<p>We also add this function to the benchmark. Let’s also add a benchmark with 1000 request, just for async methods.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb16" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb16-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## client.py</span></span>
<span id="cb16-2"></span>
<span id="cb16-3"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb16-4">    urls <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/wait"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://127.0.0.1:8000/asyncwait"</span>]</span>
<span id="cb16-5">    funcs <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [sync_get_all, thread_pool, async_main]</span>
<span id="cb16-6">    run_bench(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, funcs, urls)</span>
<span id="cb16-7">    run_bench(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>, [thread_pool, async_main], urls)</span></code></pre></div></div>
<p>The results can be suprising:</p>
<pre><code>15.84 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | sync_get_all
191.74 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | thread_pool
187.36 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | async_main
15.69 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | sync_get_all
217.35 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | thread_pool
666.23 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | async_main
234.24 reqs / sec | 1000 reqs | http://127.0.0.1:8000/wait | thread_pool
222.16 reqs / sec | 1000 reqs | http://127.0.0.1:8000/wait | async_main
316.08 reqs / sec | 1000 reqs | http://127.0.0.1:8000/asyncwait | thread_pool
1031.05 reqs / sec | 1000 reqs | http://127.0.0.1:8000/asyncwait | async_main</code></pre>
<p>It appears that the bottleneck was indeed on the client-side! When both sides are asynchronous - and there is a lot of IO - the speed is impressive!</p>
</section>
</section>
<section id="cpu-bound-workloads" class="level2">
<h2 class="anchored" data-anchor-id="cpu-bound-workloads">CPU-bound workloads</h2>
<p>This is all great, until some heavy computation is required. We refer to these as <em>CPU-bound</em> workloads, as opposed to <em>IO-bound</em>. Inspired by the legendary David Beazley’s <a href="https://youtu.be/MCs5OvhV9S4">live coding</a>, we are going to use a naive implementation of the Fibonacci sequence to perform heavy computations.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb18" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb18-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## server.py</span></span>
<span id="cb18-2"></span>
<span id="cb18-3">...</span>
<span id="cb18-4"></span>
<span id="cb18-5"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> fibo(n):</span>
<span id="cb18-6">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> n <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>:</span>
<span id="cb18-7">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb18-8">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb18-9">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> fibo(n <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> fibo(n <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb18-10"></span>
<span id="cb18-11"></span>
<span id="cb18-12"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@app.get</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/fib/</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{n}</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb18-13"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> fib(n: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>):</span>
<span id="cb18-14">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"fib"</span>: fibo(n)}</span></code></pre></div></div>
<p>Now, when I open two terminals, running <code>curl -I http://127.0.0.1:8000/fib/42</code> in one and <code>python client.py</code> in the other, we see the following results:</p>
<pre><code>8.75 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | sync_get_all
54.94 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | thread_pool
60.64 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | async_main
9.52 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | sync_get_all
53.02 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | thread_pool
46.81 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | async_main
72.87 reqs / sec | 1000 reqs | http://127.0.0.1:8000/wait | thread_pool
122.97 reqs / sec | 1000 reqs | http://127.0.0.1:8000/wait | async_main
72.36 reqs / sec | 1000 reqs | http://127.0.0.1:8000/asyncwait | thread_pool
51.73 reqs / sec | 1000 reqs | http://127.0.0.1:8000/asyncwait | async_main</code></pre>
<p>It’s not that bad, but a bit disappointing. Indeed, we have 20x less throughput for the originally most performant one (<code>asyncwait</code> route x <code>async_main</code> client).</p>
<p>What’s happening here ? In python, there is a <a href="https://wiki.python.org/moin/GlobalInterpreterLock">Global Interpreter Lock</a> (GIL). If one request takes a very long time to be processed with high-CPU activity, in the meantime other requests cannot be processed as quickly: priority is given to the computations. We will see later how to take care of this.</p>
<p>For now, we try nested recursive concurrency. Let’s add:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb20" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb20-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## server.py</span></span>
<span id="cb20-2"></span>
<span id="cb20-3">...</span>
<span id="cb20-4"></span>
<span id="cb20-5"></span>
<span id="cb20-6"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> afibo(n):</span>
<span id="cb20-7">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> n <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>:</span>
<span id="cb20-8">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb20-9">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb20-10">        fib1 <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> afibo(n <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb20-11">        fib2 <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> afibo(n <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb20-12">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> fib1 <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> fib2</span>
<span id="cb20-13"></span>
<span id="cb20-14"></span>
<span id="cb20-15"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@app.get</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/asyncfib/</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{n}</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb20-16"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> asyncfib(n: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>):</span>
<span id="cb20-17">    res <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> afibo(n)</span>
<span id="cb20-18">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"fib"</span>: res}</span></code></pre></div></div>
<p>Let’s also add a <a href="https://fastapi.tiangolo.com/tutorial/middleware/">timing middleware</a> to our FastAPI app:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb21" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb21-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## server.py</span></span>
<span id="cb21-2">...</span>
<span id="cb21-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> fastapi <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> FastAPI, Request</span>
<span id="cb21-4"></span>
<span id="cb21-5"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@app.middleware</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http"</span>)</span>
<span id="cb21-6"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> add_process_time_header(request: Request, call_next):</span>
<span id="cb21-7">    start_time <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> time.time()</span>
<span id="cb21-8">    response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> call_next(request)</span>
<span id="cb21-9">    process_time <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> time.time() <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> start_time</span>
<span id="cb21-10">    response.headers[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"X-Process-Time"</span>] <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>(process_time)</span>
<span id="cb21-11">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> response</span></code></pre></div></div>
<p>Now let’s test the speed:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb22" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb22-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-D</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-</span> http://127.0.0.1:8000/fib/30</span></code></pre></div></div>
<pre class="hl_lines=5"><code>HTTP/1.1 200 OK
server: uvicorn
content-length: 15
content-type: application/json
x-process-time: 0.17467308044433594

{"fib":1346269}</code></pre>
<p>And with async:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb24" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb24-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-D</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-</span> http://127.0.0.1:8000/asyncfib/30</span></code></pre></div></div>
<pre class="hl_lines=5"><code>HTTP/1.1 200 OK
server: uvicorn
content-length: 15
content-type: application/json
x-process-time: 0.46001315116882324

{"fib":1346269}</code></pre>
<p>It’s not that bad for <img src="https://latex.codecogs.com/png.latex?2%5E%7B30%7D"> overhead. But we see here a limitation of threads in Python: the <a href="https://julialang.org/blog/2019/07/multithreading/">same code in Julia</a> would lead to a speed-up (using parallelism)!</p>
</section>
<section id="gunicorn-and-multiprocessing" class="level2">
<h2 class="anchored" data-anchor-id="gunicorn-and-multiprocessing">Gunicorn and multiprocessing</h2>
<p>So far we’ve used FastAPI with Uvicorn. The latter can be <a href="https://www.uvicorn.org/#running-with-gunicorn">run with Gunicorn</a>. Gunicorn forks a base process into <code>n</code> worker processes, and each worker is managed by Uvicorn (with the asynchronous uvloop). Which means:</p>
<ul>
<li>Each worker is concurrent</li>
<li>The worker pool implements parallelism</li>
</ul>
<p>This way, we can have the best of both worlds: concurrency (multithreading) and parallelism (multiprocessing).</p>
<p>Let’s try this with the last setup, when we ran the benchmark while asking for the 42th Fibonacci number:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb26" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb26-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">pip</span> install gunicorn</span></code></pre></div></div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb27" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb27-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">gunicorn</span> server:app <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-w</span> 2 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-k</span> uvicorn.workers.UvicornWorker <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--reload</span></span></code></pre></div></div>
<p>we get the following results:</p>
<pre><code>19.02 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | sync_get_all
216.84 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | thread_pool
223.52 reqs / sec | 100 reqs | http://127.0.0.1:8000/wait | async_main
18.80 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | sync_get_all
400.12 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | thread_pool
208.68 reqs / sec | 100 reqs | http://127.0.0.1:8000/asyncwait | async_main
241.06 reqs / sec | 1000 reqs | http://127.0.0.1:8000/wait | thread_pool
311.40 reqs / sec | 1000 reqs | http://127.0.0.1:8000/wait | async_main
433.80 reqs / sec | 1000 reqs | http://127.0.0.1:8000/asyncwait | thread_pool
1275.48 reqs / sec | 1000 reqs | http://127.0.0.1:8000/asyncwait | async_main   </code></pre>
<p>Which is on par (if not a bit better!) than with a single Uvicorn process</p>
<p>The final files (client and server) are available as a <a href="https://gist.github.com/horaceg/2afe33aff9ba28c36757f52f13edd298">github gist</a></p>
</section>
<section id="further-resources" class="level2">
<h2 class="anchored" data-anchor-id="further-resources">Further resources</h2>
<p>I wholeheartedly recomment this amazing <a href="https://youtu.be/MCs5OvhV9S4">live-coding session</a> by David Beazley. Maybe you can google <a href="https://en.wikipedia.org/wiki/WebSocket">websockets</a> first, just to get that they open a bi-directional channel between client and server.</p>
<p>You can also read <a href="https://stackoverflow.com/a/60644649/9162021">this detailed answer</a> from stackoverflow to grasp differences between concurrency and parallelism in python.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2021,
  author = {Guy, Horace},
  title = {Concurrency in {Python} with {FastAPI}},
  date = {2021-09-10},
  url = {https://blog.horaceg.xyz/posts/python-concurrency/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2021" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2021. <span>“Concurrency in Python with FastAPI.”</span>
September 10. <a href="https://blog.horaceg.xyz/posts/python-concurrency/">https://blog.horaceg.xyz/posts/python-concurrency/</a>.
</div></div></section></div> ]]></description>
  <category>python</category>
  <category>programming</category>
  <category>concurrency</category>
  <category>parallelism</category>
  <guid>https://blog.horaceg.xyz/posts/python-concurrency/</guid>
  <pubDate>Fri, 10 Sep 2021 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Bayesian epidemiology</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/uncover-covid/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/uncover-covid/diagram_bayes_epi.png" class="img-fluid figure-img"></p>
<figcaption>A diagram of the overall workflow</figcaption>
</figure>
</div>
<p>In March 2020, lockdown measures were implemented all over the world due to the spread of Covid-19. I was at home, a bit bored and I was looking for something to do.</p>
<p><strong>TL;DR</strong>: - <a href="https://www.kaggle.com/achyrogue/impact-of-interventions-through-mobility-data">Kaggle submission</a> - Github <a href="https://github.com/horaceg/uncover">repo</a></p>
<section id="mobility-datasets" class="level2">
<h2 class="anchored" data-anchor-id="mobility-datasets">Mobility datasets</h2>
<p><a href="https://www.google.com/covid19/mobility/">Google</a> and <a href="https://covid19.apple.com/mobility">Apple</a> published datasets related to (anonymized) activity of smartphones users. They called this <em>mobility</em> data, although this doesn’t show trajectories but rather the frequentation of some categories of places, such as grocery stores, residential or workplaces.</p>
<p>At first, there were no csv files, only pdf graphs that did not show raw data. I found a reddit post that extracted some of the data from the reports (for US regions), and I extended it to make it work with worlwide data. I then published it on a <a href="https://covid19-analysis.netlify.app/posts/mobility/">small website</a> I made for the occasion.</p>
<p>The code to do the download + parsing + extraction is available on a <a href="https://github.com/horaceg/covid19-analysis/tree/master/mobility">github repo</a></p>
</section>
<section id="uncover-challenge" class="level2">
<h2 class="anchored" data-anchor-id="uncover-challenge">Uncover challenge</h2>
<p>The <a href="https://www.kaggle.com/roche-data-science-coalition/uncover">Uncover challenge</a> appeared on Kaggle a few days afterwards. From the description:</p>
<blockquote class="blockquote">
<p>The Roche Data Science Coalition (RDSC) is requesting the collaborative effort of the AI community to fight COVID-19. This challenge presents a curated collection of datasets from 20 global sources and asks you to model solutions to key questions that were developed and evaluated by a global frontline of healthcare providers, hospitals, suppliers, and policy makers.</p>
</blockquote>
<p>The task that immediately caught my attention:</p>
<blockquote class="blockquote">
<p>How is the implementation of existing strategies affecting the rates of COVID-19 infection?</p>
</blockquote>
<p>I had a (not so secret) weapon: mobility data.</p>
</section>
<section id="exploration" class="level2">
<h2 class="anchored" data-anchor-id="exploration">Exploration</h2>
<p>First I did some exploration and dataviz leveraging the <a href="https://www.acaps.org/covid-19-government-measures-dataset">ACAPS</a> dataset, Google mobility data and epidemiology data (number of cases, deaths etc.).</p>
<p>I exported a jupyter notebook to html, and since I leveraged <a href="https://altair-viz.github.io/">Altair</a> based on <a href="https://vega.github.io/vega-lite/">Vega-lite</a>, the data is embedded in the js so it works without a server.</p>
<!-- Try to interact with the plots in the [exported notebook](/measures_and_rate_explo.html). -->
</section>
<section id="compartmental-models" class="level2">
<h2 class="anchored" data-anchor-id="compartmental-models">Compartmental models</h2>
<p>Then I found this <a href="https://www.kaggle.com/anjum48/seir-hcd-model">Kaggle notebook</a> by <span class="citation" data-cites="anjum48">@anjum48</span> (<em>Datasaurus</em>), published in the Covid forecasting competition (another Kaggle competition). I like the scientific approach: based on well-known <a href="https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology#The_SEIR_model">Compartmental models</a> in epidemiology, he extends it to fit the Covid-19 situation.</p>
</section>
<section id="recruiting-the-team" class="level2">
<h2 class="anchored" data-anchor-id="recruiting-the-team">Recruiting the team</h2>
<p>After this preliminary work, I reach out to two friends for a collaboration: a data scientist and a quant analyst. They were also motivated by the project ! So we went to work on the challenge with this special forces team.</p>
</section>
<section id="probabilistic-programming" class="level2">
<h2 class="anchored" data-anchor-id="probabilistic-programming">Probabilistic programming</h2>
<p>I had heard about bayesian inference before, but did not know how it worked. We decided to explore this path. As a first step, we chose a framework. We went for the young an promising <a href="http://num.pyro.ai/en/stable/getting_started.html">Numpyro</a>, based on <a href="https://github.com/google/jax/">JAX</a> and made by some of the (very skilled) <a href="https://pyro.ai/">Pyro</a> authors. This is also how I discovered JAX.</p>
<p>An example in particular seemed to fit our problem: a prey-predator differential equations <a href="http://num.pyro.ai/en/stable/examples/ode.html">model</a>.</p>
<p>Imperial College in London had already started to publish a series of papers called the <a href="https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/covid-19-reports/">Covid-19 reports</a>. Report 13 was particularly relevant for our challenge, and can be found <a href="https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/report-13-europe-npi-impact/">here</a>.</p>
</section>
<section id="collaborating-on-a-jupyter-notebook" class="level2">
<h2 class="anchored" data-anchor-id="collaborating-on-a-jupyter-notebook">Collaborating on a Jupyter notebook</h2>
<p>Kaggle required a notebook submission. Collaboration on notebook is particularly challenging: indeed, the JSON format that mixes code and data is not git-friendly. So, I developed a small ipython magic command that allows us to modify code inside a notebook cell, and when executed it saves it into a specific file. The notebook wasn’t version-controlled, but every cell snippet was. It wasn’t ideal, since the imports were not included in every cell.</p>
</section>
<section id="the-submission" class="level2">
<h2 class="anchored" data-anchor-id="the-submission">The submission</h2>
<p><em>In fine</em>, we implemented bayesian Covid-19 models that perform:</p>
<ul>
<li>An estimation of the effective reproduction number <img src="https://latex.codecogs.com/png.latex?R_t"> as a function of mobility data and reported deaths</li>
<li>An estimation of mobility as a function of non-pharmaceutical interventions</li>
</ul>
<p>See the <a href="https://www.kaggle.com/achyrogue/impact-of-interventions-through-mobility-data">kaggle submission</a> for the full analysis.</p>
<p>The Roche team accepted our submission, so we won the challenge for this particular task.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2021,
  author = {Guy, Horace},
  title = {Bayesian Epidemiology},
  date = {2021-09-09},
  url = {https://blog.horaceg.xyz/posts/uncover-covid/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2021" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2021. <span>“Bayesian Epidemiology.”</span> September 9. <a href="https://blog.horaceg.xyz/posts/uncover-covid/">https://blog.horaceg.xyz/posts/uncover-covid/</a>.
</div></div></section></div> ]]></description>
  <category>machine learning</category>
  <category>probability</category>
  <category>bayes</category>
  <category>epidemiology</category>
  <category>kaggle</category>
  <category>probabilistic programming</category>
  <guid>https://blog.horaceg.xyz/posts/uncover-covid/</guid>
  <pubDate>Thu, 09 Sep 2021 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Things I’m excited about</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/excited-about/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/excited-about/tinkerer.webp" class="img-fluid figure-img"></p>
<figcaption>Tinkerer imagining great things while looking at a computer screen. Made with ChatGPT</figcaption>
</figure>
</div>
<p>As a data scientist by day, and tinkerer by night, I am often browsing Hacker News and the like in order to be up-to-date with the latest cool technologies.</p>
<section id="python" class="level2">
<h2 class="anchored" data-anchor-id="python">Python</h2>
<p>On the scientific computing side, there is one behemoth: Python. However, it still isn’t on par with R for statistics. <code>Pandas</code> is often criticized and compared to the supposedly more user-friendly <code>dplyr</code>. The GIL prevents true multithreading. There is a fragmented community that can have a hard time understanding each other with the introduction of:</p>
<ul>
<li>Static types</li>
<li><a href="https://en.wikipedia.org/wiki/Asynchronous_I/O">Asynchronous IO</a> (<code>asyncio</code>)</li>
<li>Scientific python:
<ul>
<li>foundation stack: <code>numpy</code>, <code>pandas</code>, <code>matplotlib</code>, <code>scipy</code>, <code>scikit-learn</code></li>
<li>automatic differentiation stacks: <code>tensorflow</code>, <code>pytorch</code>, <code>jax</code> …</li>
</ul></li>
</ul>
<p>There are web developers, tinkerers, scientists, machine learning researchers, engineers and even business people are getting up to speed with Python nowadays! This is a wonderful success, and I believe no other programming language has ever been thriving that much.</p>
<p>It could appear the needs of these communities are disparate: although it might be useful here and there, few scientists will bother with asyncio, since the workloads are mostly <a href="https://en.wikipedia.org/wiki/CPU-bound">CPU-bound</a>.</p>
<p>However, these can mix and match, as we can see with the popularity of <a href="https://fastapi.tiangolo.com/">FastAPI</a>, that leverages static types and concurrent programming, and is used for machine learning REST endpoints at e.g.&nbsp;Uber and Microsoft (as stated in their landing page).</p>
<p>Every day witnesses tens of new Python packages, many of which relate in some way to machine learning. What an exciting time to be a Python dev in machine learning! <code>:)</code></p>
<p>There’s one slight problem with this situation though: numerical code in pure python will never be nearly as fast as in a low-level language like C, C++ or Rust. Thus, there will always be a great divide between library developers that need to leverage all the machine hardware, and library users, probably scientists, that won’t be able to tinker with the internals. Of course, there are some exceptions but statistically I believe this is true to some degree. I, for one, never once even glanced at some C++ source code of e.g.&nbsp;XLA or <code>jaxlib</code>, which I use regularly.</p>
</section>
<section id="julia" class="level2">
<h2 class="anchored" data-anchor-id="julia">Julia</h2>
<p><a href="https://julialang.org/">Julia</a> is also gaining traction, a dozen years after its inception. Very fast once <a href="https://en.wikipedia.org/wiki/Just-in-time_compilation">JIT-compiled</a>, its founders want to solve the two-languages problem in scientific computing: (at least) one low-level language used by library developers for fast numerical computation, e.g.&nbsp;C, C++, Rust, and one language for high-level manipulation, e.g.&nbsp;Python, R, Matlab.</p>
<p>Julia offers a unified stack in numerical computing, with the ease of manipulation of Matlab for arrays, the development speed of Python, and the speed of C. It already has a vibrant ecosystem of packages, and is better suited than Python for statistics, optimization, differential equations. However, the huge boom right now is in Machine learning, and even though Julia has strong libraries to offer, network effects retain most people on Python.</p>
<p>One promising Julia library is the mysterious <a href="https://github.com/JuliaDiff/Diffractor.jl">Diffractor.jl</a>. Keno Fischer, the author, has gone <a href="https://youtu.be/mQnSRfseu0c">deep into Category Theory</a> to implement a next-gen (sic!) automatic differenciation system. The potential is huge: imagine you could efficiently differentiate any Julia code! The first thing I will do when it’s ready is to implement a <a href="https://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/">Markov Chain Monte Carlo</a> engine.</p>
</section>
<section id="elixir" class="level2">
<h2 class="anchored" data-anchor-id="elixir">Elixir</h2>
<p>Some very skilled colleagues of mine mentioned an emerging programming language: <a href="https://elixir-lang.org/">Elixir</a>. I didn’t bother until I discovered that <a href="https://twitter.com/josevalim">Jose Valim</a> launched the <a href="(https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available)">Numerical Elixir</a> project: - <a href="https://github.com/elixir-nx/nx">Nx</a>, for multi-dimensional arrays - <a href="https://github.com/elixir-nx/axon">Axon</a>, for neural networks - <a href="https://github.com/elixir-nx/explorer">Explorer</a>, for dataframes</p>
<p>and a <a href="https://dashbit.co/blog/announcing-livebook">blog post</a> annoncing an innovative development environment: <a href="https://github.com/livebook-dev/livebook">Livebook</a></p>
<p>This has been my gateway drug to web app development. Indeed, I googled Liveview and witnessed Chris McCord <a href="https://youtu.be/MZvmYaFkNJI">building a Twitter clone</a> in 15 minutes with it. A few days later, I was avidly following Pragmatic Studio’s <a href="https://pragmaticstudio.com/courses/phoenix-liveview">Liveview course</a> and was instantly hooked (no pun intended) on Elixir, Phoenix and Liveview. Stay tuned for new projects!</p>
</section>
<section id="front-end" class="level2">
<h2 class="anchored" data-anchor-id="front-end">Front-end</h2>
<p>I want to learn more about the web, <em>THE</em> platform. I have started to <a href="../../posts/need-for-speed">experiment with Svelte</a>, and I picked up JS (and HTML) basics along the way.</p>
<p>However, I lack the CSS fundamentals to build a pleasant webpage, which is why I send all my gratitude to the wizards that built all these <a href="../../posts/static-blog">amazing open-source Static Site Generators</a> (SSG) and themes!</p>
<p>When I am ready, <a href="https://kit.svelte.dev/">SvelteKit</a> will probably have reached 1.0 already and I will be able to make a blog with <a href="https://mdsvex.com/">MdSvex</a>, inserting Svelte components into the articles written in Markdown. I would love to build a cool portfolio like <a href="https://github.com/mhatvan/markushatvan.com">Markus Hatvan</a>.</p>
<p>I am also very intrigued by <a href="https://tailwindcss.com/">Tailwind CSS</a> which is all the rage right now ; so I don’t know if I should learn vanilla CSS or Tailwind first.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2021,
  author = {Guy, Horace},
  title = {Things {I’m} Excited About},
  date = {2021-09-08},
  url = {https://blog.horaceg.xyz/posts/excited-about/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2021" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2021. <span>“Things I’m Excited About.”</span> September 8.
<a href="https://blog.horaceg.xyz/posts/excited-about/">https://blog.horaceg.xyz/posts/excited-about/</a>.
</div></div></section></div> ]]></description>
  <category>web</category>
  <category>front-end</category>
  <category>back-end</category>
  <category>future</category>
  <category>scientific computing</category>
  <category>machine learning</category>
  <category>programming</category>
  <guid>https://blog.horaceg.xyz/posts/excited-about/</guid>
  <pubDate>Wed, 08 Sep 2021 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Setting up your own blog</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/static-blog/</link>
  <description><![CDATA[ 





<p>I’ve been wanting to publish online for some time. I was very young during the Wordpress wave, so I was never really drawn into it.</p>
<section id="jekyll" class="level2">
<h2 class="anchored" data-anchor-id="jekyll">Jekyll</h2>
<p>I learned early about static website, first with <a href="https://jekyllrb.com/">Jekyll</a> and <a href="https://pages.github.com/">Github pages</a>, in 2016. At the time, I was using Windows 10 and wasn’t quite experienced. Since Jekyll is written<sup>1</sup> in Ruby, I needed to bootstrap Ruby and the many gems required to run the blog. It was painful. I couldn’t understand the cryptic error messages ; for every update everything broke, I don’t think I used <a href="https://github.com/rbenv/rbenv"><code>rbenv</code></a> at all, etc.<sup>2</sup>.</p>
<p>I still managed to publish to Github pages somehow, and I was very proud of it. However, a few days after the first post, I gave up. I still learned to write in Markdown and initiated my software journey. It was somewhat fun to play with the terminal, the filesystem, the web.</p>
</section>
<section id="hugo" class="level2">
<h2 class="anchored" data-anchor-id="hugo">Hugo</h2>
<p>Fast-forward three years since my failed attempt. This time, I had some content: a summary of a peculiar political book I was particularly excited about. I also had a bit more experience with the terminal and I was using <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows Subsystem for Linux</a> (WSL) to have a *nix shell available.</p>
<p>I wrote the post first, in Markdown, then I hunted online for a static site generator (SSG) easier than Jekyll. This is when I found <a href="https://gohugo.io/">Hugo</a>: a single binary and you’re good to go!<sup>3</sup></p>
<p>The next step was finding a good theme. Once I found one that seemed to fit the bill (with e.g.&nbsp;built-in search), I needed to wrestle with git submodules to make it work.</p>
<p><em>In fine</em>, I managed to publish my content to the world, this time using <a href="https://www.netlify.com/">Netlify</a> to have more flexibility than Github pages. It worked okay, but I had some troubles with large image files that took a very long time to load. Moreover, the built-in search wasn’t very good and I wasn’t excited about the whole look of the website.</p>
</section>
<section id="zola" class="level2">
<h2 class="anchored" data-anchor-id="zola">Zola</h2>
<p>Two years later: here we go again. This time, I have a few years of experience under my belt, I am using Linux as my daily-driver (and MacOS at work). I’m much more comfortable using a terminal and software developemnt in general.</p>
<p>A friend of mine had been excited about <a href="https://www.rust-lang.org/">Rust</a> for a while, and I knew that there were some SSGs written in it. So, I browsed <a href="https://www.getzola.org/">Zola</a> themes hoping to find a decent-looking website. Since there aren’t that many themes for Zola, I quickly found a good one <sup>4</sup>.</p>
<p>With this setup, I had everything I was looking for:</p>
<ul>
<li>A single binary: no dependencies</li>
<li>Blazingly fast</li>
<li><a href="https://katex.org/"><img src="https://latex.codecogs.com/png.latex?%5CKaTeX"></a> support</li>
<li>A good search UI &amp; UX</li>
<li>Image processing built-in</li>
<li>A nice theme</li>
<li><a href="https://bulma.io">Bulma</a> CSS, easy for the CSS newbie that I am</li>
</ul>
<p>So i just went for it on a whim. It took me three hours to write my first post, set the dev environment up and deploying it to <a href="https://developers.cloudflare.com/pages/">Cloudflare Pages</a>. I even have web anayltics for free, <em>without additional javascript</em><sup>5</sup>: no cookies, yay!</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/static-blog/blog-screenshot.png" class="img-fluid figure-img"></p>
<figcaption>Some posts in my blog</figcaption>
</figure>
</div>
<p>Overall I am very happy with the experience so far. The website could be leaner and faster, but I believe it is fast enough. For now. And it’s free (as in beer).</p>


</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>By <a href="https://en.wikipedia.org/wiki/Tom_Preston-Werner">Tom Preston-Werner</a>, founder and former CEO of Github.↩︎</p></li>
<li id="fn2"><p>Later on, I <a href="https://youtu.be/USDbjmxEZ_I?t=1544">learned</a> that I wasn’t the only one experiencing this fatigue.↩︎</p></li>
<li id="fn3"><p>No pun intended since Hugo is written in Go.↩︎</p></li>
<li id="fn4"><p><a href="https://www.getzola.org/themes/deepthought/">DeepThought</a>↩︎</p></li>
<li id="fn5"><p>Actually this theme ships with the Google Analytics script, since there is no tree-shaking nor dead-code elimination (yet)↩︎</p></li>
</ol>
</section><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2021,
  author = {Guy, Horace},
  title = {Setting up Your Own Blog},
  date = {2021-09-07},
  url = {https://blog.horaceg.xyz/posts/static-blog/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2021" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2021. <span>“Setting up Your Own Blog.”</span> September 7.
<a href="https://blog.horaceg.xyz/posts/static-blog/">https://blog.horaceg.xyz/posts/static-blog/</a>.
</div></div></section></div> ]]></description>
  <category>blog</category>
  <category>CDN</category>
  <category>static</category>
  <category>serverless</category>
  <category>meta</category>
  <guid>https://blog.horaceg.xyz/posts/static-blog/</guid>
  <pubDate>Tue, 07 Sep 2021 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Mathematics of bookmaking</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/betting-theory/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/betting-theory/chips_poker.avif" class="img-fluid figure-img"></p>
<figcaption>(sports) Betting is similar to gambling</figcaption>
</figure>
</div>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>Have you ever placed a bet ? Wondered what the odds actually mean ?</p>
<p>It’s all linked to probability.</p>
</section>
<section id="setup" class="level2">
<h2 class="anchored" data-anchor-id="setup">Setup</h2>
<p>Let us place focus on a betting situation: a bookie offers odds on an event <img src="https://latex.codecogs.com/png.latex?A">, and on its complementary <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BA%7D">. There can be multiple binary events at the same time (or close). We will then extend to multiple outcomes.</p>
<p>In our case, we consider real-life processes, which are physical, and their outcomes. We regularly take the example of sports gambling, where processes are fixtures, which generate various outcomes.</p>
<p>For a given bet, we have (decimal) <em>odds</em> <img src="https://latex.codecogs.com/png.latex?o_A"> such that if you place a bet with <em>stake</em> <img src="https://latex.codecogs.com/png.latex?s"> on event <img src="https://latex.codecogs.com/png.latex?A">, you lose <img src="https://latex.codecogs.com/png.latex?s"> in all cases, and if you win, you get <img src="https://latex.codecogs.com/png.latex?so_A"> in addition. So your total net gain is <img src="https://latex.codecogs.com/png.latex?s(o_A%20-%201)"> in case of a win, and <img src="https://latex.codecogs.com/png.latex?-s"> otherwise.</p>
<p>Denote</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7BA%7D%20:=%20%5Cfrac%7B1%7D%7Bo_A%7D"> the inverse odds. This quantity is usually <em>not</em> a probability.</li>
</ul>
<p>Define the net unit gain:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?G%5Eo_A%20:=%20o_A%20%5Cmathbf%7B1%7D_A%20-%201"></li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?sG%5Eo_A%20=%20s(o_A%20%5Cmathbf%7B1%7D_A%20-%201)"> is then the net gain of placing a bet on event <img src="https://latex.codecogs.com/png.latex?A"> with stake s and inverse odds <img src="https://latex.codecogs.com/png.latex?%5Cpi_A">.</p>
<p>Let us also define the <em>natural</em> bet, that is the bet on <img src="https://latex.codecogs.com/png.latex?A"> with a stake of <img src="https://latex.codecogs.com/png.latex?%5Cpi_A">:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F%5E%7B%5Cpi%7D_A%20:=%20%5Cpi_A%20G%5Eo_A%20=%20%5Cmathbf%7B1%7D_A%20-%20%5Cpi_A"></li>
</ul>
<p>We will see that this quantity is useful to stake the bets in order to combine them. Intuitively, it makes sense to stake bets with low odds with large money since you are (supposedly) likely to win, and bets with high odds with fewer money since you are (supposedly) unlikely to win.</p>
<p>Define <img src="https://latex.codecogs.com/png.latex?p"> a probability on the universe <img src="https://latex.codecogs.com/png.latex?%5COmega">, and <img src="https://latex.codecogs.com/png.latex?p_A"> the probability of <img src="https://latex.codecogs.com/png.latex?A%20%5Csubset%20%5COmega">. This might be <em>the true</em> probability law on sports events, or your belief of it, or someone else’s belief. All we know it that it is a coherent belief since <img src="https://latex.codecogs.com/png.latex?p"> satisfies the axioms of probability.</p>
<p>A bet is <em>sure</em> when its gain is deterministic, i.e.&nbsp;there is not randomness involved.</p>
<p>When there is no possible confusion, we write <img src="https://latex.codecogs.com/png.latex?G_A%20%5Ccong%20G%5Eo_A"> and <img src="https://latex.codecogs.com/png.latex?F_A%20%5Ccong%20F%5E%7B%5Cpi%7D_A"></p>
</section>
<section id="coherent-odds" class="level2">
<h2 class="anchored" data-anchor-id="coherent-odds">Coherent odds</h2>
<!-- [Todo: Quelle est la relation de $\pi$ avec $\hat{p}$ ? qqc à propos de la continuité absolue de $\pi$ par rapport à $p$ et la dérivée de Radon-Nikodym? Voir aussi] -->
<!-- [Todo [Girsanov](https://en.wikipedia.org/wiki/Girsanov_theorem)] -->
<p>In this section, and in this section only, we assume that <img src="https://latex.codecogs.com/png.latex?%5Cpi"> is additive (see definition below).</p>
<section id="coherence" class="level3">
<h3 class="anchored" data-anchor-id="coherence">Coherence</h3>
<p>We say that the odds are <em>coherent</em> for a given bookie, or that a <em>book</em> is coherent, if we have the following:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7BA%20%5Ccup%20B%7D%20=%20%5Cpi_A%20+%20%5Cpi_B"> when <img src="https://latex.codecogs.com/png.latex?A%20%5Ccap%20B%20=%20%5Cemptyset"></li>
</ul>
<p>It is another way to say that <img src="https://latex.codecogs.com/png.latex?%5Cpi"> is <strong>additive</strong>.</p>
<p>Then we can deduce other rules:</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7BA%20%5Ccup%20B%7D%20=%20%5Cpi_A%20+%20%5Cpi_B%20-%20%5Cpi_%7BA%20%5Ccap%20B%7D"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cforall%20A,%20%5Cpi_%7B%5COmega%7D%20=%20%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D"></p></li>
</ul>
<p>This last equality shows that the <strong>booksum</strong> is universal. This also allows us to prove that</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5Cemptyset%7D%20=%200"></li>
</ul>
<p>Which means that <img src="https://latex.codecogs.com/png.latex?%5Cpi"> is a measure, which is finite.</p>
<!-- 
#### Additional assumptions

Assume that $\pi$ is usually absolutely continuous with respect to $p$, that is every event with zero probability doesn't have any odds (or odds of $+\infty$, theoretically). 

[If not, apply the Lebesgue decomposition $\pi = \pi^1 + \pi^2$ where $\pi^1$ is absolutely continuous with respect ot $p$ and $\pi^2$ is orthogonal to $p$; then let $\pi \leftarrow \pi^1$].

Then we can compute the Radon-Nikodym derivate:

$h := \frac{d\pi}{dp}$  thus $\pi_A = \int_A h dp = \mathbb{E}[h\mathbf{1}_A]$. 
 -->
<p>Warning: <img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5Coverline%7BA%7D%7D%20%5Cneq%201%20-%20%5Cpi_A">, rather <img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5Coverline%7BA%7D%7D%20=%20%5Cpi_%7B%5COmega%7D%20-%20%5Cpi_A">. We will see that afterwards, this allows the bookie to make a profit.</p>
<p>If we apply the right transformation to <img src="https://latex.codecogs.com/png.latex?%5Cpi">, we might get (approximately) the <strong>implied probability</strong> law <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bp%7D"> estimated by the bookie, and his margin on each bet.</p>
<p>Note that the popular saying that <img src="https://latex.codecogs.com/png.latex?%5Cpi%20%5Csimeq%20%5Chat%7Bp%7D"> is false, since those do not sum to 1. To go further see the implied probability section.</p>
</section>
<section id="product" class="level3">
<h3 class="anchored" data-anchor-id="product">Product</h3>
<p>We say that <img src="https://latex.codecogs.com/png.latex?A%20%7B%5Cperp%20%5C!%5C!%5C!%20%5Cperp%7D%20B"> (for <img src="https://latex.codecogs.com/png.latex?%5Cpi">) if we have <img src="https://latex.codecogs.com/png.latex?%5Cpi_%7BA%20%5Ccap%20B%7D%20=%20%5Cpi_A%20%5Cpi_B">.</p>
<ul>
<li>For <img src="https://latex.codecogs.com/png.latex?A%20%7B%5Cperp%20%5C!%5C!%5C!%20%5Cperp%7D%20B">, <img src="https://latex.codecogs.com/png.latex?G_%7BA%20%5Ccap%20B%7D%20+%201%20=%20(G_A%20+%201)(G_B%20+%201)"></li>
</ul>
<p>i.e.</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F_%7BA%20%5Ccap%20B%7D%20=%20(F_A%20+%20%5Cpi_A)(F_B%20+%20%5Cpi_B)%20-%20%5Cpi_A%20%5Cpi_B"></li>
</ul>
</section>
<section id="additivity" class="level3">
<h3 class="anchored" data-anchor-id="additivity">Additivity</h3>
<p>Let <img src="https://latex.codecogs.com/png.latex?A">, <img src="https://latex.codecogs.com/png.latex?B"> such that <img src="https://latex.codecogs.com/png.latex?A%20%5Ccap%20B%20=%20%5Cemptyset">. We have :</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?G_%7BA%20%5Ccup%20B%7D%20=%20%5Cfrac%7B%5Cpi_A%7D%7B%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%20G_A%20+%20%5Cfrac%7B%5Cpi_B%7D%7B%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%20G_B">.</li>
</ul>
<p>i.e.</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F_%7BA%20%5Ccup%20B%7D%20=%20F_A%20+%20F_B">.</li>
</ul>
<p>In the general case, when <img src="https://latex.codecogs.com/png.latex?A%20%5Ccap%20B%20%5Cneq%20%5Cemptyset">, we have:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_B%20=%20F_%7BA%20%5Ccap%20B%7D%20+%20F_%7BA%20%5Ccup%20B%7D"></li>
</ul>
</section>
<section id="the-sure-universal-bet" class="level3">
<h3 class="anchored" data-anchor-id="the-sure-universal-bet">The sure universal bet</h3>
<p>When we apply the additivity formula for <img src="https://latex.codecogs.com/png.latex?B%20%5Cleftarrow%20%5Coverline%7BA%7D">, we get:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?G_%7B%5COmega%7D%20=%20%5Cfrac%7B%5Cpi_A%7D%7B%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%20G_A%20+%20%5Cfrac%7B%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%7B%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%20G_%7B%5Coverline%7BA%7D%7D%20=%20%5Cfrac%7B1%7D%7B%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%20-%201"></li>
</ul>
<p>i.e.</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F_%7B%5COmega%7D%20=%20F_A%20+%20F_%7B%5Coverline%7BA%7D%7D%20=%201%20-%20(%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D)"></li>
</ul>
<p>Unsurprinsingly, this is a sure bet. It is the opposite of the <em>margin</em> of the bookie. That is the gain of the unit bet on the sure event <img src="https://latex.codecogs.com/png.latex?%5COmega">, placing a bet on both <img src="https://latex.codecogs.com/png.latex?A"> and <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BA%7D"> with the proper stakes ratio.</p>
<p>By identifying <img src="https://latex.codecogs.com/png.latex?G_%7B%5COmega%7D%20=%20%5Cmathbf%7B1_%7B%5COmega%7D%7D%5Cfrac%7B1%7D%7B%5Cpi_%7B%5COmega%7D%7D%20-%201">, we recognize that:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5COmega%7D%20=%20%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D"></li>
</ul>
<p>This quantity <img src="https://latex.codecogs.com/png.latex?%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D"> is the <strong>booksum</strong> in general (for a given event <img src="https://latex.codecogs.com/png.latex?A">).</p>
<p>Let us write what is now trivial:</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?G_%7B%5COmega%7D%20=%20%5Cfrac%7B1%20-%20%5Cpi_%7B%5COmega%7D%7D%7B%5Cpi_%7B%5COmega%7D%7D%20=%20%5Cfrac%7B1%7D%7B%5Cpi_%7B%5COmega%7D%7D%20-%201%20=%20o_%7B%5COmega%7D%20-%201"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?F_%7B%5COmega%7D%20=%201%20-%20%5Cpi_%7B%5COmega%7D"></p></li>
</ul>
</section>
<section id="transition" class="level3">
<h3 class="anchored" data-anchor-id="transition">Transition</h3>
<p>Note that we always have <img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_%7B%5Coverline%7BA%7D%7D%20=%201%20-%20(%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D)">, even if the odds are uncoherent. However, if the odds are not coherent, we can’t say that this quantity is indeed <img src="https://latex.codecogs.com/png.latex?F_%7B%5COmega%7D">. For instance, the bookie could theoretically offer other odds for the event <img src="https://latex.codecogs.com/png.latex?%5COmega">, leading to <img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5COmega%7D%20%5Cneq%20%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D">. However, would they really offer such a bet ?</p>
<p>In the following sections, we do not assume coherent odds. However, we use the notations</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%20:=%20%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Chat%7BG%7D_%7B%5COmega%7D%20:=%20%5Cfrac%7B1%7D%7B%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%7D%20-%201"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D%20=%201%20-%20(%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D)"></p></li>
</ul>
</section>
</section>
<section id="fairness" class="level2">
<h2 class="anchored" data-anchor-id="fairness">Fairness</h2>
<p>A <em>fair</em> bet implies <img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D_p%5BF_A%5D%20=%200">. Since <img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D_p%5BF_A%5D%20=%20p_A%20-%20%5Cpi_A">, we then have:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?p_A%20=%20%5Cpi_A%20=%20%5Cfrac%7B1%7D%7Bo_A%7D">.</li>
</ul>
<p>In this case, the odds are coherent since <img src="https://latex.codecogs.com/png.latex?%5Cpi%20=%20p"> is a probability measure. We have <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B1%7D%7Bo_A%7D%20+%20%5Cfrac%7B1%7D%7Bo_%7B%5Coverline%7BA%7D%7D%7D%20=%201">, that is <img src="https://latex.codecogs.com/png.latex?o_%7B%5Coverline%7BA%7D%7D%20=%20%5Cfrac%7Bo_A%7D%7Bo_A%20-%201%7D">.</p>
<p>We also have the following:</p>
<!-- * $o_A + o_{\overline{A}} = o_A o_{\overline{A}}$ -->
<!-- * $mG_A + nG_{\overline{A}} = 0 \iff n = m(o_A - 1)$ -->
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5COmega%7D%20=%201"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?G_%7B%5COmega%7D%20=%20F_%7B%5COmega%7D%20=%200"></p></li>
</ul>
<p>This last formula is crucial to understand fairness. It is useful to hedge your bets with the right amount, since e.g.&nbsp;<img src="https://latex.codecogs.com/png.latex?-G_A%20=%20%5Cfrac%7Bo_A%7D%7Bo_%7B%5Coverline%7BA%7D%7D%7D%20G_%7B%5Coverline%7BA%7D%7D">. More on this at Negative bets.</p>
<p>Note that even if bookkeepers were unwilling to make a profit, they don’t really know the true probability (is there such a thing?). So the best they can do is to offer odds that are fair for a probability measure <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bp%7D">, that is the reflection of their honest and coherent beliefs. This is equivalent to <img src="https://latex.codecogs.com/png.latex?%5Cpi%20=%20%5Chat%7Bp%7D">, and thus to the two following statements:</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cpi"> is coherent (additive), e.g.&nbsp;<img src="https://latex.codecogs.com/png.latex?%5Cpi"> is a (finite) measure</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5COmega%7D%20=%201"></p></li>
</ul>
</section>
<section id="arbitrage" class="level2">
<h2 class="anchored" data-anchor-id="arbitrage">Arbitrage</h2>
<p>Let’s compute the original hedging quantity that sums to zero in a fair betting situation. Recall that <img src="https://latex.codecogs.com/png.latex?%5Chat%7BG%7D_%7B%5COmega%7D"> is the gain of placing a bet on <img src="https://latex.codecogs.com/png.latex?A"> combined with a bet on <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BA%7D"> with the proper stakes. In an unfair (real) situation, this does not sum to zero.</p>
<p>If the booksum <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%20%3C%201">, then <img src="https://latex.codecogs.com/png.latex?%5Chat%7BG%7D_%7B%5COmega%7D%20%3E%200">: this means that one can bet on both outcomes and make a sure profit, according to any outcome. This is called an <em>arbitrage</em> (or <em>arb</em>).</p>
<p>Thus, a bookie needs to set the booksum <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%20%5Cgeq%201">, otherwise this will lead to quick bankruptcy: everyone <em>in the know</em> will bet on both outcomes (with the proper stakes ratio).</p>
<p>Moreover, the higher the booksum, the higher the profits for bookies since <img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D%20=%201%20-%20%5Cpi_%7B%5COmega%7D">. Recall that your loss is the bookie’s gain.</p>
<p>However, since maximizing the booksum means minimizing the average odds, informed bettors will likely place their bets with another bookie. Thus, for the bookie, it is all about maximizing the booksum while still attracting customers, keeping in mind that competitors do the same.</p>
</section>
<section id="value-betting" class="level2">
<h2 class="anchored" data-anchor-id="value-betting">Value betting</h2>
<p>Let us now consider this <em>smart bookie</em> configuration: <img src="https://latex.codecogs.com/png.latex?%5Cpi_%7B%5COmega%7D%20%3E%201">. Is there still any opportunity to make money as a bettor (and lose money as a bookie)? Well, <em>probably</em>. Recall the expected gains for a given event:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D_p%5BF_A%5D%20=%20p_A%20-%20%5Cpi_A"></p>
<p>This means that if you trust your belief <img src="https://latex.codecogs.com/png.latex?p_A">, and that it is higher than the one the bookie offers you:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?p_A%20%3E%20%5Cpi_A%20=%20(%201%20+%20%5Cepsilon_A)%20%5Chat%7Bp%7D_A"> with <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A"> the margin</li>
</ul>
<p>then you should bet. In practice, you should probably look for differences that are wide enough for you to bet.</p>
<p>However, in this case there is no sure bet, and hence no sure profit = there is only an estimate of the expectation. Hence, this is much riskier than an arbitrage. In order to make a profit based on this, you need to bet on many opportunities where you find a positive expectation, so that on average your gain should be positive. If you have a robust estimation method, then you will make a profit if you size your bets correctly (see e.g.&nbsp;the Kelly criterion). On the other hand, if you your estimates are often off compared to the bookies’, then you will eventually go bankrupt (see the gambler’s ruin).</p>
</section>
<section id="negative-bets" class="level2">
<h2 class="anchored" data-anchor-id="negative-bets">Negative bets</h2>
<p><em>Negative</em> bets are simply when you take the role of the bookie: you offer someone else a bet. Then, their gain is exactly your loss, and vice-versa. Hence, placing a negative bet is simply the opposite of the positive bet that the person takes. But what if you cannot offer someone else a bet at the price you buy it? It is hard to find either a bookie that allows you to take his place, or an exchange that does not take commissions.</p>
<p>Well, we have to resort to <img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D">. Recall that, in the general case:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D%20=%20F_A%20+%20F_%7B%5Coverline%7BA%7D%7D%20=%201%20-%20%5Chat%7B%5Cpi%7D_%7B%5COmega%7D">.</p>
<p>If the <strong>fair situation</strong>, <img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D%20=%200">. Thus, we can simply place the opposite bet with the proper stake:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?-%20F_A%20=%20F_%7B%5Coverline%7BA%7D%7D"></li>
</ul>
<p>However, in the <strong>unfair situation</strong>, considering a smart bookie, we have:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F_%7B%5Coverline%7BA%7D%7D%20=%20-%20F_A%20-%20%5Cepsilon_A"></li>
</ul>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A%20=%20%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%20-%201%20%3E%200">. This is the bookie’s margin: you cannot hedge yourself at net zero cost. You must pay the price to the bookie. You can clearly see that again by placing a sure net negative bet:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_%7B%5Coverline%7BA%7D%7D%20=%20-%20%5Cepsilon_A%20%3C%200"></li>
</ul>
<p>Note that if the bookie is not smart, or that you are dealing with several bookies at the same time, treating them as one bookie offering the highest odds for any event, you might see an opportunity where <img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D%20%3E%200">, that is <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A%20%3C%200">. This quantity becomes you sure net gain in this arbitrage case.</p>
<p>See the appendix for the construction of surebets with the help of reversing (negative) bets.</p>
</section>
<section id="dutch-book" class="level2">
<h2 class="anchored" data-anchor-id="dutch-book">Dutch book</h2>
<p>Provided that we can back or lay any bet offered to us, that is, for a given odds, play the role of the punter <em>or</em> the role of the bookie. In the case of incoherent odds, a bettor can build a <em>dutch book</em> against the bookie, that is a set of bets that constitutes a surebet.</p>
<!-- 
#### Compatible events

[WIP]

If we don't necessarily have $A \cap B \neq \emptyset$, then we need to set the pseudo-bet:

### Product

[WIP]

Consider $A {\perp \!\!\! \perp} B$. We cannot simply bet on the intersection of two bets, we can only bet on the union. We need to use set theory: $A \cap B = \overline{\overline{A} \cup \overline{B}}$. Now, let use build a composite bet on this event.

Let :

* $C := A \cap B$.

* $\hat{\pi}_C := \pi_A \pi_B$

Note that $\overline{C} = \overline{A} \cup \overline{B}$.

* $\pi_C G_C = - \pi_{\overline{C}} G_{\overline{C}} - \epsilon$

Recall the coherent case, and compute the difference:

* $\pi_C G_C - \pi_A \pi_B [(G_A + 1)(G_B + 1) - 1] = \pi_A \pi_B - \pi_C$ -->
</section>
<section id="thinking-like-a-bookie" class="level2">
<h2 class="anchored" data-anchor-id="thinking-like-a-bookie">Thinking like a bookie</h2>
<!-- 
Let us consider the case of an obviously _mispriced_ event: imagine a game of basketball, where there is no draw. Assume that the underdog has lower odds than the favourite: if $A := [ \text{The favourite wins} ]$, $o_A > o_{\overline{A}}$ thus $\pi_A < \pi_{\overline{A}}$. The "true" probabilities are such that $p_A > p(\overline{A})$. Then you just need to bet on the favourite:  -->
<p>If we think like a bookie, here are our tasks in order to thrive:</p>
<ul>
<li>Estimate the true probabilities as precisely as possible</li>
<li>Set the odds lower than the estimated ones (inverse odds are thus higher than estimated probabilities)</li>
<li>Attract bettors by offering high enough odds</li>
<li>Minimize the risk</li>
</ul>
<!-- This could be expressed as:
* $\miF_{\pi_A, \pi_{\overline{A}}} \hat{p}_A - \pi_A + \hat{p}_{\overline{A}} - \pi_{\overline{A}}$
* $\max_{\pi_A, \pi_{\overline{A}}}|\pi_A + \pi_{\overline{A}}|$ -->
<p>As a bookie, you want to minimize the <em>risk</em>, i.e.&nbsp;the largest sum of money you could potentially lose. Consider for example that a large number of people do bet on an unlikely outcome, with high odds, and this event realizes. In order to avoid this, the bookie can intentionally skew the odds while realizing that many people (or few people with large sums of money) are betting on this outcome. This can also mean that there is a value opportunity for bettors, i.e.&nbsp;that the odds are not set properly. In this example, the bookie can monitor the volume of bets, while computing the risk, and decrease it when it is too high by decreasing the corresponding odds and increasing the odds of opposite events, in order to attract bettors. He can also place bets at another bookie in order to hedge.</p>
<!-- 
The above discussion is valid for bookies, but also for players that want to play seriously, that is invest large sums of money in the market. Even if you can't set the odds, eventually it is the same: you have your estimates of the true probability, another player has his, and you confront them. Bookies that set the odds need to be careful to balance their books in order not to create a _dutch book_ against them. -->
</section>
<section id="implied-probability" class="level2">
<h2 class="anchored" data-anchor-id="implied-probability">Implied probability</h2>
<p>We want to know the <em>implied probabilities</em> of the odds, the probability estimated by the bookie. We actually need to reverse engineer the recipe that they use in order to maximize profit based on their estimate of the true probabilities, in order to recover them. This is basically an inverse problem.</p>
<p>Denote:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cpi_A%20=%20(1%20+%20%5Cepsilon_A)%20%5Chat%7Bp%7D_A"></li>
</ul>
<p>with <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bp%7D_A"> the bookie’s (coherent) belief, <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A"> the bookie’s margin on event A. The bookie certainly hopes that <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A%20%3E%200">, and tries to do that.</p>
<section id="basic-normalization" class="level3">
<h3 class="anchored" data-anchor-id="basic-normalization">Basic normalization</h3>
<p>An easy technique to get a probability from consistent (and unfair) odds is to normalize the inverse odds:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Chat%7Bp%7D%20:=%20%5Cfrac%7B1%7D%7B%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%7D%5Cpi"></li>
</ul>
<p>This assumes that the <img src="https://latex.codecogs.com/png.latex?%5Cforall%20A,%20%5Cepsilon_A%20=%20%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%20-%201">: the bookies’ margin is uniform on all outcomes, which is probably false in real situations. However, this provides a first guess that is easy to compute.</p>
<p>If we assume that <img src="https://latex.codecogs.com/png.latex?%5Cpi"> is additive, it is immediate that we have <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bp%7D_%7B%5COmega%7D%20=%201"> and thus <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bp%7D"> is a probability measure.</p>
</section>
<section id="shins-model" class="level3">
<h3 class="anchored" data-anchor-id="shins-model">Shin’s model</h3>
<p>See Shin 1991, Strumbelj 2014 for further analysis.</p>
<!-- ## Sizing bets

### Kelly criterion

[Todo] -->
</section>
</section>
<section id="handling-multiple-odds" class="level2">
<h2 class="anchored" data-anchor-id="handling-multiple-odds">Handling multiple odds</h2>
<p>If we have access to multiple odds at a given time, we have virtually access to all the odds in between. Why would you bet in-between though, since you should take the highest odds?</p>
<p>Also, this trick might be useful to <em>reduce</em> a position to one equivalent bet, or <em>expand</em> a single bet into multiple ones. This might also be useful to test a risk assessment system (with e.g.&nbsp;a property-based testing framework).</p>
<p>A more realistic situation arises when we place multiple bets on the same event at different times, while odds have been varying (in-play betting for example). If you did spot value at time <img src="https://latex.codecogs.com/png.latex?t"> and the odds move in your favour, then you should still bet at time <img src="https://latex.codecogs.com/png.latex?t%20+%20%5Cdelta%20t">. In order to simplify you position, you can virtuallly reduce your two bets to one equivalent bet at odds in-between.</p>
<section id="single-event---convexity" class="level3">
<h3 class="anchored" data-anchor-id="single-event---convexity">Single event - Convexity</h3>
<p>Observe that for any given event, <img src="https://latex.codecogs.com/png.latex?%5C%7BF_A%5E%7B%5Cpi%7D%5C%20%7C%20%5Cpi%20%5Cge%200%20%5C%7D"> is a convex set:</p>
<p>For <img src="https://latex.codecogs.com/png.latex?%5Calpha%20%5Cin%20%5B0,%201%5D">, we have:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Calpha%20F_A%5E%7B%5Cpi_1%7D%20+%20(1%20-%20%5Calpha)%20F_A%5E%7B%5Cpi_2%7D%20=%20F_A%5E%7B%5Calpha%20%5Cpi_1%20+%20(1%20-%20%5Calpha)%5Cpi_2%7D">.</p>
<p>This gives us a useful formula to combine gains, betting on mutiple odds at once:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bequation*%7D%0A%5Cbegin%7Bsplit%7D%0Ax%20F%5E%7B%5Cpi_1%7D_A%20+%20y%20F_A%5E%7B%5Cpi_2%7D%20&amp;%20=%20(x%20+%20y)%20(%5Cfrac%7Bx%7D%7Bx%20+%20y%7DF_A%5E%7B%5Cpi_1%7D%20+%20%5Cfrac%7By%7D%7Bx+y%7DF_A%5E%7B%5Cpi_2%7D)%20%5C%5C%0A&amp;%20=%20(x%20+%20y)%20F_A%5E%7B%5Calpha%20%5Cpi_1%20+%20(1%20-%20%5Calpha)%20%5Cpi_2%7D%0A%5Cend%7Bsplit%7D%0A%5Cend%7Bequation*%7D%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Calpha%20=%20%5Cfrac%7Bx%7D%7Bx+y%7D">.</p>
<p>In practice, you can also use <img src="https://latex.codecogs.com/png.latex?G"> with the odds:</p>
<p><img src="https://latex.codecogs.com/png.latex?x%20G%5E%7Bo_1%7D%20+%20y%20G%5E%7Bo_2%7D%20=%20(x%20+%20y)%20G%5E%7B%5Calpha%20o_1%20+%20(1%20-%20%5Calpha)%20o_2%7D"></p>
</section>
<section id="single-event-and-its-complimentary" class="level3">
<h3 class="anchored" data-anchor-id="single-event-and-its-complimentary">Single event and its complimentary</h3>
<p>For <img src="https://latex.codecogs.com/png.latex?%5Calpha%20%5Cin%20%5B0,%201%5D">, we have:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bequation*%7D%0A%5Cbegin%7Bsplit%7D%0A(1%20-%20%5Calpha)%20F_A%5E%7B%5Cpi%7D%20+%20%5Calpha%20F_%7B%5Coverline%7BA%7D%7D%5E%7B%5Cpi%7D%20&amp;%20=%20(1%20-%202%20%5Calpha)%20F_A%5E%7B%5Cpi%7D%20-%20%5Calpha%20%5Cepsilon_A%20%5C%5C%0A&amp;%20=%20(2%20%5Calpha%20-%201)%20F_%7B%5Coverline%7BA%7D%7D%5E%7B%5Cpi%7D%20-%20(1%20-%20%5Calpha)%20%5Cepsilon_A%0A%5Cend%7Bsplit%7D%0A%5Cend%7Bequation*%7D%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A%20=%20%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%20-%201"></p>
<p>In the <em>smart bookie</em> setting, <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_A%20%3E%200">.</p>
<p>for arbitraty amounts <img src="https://latex.codecogs.com/png.latex?x,%20y%20%5Cgeq%200">, we have:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bequation*%7D%0A%5Cbegin%7Bsplit%7D%0Ax%20F_A%5E%7B%5Cpi%7D%20+%20y%20F_%7B%5Coverline%7BA%7D%7D%5E%7B%5Cpi%7D%20&amp;%20=%20(x%20-%20y)F_A%5E%7B%5Cpi%7D%20-%20y%20%5Cepsilon_A%20%5C%5C%0A&amp;%20=%20(y%20-%20x)F_%7B%5Coverline%7BA%7D%7D%5E%7B%5Cpi%7D%20-%20x%20%5Cepsilon_A%0A%5Cend%7Bsplit%7D%0A%5Cend%7Bequation*%7D%0A"></p>
</section>
<section id="combining-both" class="level3">
<h3 class="anchored" data-anchor-id="combining-both">Combining both</h3>
<p>We arrive thus at the combining formula:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bequation*%7D%0A%5Cbegin%7Bsplit%7D%0Ax%20F_A%5E%7B%5Cpi_1%7D%20+%20y%20F_A%5E%7B%5Cpi_2%7D%20+%20z%20F_%7B%5Coverline%7BA%7D%7D%5E%7B%5Cpi_3%7D%20&amp;%20=%20(x%20+%20y%20-%20z)%20F_A%5E%7B%5Calpha%20%5Cpi_1%20+%20(1%20-%20%5Calpha)%20%5Cpi_2%7D%20-%20z%20%5B%5Calpha%20%5Cpi_1%20+%20(1%20-%20%5Calpha)%20%5Cpi_2%20+%20%5Cpi_3%20-%201%5D%0A%5Cend%7Bsplit%7D%0A%5Cend%7Bequation*%7D%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Calpha%20=%20%5Cfrac%7Bx%7D%7Bx%20+%20y%7D"></p>
<!-- ## Odds varying through time

[todo]

$\pi_A^t$ represents the inverse-odds through time. -->
</section>
</section>
<section id="appendix" class="level2">
<h2 class="anchored" data-anchor-id="appendix">Appendix</h2>
<section id="measure-theory" class="level3">
<h3 class="anchored" data-anchor-id="measure-theory">Measure theory</h3>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmu%20(A)%20=%20%5Cint_A%20%5Cmathbf%7B1%7D_Ad%5Cmu"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?1_%7BA%20%5Ccap%20B%7D%20=%201_A%201_B"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?1_%7BA%20%5Ccup%20B%7D%20=%201_A%20+%201_B%20-%201_%7BA%20%5Ccap%20B%7D"></p></li>
</ul>
</section>
<section id="probability-theory" class="level3">
<h3 class="anchored" data-anchor-id="probability-theory">Probability theory</h3>
<p>A probablity <img src="https://latex.codecogs.com/png.latex?p"> defined on any <img src="https://latex.codecogs.com/png.latex?A%20%5Csubset%20%5COmega"> is a finite measure that satisfies the property <img src="https://latex.codecogs.com/png.latex?p_%7B%5COmega%7D%20=%201">.</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?E_p%5B1_A%5D%20=%20p_A"></p></li>
<li><p>We say <img src="https://latex.codecogs.com/png.latex?A%20%7B%5Cperp%20%5C!%5C!%5C!%20%5Cperp%7D%20B"> for <img src="https://latex.codecogs.com/png.latex?p"> iff <img src="https://latex.codecogs.com/png.latex?p_%7BA%20%5Ccap%20B%7D%20=%20p_A%20p_B"> i.e.&nbsp;<img src="https://latex.codecogs.com/png.latex?E_p%5B%5Cmathbf%7B1%7D_A%20%5Cmathbf%7B1%7D_B%5D%20=%20%5Cmathbb%7BE%7D_p%5B%5Cmathbf%7B1%7D_A%5D%20%5Cmathbb%7BE%7D_p%5B%5Cmathbf%7B1%7D_B%5D"></p></li>
<li><p>More generally, we say <img src="https://latex.codecogs.com/png.latex?X%20%7B%5Cperp%20%5C!%5C!%5C!%20%5Cperp%7D%20Y"> for <img src="https://latex.codecogs.com/png.latex?p"> iff <img src="https://latex.codecogs.com/png.latex?E_p%5BXY%5D%20=%20E_p%5BX%5D%20E_p%5BY%5D"></p></li>
</ul>
</section>
<section id="basic-bets-properties" class="level3">
<h3 class="anchored" data-anchor-id="basic-bets-properties">Basic bets properties</h3>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cpi%20%3E%200">, <img src="https://latex.codecogs.com/png.latex?o%20%3E%201"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D_p%5BG_A%5D%20=%20%5Cfrac%7Bp_A%7D%7B%5Cpi_A%7D%20-%201"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BV%7D_p(G_A)%20=%20%5Cfrac%7Bp_A%20(1%20-%20p_A)%7D%7B%5Cpi_A%5E2%7D"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D_p%5BF_A%5D%20=%20p_A%20-%20%5Cpi_A"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BV%7D_p(F_A)%20=%20p_A%20(1%20-%20p_A)"></p></li>
</ul>
</section>
<section id="surebets-on-an-incoherent-book" class="level3">
<h3 class="anchored" data-anchor-id="surebets-on-an-incoherent-book">Surebets on an incoherent book</h3>
<section id="incompatible-events" class="level4">
<h4 class="anchored" data-anchor-id="incompatible-events">Incompatible events</h4>
<p>Let’s consider the case when the odds are incoherent. In the case of <img src="https://latex.codecogs.com/png.latex?A%20%5Ccap%20B%20=%20%5Cemptyset">, if you chose to bet both on <img src="https://latex.codecogs.com/png.latex?A"> and <img src="https://latex.codecogs.com/png.latex?B"> separately, then you can actually (implicitly) bet on <img src="https://latex.codecogs.com/png.latex?A%20%5Ccup%20B"> with the proper stakes proportion. We assume that the bookie offers (incoherent) odds for <img src="https://latex.codecogs.com/png.latex?A">, <img src="https://latex.codecogs.com/png.latex?B">, <img src="https://latex.codecogs.com/png.latex?A%20%5Ccup%20B"> and <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BA%20%5Ccup%20B%7D%20=%20%5Coverline%7BA%7D%20%5Ccap%20%5Coverline%7BB%7D">. This situation never arises in the scope of binary outcomes sports but rather typically in football (soccer): <img src="https://latex.codecogs.com/png.latex?%5C%7BA,%20B,%20%5Coverline%7BA%20%5Ccup%20B%7D%5C%7D%20%5Ccong%20%5C%5C%7BHome,%20Away,%20Draw%5C%5C%7D"></p>
<p>We introduce proper notation:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?C%20:=%20A%20%5Ccup%20B"></li>
</ul>
<p>We want to compare <img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_B"> and <img src="https://latex.codecogs.com/png.latex?F_C">.</p>
<section id="one-side" class="level5">
<h5 class="anchored" data-anchor-id="one-side">One side</h5>
<p>We compute the theoretical sum of a positive and a negative bet:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_B%20-%20F_C%20=%20%5Cpi_C%20-%20(%5Cpi_A%20+%20%5Cpi_B)"></p>
<p>If the “coherent” version of the inverse odds for <img src="https://latex.codecogs.com/png.latex?C"> are low enough, there is an opportunity. One has to bet with the inverse formula:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_%7B%5Coverline%7BC%7D%7D%20=%20-%20F_C%20+%20(%5Cpi_C%20+%20%5Cpi_%7B%5Coverline%7BC%7D%7D%20-%201)"></p>
<p>Consider the betting opportunity:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5COmega%7D%20:=%20F_A%20+%20F_B%20+%20F_%7B%5Coverline%7BC%7D%7D%20=%20(%5Cpi_C%20-%20%5Cpi_A%20-%20%5Cpi_B%20)-%20(%5Cpi_C%20+%20%5Cpi_%7B%5Coverline%7BC%7D%7D%20-%201)%20=%201%20-%20(%5Cpi_A%20+%20%5Cpi_B%20+%20%5Cpi_%7B%5Coverline%7BC%7D%7D)"></li>
</ul>
<p>We have a sure (triple) bet. We will make a profit if and only if this quantity is positive, i. e.:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cpi%7D_%7B%5COmega%7D%20:=%20%5Cpi_A%20+%20%5Cpi_B%20+%20%5Cpi_%7B%5Coverline%7BC%7D%7D%20%3C%201"></li>
</ul>
<p>This is exactly the same arbitrage formula as before, but with three outcomes.</p>
</section>
<section id="the-other-side" class="level5">
<h5 class="anchored" data-anchor-id="the-other-side">The other side</h5>
<p>This time we target <img src="https://latex.codecogs.com/png.latex?F_C%20-%20F_A%20-%20F_B">. In this case, we place the following bets:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_%7B%5Coverline%7BA%7D%7D%20=%20-%20F_A%20-%20(%5Cpi_A%20+%20%5Cpi_%7B%5Coverline%7BA%7D%7D%20-%201)"></p>
<p>We use exactly the same formula for <img src="https://latex.codecogs.com/png.latex?B">, and we get a betting opportunity:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_%7B%5Coverline%7BA%7D%7D%20+%20F_%7B%5Coverline%7BB%7D%7D%20+%20F_C%20=%202%20-%20(%5Cpi_%7B%5Coverline%7BA%7D%7D%20+%20%5Cpi_%7B%5Coverline%7BB%7D%7D%20+%20%5Cpi_C)"></p>
</section>
</section>
<section id="general-case" class="level4">
<h4 class="anchored" data-anchor-id="general-case">General case</h4>
<p>This time, we do not assume <img src="https://latex.codecogs.com/png.latex?A%20%5Ccap%20B%20=%20%5Cemptyset">.</p>
<p>We leverage the following:</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?1_%7BA%20%5Ccap%20B%7D%20=%201_A%20+%201_B%20-%201_%7BA%20%5Ccup%20B%7D"> i.e.&nbsp;<img src="https://latex.codecogs.com/png.latex?1_D%20=%201_A%20+%201_B%20-%201_C"></p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?1_%7BA%20%5Ccap%20B%7D%20=%201_A%201_B">.</p></li>
</ul>
<p>Let <img src="https://latex.codecogs.com/png.latex?D%20:=%20A%20%5Ccap%20B">.</p>
<p>We essentially want to compare <img src="https://latex.codecogs.com/png.latex?F_D"> and <img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_B%20-%20F_C">.</p>
<p>Then, consider the following pseudo-bet with the pseudo-odds <img src="https://latex.codecogs.com/png.latex?%5Cpi_A%20+%20%5Cpi_B%20-%20%5Cpi_C">:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_B%20-%20F_C%20=%20%5Cmathbf%7B1%7D_A%20+%20%5Cmathbf%7B1%7D_B%20-%20%5Cmathbf%7B1%7D_C%20-%20(%5Cpi_A%20+%20%5Cpi_B%20-%20%5Cpi_C)"></p>
<p>Now we need to compare this to</p>
<p><img src="https://latex.codecogs.com/png.latex?F_D%20=%20%5Cmathbf%7B1%7D_A%20+%20%5Cmathbf%7B1%7D_B%20-%20%5Cmathbf%7B1%7D_C%20-%20%5Cpi_D"></p>
<section id="one-side-1" class="level5">
<h5 class="anchored" data-anchor-id="one-side-1">One side</h5>
<p>First, we target <img src="https://latex.codecogs.com/png.latex?F_A%20+%20F_B%20-%20F_C%20-%20F_D">. In order to actually place an approximation of this bet, we need to reverse the two negative bets <img src="https://latex.codecogs.com/png.latex?-F_C"> and <img src="https://latex.codecogs.com/png.latex?-F_D">.</p>
<p>First we have:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_%7B%5Coverline%7BC%7D%7D%20=%20-%20%5Cmathbf%7B1%7D_C%20+%20%5Cpi_C%20+%201%20-%20(%5Cpi_C%20+%20%5Cpi_%7B%5Coverline%7BC%7D%7D)%20=%20-%20%5Cmathbf%7B1%7D_C%20+%201%20-%20%5Cpi_%7B%5Coverline%7BC%7D%7D"></p>
<p>With the same reasoning, we have use <img src="https://latex.codecogs.com/png.latex?F_%7B%5Coverline%7BD%7D%7D%20=%20-%20F_%7BD%7D%20-%20(%5Cpi_D%20+%20%5Cpi_%7B%5Coverline%7BD%7D%7D%20-%201)%20=%20-%20(%5Cmathbf%7B1%7D_A%20+%20%5Cmathbf%7B1%7D_B%20-%20%5Cmathbf%7B1%7D_C)%20+%201%20-%20%5Cpi_%7B%5Coverline%7BD%7D%7D">.</p>
<p>We get the sure bet combinations:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5Cepsilon_1%7D%20:=%20F_A%20+%20F_B%20+%20F_%7B%5Coverline%7BC%7D%7D%20+%20F_%7B%5Coverline%7BD%7D%7D%20=%202%20-%20(%5Cpi_A%20+%20%5Cpi_B%20-%20%5Cpi_%7B%5Coverline%7BC%7D%7D%20-%20%5Cpi_%7B%5Coverline%7BD%7D%7D)"></p>
</section>
<section id="the-other-side-1" class="level5">
<h5 class="anchored" data-anchor-id="the-other-side-1">The other side</h5>
<p>Consider the opposite quantity:</p>
<p><img src="https://latex.codecogs.com/png.latex?F_D%20+%20F_C%20-%20F_A%20-%20F_B"></p>
<p>We need to reverse the bets on A and B to get a sure bet:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Chat%7BF%7D_%7B%5Cepsilon_2%7D%20:=%20F_%7B%5Coverline%7BA%7D%7D%20+%20F_%7B%5Coverline%7BB%7D%7D%20+%20F_C%20+%20F_D%20=%202%20-%20(%5Cpi_C%20+%20%5Cpi_D%20-%20%5Cpi_%7B%5Coverline%7BA%7D%7D%20-%20%5Cpi_%7B%5Coverline%7BB%7D%7D)"></p>
<p>Provided we can place bets on <img src="https://latex.codecogs.com/png.latex?A">, <img src="https://latex.codecogs.com/png.latex?B">, <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BC%7D%20=%20%5Coverline%7BA%20%5Ccup%20B%7D%20=%20%5Coverline%7BA%7D%20%5Ccap%20%5Coverline%7BB%7D"> and <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BD%7D%20=%20%5Coverline%7BA%20%5Ccap%20B%7D%20=%20%5Coverline%7BA%7D%20%5Ccup%20%5Coverline%7BB%7D">, we need to check the two above-mentioned quantities. However, it is usually not possible to compose bets in such a fashion: the bookies does not offer any bet on <img src="https://latex.codecogs.com/png.latex?%5Coverline%7BA%7D%20%5Ccup%20%5Coverline%7BB%7D">.</p>


</section>
</section>
</section>
</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2021,
  author = {Guy, Horace},
  title = {Mathematics of Bookmaking},
  date = {2021-09-06},
  url = {https://blog.horaceg.xyz/posts/betting-theory/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2021" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2021. <span>“Mathematics of Bookmaking.”</span> September
6. <a href="https://blog.horaceg.xyz/posts/betting-theory/">https://blog.horaceg.xyz/posts/betting-theory/</a>.
</div></div></section></div> ]]></description>
  <category>math</category>
  <category>probability</category>
  <category>betting</category>
  <category>bookmaking</category>
  <guid>https://blog.horaceg.xyz/posts/betting-theory/</guid>
  <pubDate>Mon, 06 Sep 2021 00:00:00 GMT</pubDate>
</item>
<item>
  <title>The need for speed</title>
  <dc:creator>Horace Guy</dc:creator>
  <link>https://blog.horaceg.xyz/posts/need-for-speed/</link>
  <description><![CDATA[ 





<p>Lately, I have been interested in the foundations of machine learning (ML) in a pedagogical perspective. A first step to grasp the power of ML is gradient descent. So, I was determined to demonstrate the way it works with an interactive app.</p>
<section id="python-and-jupyter-world" class="level2">
<h2 class="anchored" data-anchor-id="python-and-jupyter-world">Python and Jupyter world</h2>
<p>The first thing I made used <a href="https://jupyter.org/">Jupyter</a>, and a nifty package called <a href="https://github.com/voila-dashboards/voila"><em>Voilà</em></a>, that turns a notebook into a standalone app. For automatic differentiation, I used vanilla <a href="https://github.com/google/jax/">JAX</a>. For the front-end, I used ipywidgets, <a href="https://github.com/bqplot/bqplot">bqplot</a> and matplotlib [^ipympl].</p>
<p>Bqplot is amazing for classic plots, plays very well with ipywidgets for reactivity. I was unable to find a way to make it display a vector field nicely though, so I had to use interactive matplotlib, which is heavier and slower, to display the gradient of the loss.</p>
<p>I deployed this on <a href="https://www.heroku.com/">Heroku</a> with a small Procfile:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">web:</span> voila <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--port</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$PORT</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--no-browser</span> techni.ipynb</span></code></pre></div></div>
<p>And boom! It was live!</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/need-for-speed/voila-app.png" class="img-fluid figure-img"></p>
<figcaption>voila app</figcaption>
</figure>
</div>
<section id="speed-of-light" class="level3">
<h3 class="anchored" data-anchor-id="speed-of-light">Speed of light</h3>
<p>In this setting, every state change triggers a round-trip to the server, through <a href="https://en.wikipedia.org/wiki/WebSocket">websockets</a> (Jupyter is based on <a href="https://www.tornadoweb.org/en/stable/">Tornado</a>). Locally, this works great since there is zero latency between my computer and my browser (running on… my computer).</p>
<p>However, whenever I browsed the Heroku-hosted app, there were significant lags. There is now a latency due to the speed of light through the wire, which ranges from 100ms to 600ms depending on your region of the world.</p>
</section>
</section>
<section id="svelte-to-the-rescue" class="level2">
<h2 class="anchored" data-anchor-id="svelte-to-the-rescue">Svelte to the rescue</h2>
<p>I needed a proper front-end, so that every action on the client does not require a round-trip to the server. I also would have more control over how to react once a new payload is sent from the server.</p>
<p>Moreover, the dataviz I made were functional and did a good job of explaining the concept. However, they were a bit primitive: no transition nor interpolation between distinct states, which provides a bit of a boorish experience.</p>
<p>I decided to leverage the new hot JS framework <a href="https://svelte.dev">Svelte</a>, with reactivity built-in. So, I surgically removed the JAX bits of the code and moves them to a back-end built with <a href="https://fastapi.tiangolo.com/">FastAPI</a>, that the Svelte front-end calls. The front-end is hosted on <a href="https://developers.cloudflare.com/pages/">Cloudflare pages</a>, and calls the API whenever the hyperparameters of the descent are modified.</p>
<p>For plotting, I used the experimental <a href="https://pancake-charts.surge.sh/"><code>pancake</code></a> library by the creator of Svelte, Rich Harris. Thanks to Svelte <code>tweened</code> and <code>spring</code> <a href="https://svelte.dev/docs#svelte_motion">motion functions</a>, the transitions are smooth and easy on the eyes.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.horaceg.xyz/posts/need-for-speed/gradfront-alpha-vertical.png" class="img-fluid figure-img"></p>
<figcaption>First Svelte version</figcaption>
</figure>
</div>
</section>
<section id="going-world-scale" class="level2">
<h2 class="anchored" data-anchor-id="going-world-scale">Going world-scale</h2>
<p>There is a shortcoming of this setting: the latency is still there. It is handled in a better way with async javascript, but if the API is hosted in, say Frankfurt, then a user in Sydney is not going to have a very snappy experience.</p>
<p>I was wondering: could it be possible to replicate the backend in a few data centers, and route users to the closest one in order to minimize the latency? Well, after a few days of research I found that this is exactly what <a href="https://fly.io">Fly</a> provides. So, I went a bit crazy and replicated the back-end in 10 regions and thoroughly tested the latency with a VPN (accounting for the additional round-trips). It seemed to work well enough!</p>
<p>Now I had</p>
<ul>
<li>A front-end replicated on a CDN on 200+ locations</li>
<li>A back-end replicated at 10 locations</li>
</ul>
<p>At this point, you might think that this is a tad overkill for a simple pedagogical app, and I don’t think you would be very wrong.</p>
</section>
<section id="to-the-browser-tensorflowjs" class="level2">
<h2 class="anchored" data-anchor-id="to-the-browser-tensorflowjs">To the browser: TensorflowJS</h2>
<p>I ported the JAX code to <a href="https://www.tensorflow.org/js">TensorflowJS</a>, on <a href="https://observablehq.com/@horaceg/gradient-descent">Observable</a> at first and then directly in the Svelte front-end. Now I have a fully client-side app: no server, yay !?</p>
<p>For the fast 30-step gradient descent, this works great and the computing happens instantly. However, when I add a simple neural network, I hit the performance wall: what takes 5ms in (already JIT-compiled) JAX, takes 300ms in the browser! yay…</p>
<ul>
<li><a href="https://gradfront.pages.dev/">Here</a> is the current master</li>
<li><a href="https://f36dfeb7.gradfront.pages.dev/">there</a> is the neural network version</li>
<li><a href="https://deploy-preview-1--gradient-descent.netlify.app/">there</a> the version with a contour plot of the gradient of the loss</li>
</ul>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{guy2021,
  author = {Guy, Horace},
  title = {The Need for Speed},
  date = {2021-09-03},
  url = {https://blog.horaceg.xyz/posts/need-for-speed/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-guy2021" class="csl-entry quarto-appendix-citeas">
Guy, Horace. 2021. <span>“The Need for Speed.”</span> September 3. <a href="https://blog.horaceg.xyz/posts/need-for-speed/">https://blog.horaceg.xyz/posts/need-for-speed/</a>.
</div></div></section></div> ]]></description>
  <category>machine learning</category>
  <category>dataviz</category>
  <category>speed</category>
  <category>deployment</category>
  <category>latency</category>
  <guid>https://blog.horaceg.xyz/posts/need-for-speed/</guid>
  <pubDate>Fri, 03 Sep 2021 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
