For a long time, I wanted to implement a Racket language module with a non-lispy surface syntax. Lox, from Crafting Interpreters, was an obvious candidate: I didn't want to invent a new syntax nor new semantics and I already ported the project to C#. In this case, my main objective was to leverage Racket language-building facilities while learning Racket, Scheme, and macros.

I attempted this already a few years ago, with little success. This time I dropped yacc and lex libraries and instead followed the approach from the book more closely, along with the C# version I had written earlier. The result is not especially functional in style: the scanner and parser are fairly imperative and rely on mutation, mainly because that made the code easier to port from the earlier implementations. Another big help came from LLMs, I used GitHub Copilot and it helped me fill some gaps in my knowledge and troubleshoot issues that I honestly didn't have enough competencies to solve.

I do not use GitHub Copilot autocomplete because that removes all the fun from coding but I "chatted" extensively and I also asked it to generate parts I was not particularly interested in, such as the colorer[^1].

The code is available on GitHub here. In the post I'll go through the implementation, highlighting all the parts that I consider interesting or helpful.

Recently, I stumbled upon this paper Synthesizing Object-Oriented and Functional Design to Promote Re-Use. The paper wants to provide a solution to the expression problem. The authors suggest an improved version of the visitor pattern that they call "extensible visitor pattern" which is essentially a combination of the visitor pattern with the factory method pattern.

While the paper and the expression problem statement don't explicitly mention SOLID principles, it looks like what they are really doing is exploring how to evolve a code base while respecting the Open/Closed Principle:

Software entities (like classes, methods, and functions) should be open for extension, but closed for modification.

Lately, I've been exploring various examples from Sutton and Barto's "Reinforcement Learning: An Introduction" book using C# and I already shared a few of them on this blog:

• Tic-tac-toe reinforcement learning with C#
• Ten armed testbed for the Bandit problem with C#
• Multi-armed bandit exercise 2.5 with C#

Today I'll be focusing on the gridworld example from chapter 3 of the book. The code is available in the existing repo as a new project. Gridworld is a simple example used to illustrate the Bellman equations and iterative policy evaluation. An excerpt from the book describes the environment:

The cells of the grid correspond to the states of the environment. At each cell, four actions are possible: north, south, east, and west, which deterministically cause the agent to move one cell in the respective direction on the grid. Actions that would take the agent off the grid leave its location unchanged, but also result in a reward of -1. Other actions result in a reward of 0, except those that move the agent out of the special states A and B. From state A, all four actions yield a reward of +10 and take the agent to A'. From state B, all actions yield a reward of +5 and take the agent to B'.

— Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., Chapter 3.

A few weeks back I became aware of an attempt to create a technology agnostic standard to send and receive webhooks securely. The proposal is here.

In my opinion, the standard is straightforward and non-controversial. The only part that is mandatory is the signing part, other parts are optional and somehow suggesting best practices.

In the standard-webhooks repo you'll find multiple sample implementations and some links to community implementations including one for C#. I was researching webhooks standards to implement my own packages and to play a bit with copilot agent on github. So I decided to go ahead and do it.

After thinking about it for some time I decided to add a comment system to the blog. What I hope is that I will get valuable feedback from readers, which will help me improve the content, my understanding of the topics I write about, and the overall quality of the blog.

How It Was Built​

I decided all of this while on a beach, in Italy and I didn't have access to a coding environment. So I decided to treat myself with a copilot pro license and try to get it implemented by the copilot agent.

Somehow it worked out, I realized quickly that I didn't give enough information to complete the task, but with some back-and-forth, we got there in the end.

I had the Ray Tracing in One Weekend book in my todo list for a long time and I had finally the time to go through it.

There is not much to say, except than rather unusually I used go instead of C# to go through the book. The code is a rather simple port from the C++ original, my only addition was a render method that used goroutines and channels to speed up the execution. It works fairly well and tops up the CPUs on my desktop machine.

The repo contains the image with 10 samples per pixel and the image with 100 samples per pixel. The quality difference is evident! To check it out you need to clone the repo, or download the raw files, and have a ppm image viewer installed.

Recently I tried to code the 10 armed testbed example from chapter 2 of Sutton and Barto Reinforcement Learning: an introduction book.

The chapter continues introducing new theory elements and strategies to improve the approach shown in the 10 armed example. In particular, one of the points is about non-stationary problems.

The 10 armed testbed was a stationary problem, the probability distributions of the different actions don't change over time. If you remember the sample, at the beginning of the round we computed 10 random values, those values are then used to be the mean of a normal distribution from which we will pick the rewards at each step. The constant part is that this normal distributions don't change from a step to another, they stay the same for the whole round execution.

The focus of the exercise is to understand how the estimated reward computation impacts the performance of the $\epsilon$-greedy strategy. In the 10 armed testbed, the estimate reward was computed averaging the rewards obtained from each action when selected. Note that this approach consider each reward with the same relative value, however in a non-stationary problem, where probability distributions change over time we would like to give more weight or importance to more recent rewards because they represent more realistically the current distribution the reward is generated from.

The text of the exercise is

Design and conduct an experiment to demonstrate the difficulties that sample-average methods have for nonstationary problems. Use a modified version of the 10-armed testbed in which all the $q_{*}(a)$ start out equal and then take independent random walks (say by adding a normally distributed increment with mean 0 and standard deviation 0.01 to all the $q_{*}(a)$ on each step). Prepare plots like Figure 2.2 for an action-value method using sample averages, incrementally computed, and another action-value method using a constant step-size parameter, $\alpha$ = 0.1. Use $\epsilon = 0.1$ and longer runs, say of 10,000 steps.

Figure 2.2 refers to the average reward graph and the best arm selection rate graph, the same graphs I produced in the previous post. The $\epsilon=0.1$ refers to the $\epsilon$-greedy strategy to be used, both in the case of sample averages and in the constant step-size parameter.

I'm continuing my attempt to reproduce examples from Reinforcement Learning: An Introduction book using C#.

In a previous post I reproduced the tic-tac-toe example with some improvements and clarification with respect to the original text. I think it's worth taking a look at it.

Today I'm reproducing the ten armed testbed for the Bandit problem, in particular I want to reproduce the two graphs showing the average reward improvements and the selection rate of the best arm.

The problem, as stated in the book is the following:

You are faced repeatedly with a choice among k different options, or actions. After each choice you receive a numerical reward chosen from a stationary probability distribution that depends on the action you selected. Your objective is to maximize the expected total reward over some time period, for example, over 1000 action selections, or time steps.

A few years ago I needed to troubleshoot some imports based on csv files and I had one big issue, depending on the language the user was using on Windows, Excel would export csv files with different separators. Being located in Italy, this means that the separator was ;. The same was true when trying to read csv with Excel, the separator is chosed based on the language of the OS.

I prefer to set up everything in English because I think it is easier to troubleshoot issues when the error is not localized to Italian.

So I couldn't read csv just opening them on Excel because of the separator, Excel also formats field depending on the content. It might hide leading zeros if it thinks a column is filled with numbers, dates are problematic and so on. I wanted to be able to see the actual content of the csv in a tabular format without fighting with Excel for the separator or for automatic formatting of columns.

I decided to go ahead and build a simple parser, hostable on a website that would read a csv and just print the values without any custom formatting and with the possibility to choose the separator. Of course today I would go with a vs code extension, I'm sure there are plenty doing something similar better than I did in a few hours but at the time this small parser was enough to support my need.

CSV is not a well defined or well agreed format, however there is a IEFT RFC that documents the format. Following that I implemented a parser in typescript that read a csv and creates an html table with the content manipulating the DOM directly.

The repo is this one: https://github.com/davidelettieri/yacv

Online demo: https://davidelettieri.github.io/yacv/

Just today I revisited the code, adding some basic tests and updating typescript version.

The asynchronous request reply pattern is useful in cases where an application has a client application running in a browser where the user starts an operation and is waiting for an immediate feedback and possibly a success notification. An issue arises only if the requested operation is long running, even a few seconds is unacceptable to return a feedback to a waiting user. Microsoft has a nice documentation about this pattern which essentially propose to have 3 endpoints:

• a POST endpoint to start the long running operation: POST /operations
• a GET endpoint to check for the status of the operation GET /operations/{id}/status
• a GET endpoint pointing to the created resource GET /operations/{id}

The actual path for the endpoints might vary and aren't explicitely provided in the Microsoft docs, I'm proposing those for the sake of the post, based on your requirements and the semantic of your API you might want have different ones. The interaction flow between the client and the backend is this one:

For example the POST /operations can return something like

{
  "id": 1,
  "links": [
    {
      "rel":"status_check",
      "href":"https://api.contoso.com/operations/1/status", 
      "action":"GET" 
    },
  ]
}

This flow allows the backend to perform the asynchronous/long running operation and any client can poll the status endpoint and notify the user when the operation is completed. The big advantage of this setup is that it is easy to implement, for example websockets could be an option but they are usually harder to use and you will probably need a third party library to use them. A downside of the polling approach is that a client can execute a lot of retries in a short amount of time, Microsoft suggests to provide a Retry-After header that the client should honor and wait for the indicated time, if you own the client this is an easy requirement to satisfy. While this pattern has a clear use case with browser based clients, for a machine to machine integration I would prefer implementing webhooks bot as a producer and as a consumer. However a company with a low tech maturity / capacity can implementing the polling approach very easily.

In my opinion there is another approach to this pattern, for a browser client, that avoids having the polling on the endpoint and doesn't require websockets. If you have a dedicated BFF for a client app this might be the solution for you.