Simple simulations for system builders (Marc Brooker). System designers care about questions like “How will the system behave under overload?” or “How sensitive is the design to latency?”. By “writing small simulators that simulate the behaviour of simple models”, Marc shows an approach to explore and reason about the possible answers to such questions.
Design docs at Google. When not to write a design doc: A clear indicator that a doc might not be necessary is when a design doc is an implementation manual, that doesn’t go into trade-offs, alternatives, and explanation of the decision-making; — write the actual application instead.
The complete history and strategy of Amazon.com and The complete history and strategy of AWS (Acquired podcast).
In the world, where the growth of storages is massively outpacing the improvements in the speed of the Internet, a database is the most sticky technology. That’s how AWS locks the enterprises in: it’s impossible to migrate the enterprise-grade of data out of AWS (“It took thirty years for Amazon to migrate off Oracle to AWS”).
Number of developers will tell you, they prefer building “a new thing”, rather than maintaining existing “legacy” projects.
While working on “green field”, “day zero” projects can definitely be fun, they rarely feel real to me. The “real” project lives in production. It has users. It has load. It demands improvement.
Explaining the unexplainable: buffers in PostgreSQL. Shared buffers are those, which’re’ “shared” between several DB sessions, i.e. data pages, indices, etc; local buffers, are “local” to a session, i.e. for temporal tables; temp buffers are for intermediate objects, i.e. when the DBMS does hashing and sorting.
Before jumping on the “best practices of software engineering” train, after reading the classics, or someone’s re-post on social media, ask what was the context, in which these “best practices” were established. Some of those made sense in the environment, where the product’s release cycle has been measured with months. But that doesn’t mean they make sense, when we ship/fix/ship/new-requirements/ship/update/ship the product several times every week.
Scott’s Bass Lessons and BassBuzz on YouTube. I’m learning how to play bass guitar 🎸 now. So far, these two channels were the most helpful with both the practice lessons, and the inspiration to move forward.
Event-driven architecture done right (Tim Berglung, Devoxx Poland 2021). Try to learn what you’re trying to do, before you elaborate the architecture, if uncertainty is very high, and you don’t know exactly what businesses are asking. Just start with something (architecturally) simple.
Acquired’s lessons learned from 200 company stories: optimism always wins (Sony); nothing can stop the will to survive (Nvidia); it’s never too late (TSMC); focus on what makes your beer taste better (Amazon and all utility companies); don’t be talent — own the business (Oprah, Tail Swift); you’ll get the partners you ask for (Amazon’s “If you’re not on my bus, get off”), and more.
Go 1.19beta1. As usual, lots of good improvements in the language’s runtime and the compiler, with one particularly interesting addition being the new “knob” runtime/debug.SetMemoryLimit.
Queries in PostgreSQL: 4. Index scan (Postgres Pro). An in-depth overview of how PostgreSQL decides if it will use an index.
One particular thing I had no idea about before I read the article was that “The Index Scan cost is highly dependent on the correlation between the physical order of the tuples on disk and the order in which the access method returns the IDs”. That explains several cases from my own experience, where postgres kept using “unexpected” sequential scans, after we added “another index” to the database.
Digital object identifier (DOI) (Wikipedia). A DOI is a persistent identifier or handle used to identify various objects uniquely. It aims to be “resolvable”, usually to some form of access to the information object to which the DOI refers. This is achieved by binding the DOI to metadata about the object, such as a URL. Thus, by being actionable and interoperable, a DOI differs from identifiers such as ISBNs, which aim only to identify their referents uniquely.
Platforms and Power (Acquired). “7 Powers” author Hamilton Helmer and Chenyi Shi (Strategy Capital), joined Acquired Podcast to discuss platform businesses, and how the “Power” framework applies to them.
Halfthings (Mat Ryer).
Building something for the users to play with, to touch, to feel, to break, makes all the difference and moves the conversations away from the meta.
Doing “one thing” or “building an MVP” can easily pull you into a “too much” for a validation phase. Build a “halfthing” instead.
Rethinking Classical Concurrency Patterns (Slides), Bryan C. Mills, GopherCon 2018. Brian demos the implementation of typical concurrency patterns, following the famous Go proverb: “Don’t communicate by sharing memory; share memory by communicating.”
GoLab Keynote, Bill Kennedy, GoLab 2018. Bill demoes the ways to reason about the performance of a Go application, with the help of Go runtime tracer.
The Scheduler Saga, Kavya Joshi, GopherCon 2018. Kavya shows what stands behind the magic of Go runtime.
Understanding Channels, Kavya Joshi, GopherCon 2017. Kavya explains how channels work under the hood.
Keeping the list here, in public, should help my future self, in a situation where I’m stuck with a mind-blocker, and I need to quickly pull out a piece of community wisdom from the backyards of my memory. The list isn’t meant to be complete, and I expect to add more links here, moving forward.
Did I miss any? Share your suggestions with me on Twitter.
Everything I list below are the steps I found relevant, after I’ve self-tested positive for COVID in Berlin, in the end of January 2022.
Friday, 28th January 2022
I woke up with some mild symptoms of cold. On the day before I worked from home, and only had a usual an hour-long walk around Prenzlauer Berg — Mitte after the workday. The rapid antigen test (Schnelltest) was negative.
While still working from home, I took a “quiet Friday” at work, to simply focused on some mundane routines.
I went for a short walk in the afternoon, bought some grocery, and headed home.
Saturday, 29th January
Didn’t feel anywhere better or worse: mostly had a runny nose and a dry throat. I didn’t want to wait in the line for a quick-test, so I’ve just walked around the city for an hour and went home.
Sunday, 30th January
It felt the same as it’d been the day before, although the dry throat started to feel a bit more intense. Walked around for an hour, came home, made some coffee ;)
A couple hours later I started to cough. I made a self-test — we still had some at home — and,
When Go code propagates an error, the following pattern is very popular:
fmt.Errorf("failed to find a parking slot: %w", err)
// Or
fmt.Errorf("could not call mom: %w", err)
These “could not”, “failed to”, “unable to” make sense when my mind is in the local context of the function, method or package. But, in most of the cases I have to deal with, it makes the resulting log message overloaded with informational garbage:
unable to ask about the cat: failed to call mom: failed to do request: Get https://: context canceled
While discussing this issue with a colleague, we came up with the following “better” strategy:
only the logger should express its attitude to the facts, using words “error”, “failed”, etc
the business code must operate only with facts, e.g. “call mom”.
My definition of “good coffee places” includes, although, by no mean limited by, offering a “not too fruity” (sometimes “chocolaty”) Filter coffee or a decent Americano.
One particularly confusing type of the resource for me was cpu. For example, in the manifest above, the my-app container declares a request for “100m” of the CPU. What does that mean?
Limits and requests for CPU resources are measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors.
CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine.
A misleading (but fairly common) mental modal amongst the developers, is that the application inside the Pod’s container is “boxed” by the container runtime, as if the application received a dedicated slice of the machine’s resources. That’s not exactly how it works. At least, it’s not if Kubernetes uses Docker as the underlying container runtime.
I find it easier to think about the requested resources as a way for an application to “hint” to Kubernetes scheduler about how much resources the application thinks it needs.
Kubernetes scheduler keeps the “accounting” for how much of total resources the tenants of the cluster requested, and how much the cluster’s machines have to offer. It’s important to stress that the “accounting” is done only base on the requests for the resources. That is, the scheduler doesn’t check if the container uses the resources it requested. If a container requested more than half of the node’s CPU resources, e.g. “1100m” on a 2 vCPU node, Kubernetes won’t deploy more than a single replica of this container to that node, nomatter if the application inside the container is idle.
Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node. Note that although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails.
What has been requested, has been booked.
Of course, “accounting” of the existing resources is only half of the story. In the same documentation about managing containers resources, they talk about the implementation details of what happens after a container gets scheduled to a node, refering to Docker’s documentation about CPU share constraint. This details for how Docker itself juggles the CPU shares between the running containers can be even more confusing but the interesting part to remember is the following:
The proportion [of CPU cycles] will only apply when CPU-intensive processes are running. When tasks in one container are idle, other containers can use the left-over CPU time. [..]
On a multi-core system, the shares of CPU time are distributed over all CPU cores. Even if a container is limited to less than 100% of CPU time, it can use 100% of each individual CPU core.
The important difference between the ideas of a “box” and a “hint” is that the “hint” doesn’t prevent the application from consuming the whole node’s CPU resources, when it requested only half of it — containers are not the VMs, afterall. Using the same example of a cluster’s node with 2 vCPU, a multi-threaded application inside a container still sees two CPU cores. If there are no other tenants on the node, there is no one who the application has to compete for its share of CPU resources.
As mentioned earlier, this applies to Docker container runtime. Don’t forget to consult with the documentation provided by the particular Kubernetes distribution when you make the decisions, that includes the fine-tunning of cluster resources. Docker is still the runtime AWS EKS uses for Kubernetes up to version 1.21. But things might work differently for your cluster as Kubernetes providers switch to alternative container runtimes.
Many years ago I embraced “tabless” development workflow: I use buffers, when I’m in Vim; I also switch tabs off in both Goland and VS Code, as the first thing after I install the IDEs to a new laptop.
I’m trying the same with Firefox web-browser now:
Enable toolkit.legacyUserProfileCustomizations.stylesheets switch in Firefox’s config (via about:config page or inside user.js).
Place the styles below into %PROFILE%/chrome/userChrome.css (I pick the actual path to the profile directory from Firefox’s about:support page).
// Hide the tabs.
// Beware that hidding the tabs with "display: none" will ruine you browser's recent history,
// I've learned that in a hard way ;)
#TabsToolbar > .toolbar-items {
opacity: 0;
pointer-events: none;
}
// Pull the navigation bar up, on top of the empty space, that left after we'd hidden the tabs.
#nav-bar {
margin-top: calc((7px + var(--tab-min-height)) * -1);
}
// On macOS, when not in full screen, shift the urlbar's panel to the right,
// after close-minimise-expand buttons.
:root:not([inFullscreen]) #nav-bar-customization-target {
margin-left: 65px;
}
(Optional) In Firefox’s “General” preferences, switch off “Ctrl+Tab circles through tabs”. With that, pressing Ctrl+Tab exposes all currently open pages (similar to how on macOS or other OS one switches between the opened applications with ⌘+Tab).
Overall I’m fairly happy with how it ended up. Although, some things aren’t quite ideal yet (might add more as I use this setup):
I wish there was a shortkey to enter a “modal mode”, where I could filter the list of open pages, to search for a particular page, and to switch to this page. Something similar to :ls command in Vim, or ⌘+E in Goland. I can use Firefox’s “Search amongst Tabs” for that (press ⌘+L to focus into the URL bar, and querying with “%[space]”) but that requires some getting used to.
Firefox has “Show all tabs” button (Ctrl+Shift+Tab) but the way it works, at least in Firefox 91, is very confusing and random. It seems to me, its behaviour is tightly coupled to the browser’s Tab UI.
At least for now, my homelab cluster (4x Raspberry Pi, k3s, etc) is available only for the devices on my local network, inside a custom DNS zone k8s.pi.home. I don’t think there are practical reasons to run anything with HTTPS in that setup, but there are cases, like browser extensions, where it’s required.
Turned out, in 2021, it’s fairly straight forward to set up a Certificate Authority (CA), that will issue TLS certificates to “secure” the ingress resources. At least, it’s way simpler comparing to how I remember it was back in the days. All thanks to cert-manager and some YAML.
First thing is to install cert-manager to the cluster. k3s comes with helm-controller, that gives us a way to manage helm charts with Custom Resource Definitions (CRD). The following manifest defines a new namespace, and a resource of a kind HelmChart, to install cert-manager inside this namespace:
And now, I can use selfsigned-issuer to issue TLS certificates for the ingress resources (Traefik ingress in the k3s’s case). E.g. to play around, I run an open-source version of LanguageTool server. The ingress manifests for the server looks like as following:
Of course, any certificate signed by my CA won’t be automatically trusted by anyone, including my own system. If I try to access https://languagetool.k8s.pi.home, any HTTP client will raise a “failed to verify the legitimacy of the server” issue. I don’t know if there is a better way to solve that, but I can hack that around by installing the cluster’s root CA certificate into the system’s keychain, and telling the system, that it should “trust” the certificate:
$ kubectl get secret/selfsigned-ca-root-secret -o json \
| jq -r '.data["ca.crt"]' \
| base64 -D > ~/tmp/selfsigned-root-ca.crt
$ open ~/tmp/selfsigned-root-ca.crt
Choose “Always trust” the certificate in the keychain’s certificate settings. The server looks legitimate now!
Initially, when I assembled a homelab cluster of Raspberry Pis, everything was directly connected to my Wi-Fi router with the Ethernet cables. This worked fine but this “stack of boards” behind the sofa in the centre of our small flat bugged me a bit.
Last year I decided to reorganise the cluster, turning it into a wireless-to-wired island, which I could relocate anywhere within the flat, without doing any special cable management, while staying cheap and avoid stacking the appartment with even more gadgets. After going through a number of trials and errors, the final setup looks as the following:
Different colours contour the connections between two logical subnets — more on that later. Here what we have on the schema (top to bottom):
Last year I made this one, after involving myself into a very random discussion on Twitter, about the unnecessary complexity and none-sense of modern days’ development environments.
Every time I sketch an HTTP API in Go, I wrap the code of request handlers around these small but very convenient bits.
My handlers are methods or functions, that serve a request and, either write a (positive) response or return an error.
// HandlerFunc is an HTTP handler function, that handles an HTTP request.
// It writes the response to http.ResponseWriter or returns an error.
type HandlerFunc func(w http.ResponseWriter, r *http.Request) error
// Handler is an adaptor for HandlerFunc, that converts the handler into http.Handler.
// It makes sure all errors returned from h are handled in a consistent manner.
func Handler(h HandlerFunc) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
err := h(w, r)
if err != nil {
handleError(w, r, err)
}
})
}
An error returned from HandlerFunc can be an indicator of a failure in request processing, statusError, or a general “something didn’t work”-error. The later can contain the internal details, that the API must never expose to the user.
// StatusError wraps an error err and contains the suggestion regarding to
// how the error should be communicated to the user.
//
// code must be a valid HTTP status code; text is the message to reply to user.
func StatusError(code int, text string, err error) error {
return &statusError{
Code: code,
Text: text,
Err: err,
}
}
type statusError struct {
Code int
Text string
Err error
}
func (s statusError) Error() string {
return s.Text
}
func (s statusError) Unwrap() error {
return s.Err
}
handleError is a helper function, which makes sure all errors returned from HandlerFunc are handled and replied to the user consistently. The internal details — the cause of the error — aren’t exposed to the user, but the helper can provide a unified logging and metrics, which would be convenient when debugging the error later:
var ErrNotFound = StatusError(http.StatusNotFound, "Nothing found", nil)
func handleError(w http.ResponseWriter, r *http.Request, err error) {
var statusErr *statusError
if !errors.As(err, &statusErr) {
statusErr = &statusError{Code: http.StatusInternalServerError, Text: "Internal server error", Err: err}
}
rid := RequestIDFromContext(r.Context())
resp := errResponse{
RequestID: rid,
Error: statusErr.Text,
}
replyJSON(w, resp, statusErr.Code)
// underlying error can be nil, as a special case, when the error is a client-side problem
if err := errors.Unwrap(statusErr); err != nil {
log.Errorw("request failed", "request-id", rid, "uri", r.RequestURI, "error", err)
}
}
replyJSON is a helper function, which writes a JSON string to http.ResponseWriter, setting the proper HTTP headers.
Below is an extract from a hypothetical API, with one single route /api/login, that takes an email, and replies with JSON, that contains this account’s ID.
func setupRoutes(mux *http.ServeMux) {
authHandler := NewAuthHandler(···)
mux.Handle("/api/login", Handler(authHandler.HandleLogin))
}
type AuthHandler {
// internal dependencies
}
func (h *AuthHandler) HandleLogin(w http.ResponseWriter, r *http.Request) error {
ctx := r.Context()
req, err := DecodeLoginRequest(r)
if err != nil {
return fmt.Errorf("decode login request: %w", err)
}
if req.Email == "" {
return StatusError(http.StatusBadRequest, "email is required", nil)
}
accID, err := h.datastore.GetAccountByEmail(ctx, req.Email)
if errors.Is(err, ErrNotExists) {
return StatusError(http.StatusForbidden, "account does not exist", err)
}
if err != nil {
return fmt.Errorf("get account for %q: %w", req.Email, err)
}
resp := struct {
ID string `json:"id"`
}{
ID: accID,
}
// ReplyJSON is a wrapper around internal replyJSON, that always responses with http.StatusOK
ReplyJSON(w, resp)
return nil
}
Do you have your own little things, that help you to lay out the boilerplate? Discuss this note on Twitter or Reddit.
With JSON data type it’s easy to treat Postgres as a document database, which doesn’t need strong schema. One can define a table, that has a field of a type jsonb and insert any valid JSON string (a “document”).
I’ve learned lately, that Postgres’s jsonb prohibits insertion of a valid JSON string if the string contains NULL (U+0000) character. Postgres’s own docs on JSON Types says:
RFC 7159 permits JSON strings to contain Unicode escape sequences denoted by \uXXXX. In the input function for the json type, Unicode escapes are allowed regardless of the database encoding, and are checked only for syntactic correctness. However, the input function for jsonb is stricter: it disallows Unicode escapes for non-ASCII characters (those above U+007F) unless the database encoding is UTF8. The jsonb type also rejects \u0000 (because that cannot be represented in PostgreSQL’s text type), and it insists that any use of Unicode surrogate pairs to designate characters outside the Unicode Basic Multilingual Plane be correct.
In my case, a Go backend inserts tracing logs to Postgres. A trace consists of multiple “spans”, some of which can contain the reply from an external API. As we found out, sometimes, in the event of a failure, the API replies with an empty GIF <facepalm/>. Our backend converts the response to a string, marshals it to a JSON and later tries to insert the JSON into a Postgres table.
Above, json.Marshal produces a perfectly valid JSON. But if I try to insert it into a Postgres table as jsonb, the insert fails with “unsupported Unicode escape sequence”:
= CREATE TABLE logs (data jsonb);
= INSERT INTO logs VALUES ('{"resp":"GIF89a\u0001\u0000\u0001\u0000\ufffd\···\u0001D\u0000;"}');
ERROR: unsupported Unicode escape sequence
LINE 1: insert into logs values ('{"resp":"GIF89a\u0001\u0000\u0001\...
^
DETAIL: \u0000 cannot be converted to text.
CONTEXT: JSON data, line 1: {"resp":...
Because in my code, there were only a couple of places where I didn’t control the actual data, that went into a span, the way I’ve chosen to handle that was by introducing a wrapper type, that implements json.Marshaller. The wrapper checks the value is a valid UTF-8 sequence and doesn’t contain NULL character before it marshals the value into a JSON string. If the value is not a valid UTF-8, the marshaller sees it as a binary data and base64-encodes it.
// RawText handles invalid UTF-8 and NULL-bytes, encoding them as base64-string.
// Because we have to make sure the resulting JSON will be compatible with Postgres's jsonb,
// we must use RawText when we don't control the data, e.g. when log the error from an external API.
// Refer to https://www.postgresql.org/docs/10/datatype-json.html
type RawText []byte
func (v RawText) MarshalEasyJSON(w *Writer) {
if utf8.Valid(v) && !bytes.ContainsRune(v, '\u0000') {
// "valid" text is marshalled as string
w.String(string(v))
} else {
// "invalid" text is marshalled as binary data
w.Raw(json.Marshal([]byte(v)))
}
}
2020 was (technically, still is) an extraordinary year. For many people, this was the worst, the weirdest, the most depressing year; the year we wish was a bad dream. With all the respect for the terrible things, lots of people went through, for me, it was a year to remember.
In 2020 I learned how to work from home. It’s still not super fun, but it’s possible. As I look at that now, there are obvious benefits: I can start and end my day much more freely; I currently don’t feel much pressure when willing to go for a walk or a run at 11:00, or when I finish my workday at 16:00 (I tend to start around 8:30 while at home). But that being said, I still don’t consider staying full-remote and I want to go back to the office, co-working, or a random coffee bar.
In 2020 I bought a bicycle. After three years in Berlin, I’m with “all of those people” now :) Can’t say I’m super excited about the purchase. Mostly because I live in the walking distance from the office. With the bicycle, my “commute” is only five or seven minutes shorter. I have to wait at the traffic lights a lot; plus I have to spend extra minutes to lock/unlock the bicycle.
In 2020 I walked, on average, one kilometre per day more, than in the previous year (5.5 km vs 4.2 km). A bit of surprise but, overall this year, I walked less than I did in 2019. I bet that’s because this year we, obviously, didn’t travel much, so we didn’t have a chance to spend weeks walking around a new city, neither in Spring, nor on Christmas and New Year.
In 2020 we adopted a cat. The CAT, who’s now called Gagarin! He came to us from the middle of “nowhere-Russia” as a frightened little kitten and spent the first month under our bed, avoiding walking under the light. And now, after four months, he comes to have a nap on our pillow (and on my head) at 7:00, as his way to signal that it’s time to feed him.
In 2020 I assembled my four nodes Raspberry Pi Kubernetes cluster, but I haven’t made much use of it yet. As promised to myself in December 2019, during the year I learned and touched some new Kubernetes details, and I have tons of things I still want to get my hands into to understand on a more in-depth level of details.
In 2020 I started to learn Rust. I tried learning it in 2018 and in 2019. But this year, I actually spent a month or two writing code, not only reading/watching/listening to/following the content. It feels refreshing. Programming for embedded devices and game-dev are the most attractive fields that I want to explore further with Rust.
In 2020 I had two job interviews. None of them were successful for me, although I tend to believe that I’ve passed both. I have plans to draft a note about the essential questions one must ask the HR during the first call, somewhere later.
In 2020 I did and learned, and read, and listened to, and watched, and thought about a bunch of things.
This is another thought experiment, this time on the importance of my feedback in the code-reviews.
Providing you’re working on a project maintained by a set team of N people. What would happen with the codebase if, for six months, in code-reviews, you started to accept changes for which, you generally leave feedback?
“Variables, struct or function names are named poorly” — accept. “We already have a package, that solves a similar problem” — accept. “A change brings inconsistency to the codebase” — accept. “The pattern doesn’t belong here” — accept.
Go is a famously opinionated programming language. But does being opinionated help and scale outside of the language and its standard library?
Do you have the answer? Discuss this note on r/golang Reddit or share your thoughts on Twitter.
With Apple’s new M1 Macs showing (reportedly) huge performance improvement, compared to “old”, Intel-based Macs, I wonder what would hold Qualcomm (Snapdragon CPU) and others from doing “the same” and moving into laptop/desktop territory?
Microsoft already has Surface Pro X — an ARM-based Windows computer. They also have a version of Win10 for ARM, that one can even run on Raspberry Pi (still beta quality, I believe). Could 2021 become the year of ARM on desktop?
Even more interesting, Amazon’s Graviton is showing (again, reportedly) an excellent performance, while staying at reasonably low cost. What would stop Google/Microsoft from moving into ARM-based CPU territory for their clouds?
I’m not a big fun of the aesthetics of VS Code, at least not on macOS. In particular, I don’t like the way how its code editor renders fonts. But today I learned!
To make the fonts look better — I’m on macOS — set an alternative font-variant, e.g. “Comic Code Ligatures Medium” instead of “Regular”, in the editor’s settings. To specify font-variant, remove spaces in between the name of the font and pass the variant after the hyphen. That is, instead of 'Comic Code Ligatures', set the following:
Full disclosure, my IDE of choice is GoLand, and it has been so since it was only a plugin for IntelliJ IDEA. Even though I can comfortably use VS Code or Vim when I need to make a small change or look something up in the code, I need the IDE to effectively work on a large codebase.
The backstory for this note is that I’m working on a small TypeScript/React application in my spare time this month. For something that small, VS Code works great fine. In fact, I’m writing this particular note in VS Code too ;)
It’s one year since I posted the very first note here. Unbelivable! Despite my own concerns I did published random posts over the course of the previous twelve months.
For the next round I came up with some personal goals:
Keep posting.
Work on your grammar.
Add “Archive” and “Tags” sections.
Find a better approach for managing drafts.
Bring back the dark theme but figure out what to do with the illustrations.
The main component of profefe, a system for continuous profiling, is profefe-collector — a service that receives profiling data, taken from an application and persists the data in collector’s storage (design document describes it in more details). Receiving data from an external source (for example, profefe-agent), indicates that profefe, as an observability system, prefers “push” model. Why not “pulling” the profiles directly from the application?
Both push and pull models have their benefits and drawbacks.
A collector that pulls profiling data from running applications could simplify integration into existing infrastructure because there would be no need in making changes in the applications that already exposed pprof HTTP endpoint. Making sure that every application integrated and configured profefe-agent would be a challenging job in a large organisation.
On the other hand, pull model requires pprof servers to be exposed and available for the collector, so it could fetch (pull) profiling data. That can also be challenging in the deployments, where applications are collocated on the bare-metal machines. Every application (application’s instance) would have to communicate a unique TCP port for its pprof server.
To work as a pull-system, the collector must be able to discover the pprof servers, thus it requires a mechanism for service discovery (SD), to be usable at scale. Unfortunately, there isn’t a universal SD protocol or a provider, an observability system could be built upon.
Prometheus, the best example of an open-source system, which uses pull model for data collection, have to support several different SD systems in their code base. At some point they ended up introducing their own general protocol, that expects a “middle-man-service”, which translates the data from a SD system into a list of Prometheus targets (Update, this comment from u/bbrazil does a better job explaining the state of SD in Prometheus). There is no clear way for an open-source system to be both flexible and don’t end up being a pile of “plugins”, that no one is willing to maintain or break.
From the start of profefe project, several years ago, I had the idea that translating push into pull would be easier for an end-user. That is if a small deployment already exposes a pprof server, writing an external job that pulls the profiles from the applications, annotates them with meta-data, and pushes the data into the collector, can be as easy as spawning a cronjob in a sidecar. kube-profefe solves that nicely for deployments running in Kubernetes. At some point, I hoped to come up with something similar for Nomad+Consul if the experiments ended successfully.
Translating pull into push is a similarly possible but because profefe didn’t have to support any SD mechanisms from the start, that simplified the overall code base and allowed us to focus on the collector and the API for profiles quering.
profefe-collector does uses push model. But one can deploy profefe so it reflected the use cases your organisation has.
Do you use continuous profiling? Let me know about your experience. Share your thoughts on Twitter or discuss on r/golang.
Sometimes I write small services in Go from scratch. And every time main.go ends up looking almost the same:
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sigs := make(chan os.Signal, 2)
signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
go func() {
<-sigs
signal.Stop(sigs)
cancel()
}()
if err := run(ctx, os.Args[1:]); err != nil {
log.Fatalln(err)
}
}
type Config struct {
HTTPAddr string
HTTPShutdownTimeout time.Duration
}
func run(ctx context.Context, args []string) error {
flags := flag.NewFlagSet("", flag.ExitOnError)
var conf Config
flags.StringVar(&conf.HTTPAddr, "http-addr", "127.0.0.1:10080", "address to listen on")
flags.DurationVar(&conf.HTTPShutdownTimeout, "http-shutdown-timeout", 5*time.Second, "server shutdown timeout")
if err := flags.Parse(args); err != nil {
return err
}
// TODO: define the handler, the routing, and wire the dependencies with the main context
mux := http.NewServeMux()
server := &http.Server{
Addr: conf.HTTPAddr,
Handler: mux,
}
errs := make(chan error, 1)
go func() {
log.Printf("starting: addr %s", server.Addr)
errs <- server.ListenAndServe()
}()
select {
case <-ctx.Done():
log.Println("exiting...")
case err := <-errs:
return err
}
// create new context because top-most one is already canceled
ctx, cancel := context.WithTimeout(context.Background(), conf.HTTPShutdownTimeout)
defer cancel()
return server.Shutdown(ctx)
}
Of course, not every service requires an HTTP server, but the general idea stands.
When Go will introduce signal.NotifyContext(), the signals handling in main() function will be much smaller (we’re at go1.15rc1 as I’m writing that and the change hasn’t landed into the release yet).
I love how transparent is the flow here and how everything is scoped inside run() function. This structure forces you to eliminate global state, making unit or integration testing almost trivial — at least, in theory ;)
It might feel like too much of boilerplate code for a “small” service. In practice, though, I don’t recall any time this caused any real troubles to me. The beauty of Go is in explicitness.
I’ve been playing with an e-paper ESP866 Driver Board, and a 2.7" E-Ink display from Waveshare. Arduino C++ looks manageable. One strange thing, though. In both board’s documentation and GxEPD2 library’s examples, they say the display is connected to pins as BUSY → GPIO16, RST → GPIO5, DC → GPIO4, CS → GPIO15. This mapping seems wrong.
After digging through the code examples from Waveshare’s Wiki, the correct mapping is the following:
Sticky (or “fixed”) headers are everywhere. It feels that, nowadays, every web designer’s first attempt to site’s navigation starts with a sticky header. I hate this.
Who should be in charge of logging context: a component’s owner or the component itself?
type Logger interface {
With(...kvpairs) Logger
}
type Storage struct {
logger Logger
}
// OPTION 1: component's owner defines the context of component's logger
func main() {
_ = NewStorage(logger.With("component", "storage"))
}
// OPTION 2: component itself is in charge of its logging context
func NewStorage(logger Logger) (st *Storage) {
return &Storage{
logger: logger.With("component", "storage"),
}
}
Fun fact: a couple months back, we ruined the team’s Friday, by debating about a similar topic in the context of (Graphite) metrics namespaces. It has become even more intricate since then :/
Update (2020-04-15)
Many people on Twitter suggest that Option 1 is an obvious choice because only application knows how to name the components. I totally agree with that.
As I wrote later, the real dilemma is not about “application and component” but about “owner of the component”. Function main, in the example above, was a silly example, that tried (and failed) to illustrate the question in a code.
Let’s try another (silly) example:
// there are buch of different handlers (maybe ten) in this application
type Handler1 struct { logger Logger }
func (h *Handler1) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// OPTION 1
req := NewRequst(h.logger.With("component", "request"), r)
}
type Handler2 struct { logger Logger }
func (h *Handler2) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// OPTION 1, still
req := NewRequst(h.logger.With("component", "request"), r)
}
type Request struct {
logger Logger
}
// OPTION 2
func NewRequst(logger Logger, *r http.Request) *Request {
return &Request{
logger: logger.With("component", "request"),
}
}
We want to have a consistent nomenclature across the application’s logs.
Is the choice still obvious? ;)
Do you have an opinion? Share it with me on Twitter.
Participating in self-isolation is more fun when you have toys to play. As a fun weekend project, I wanted to look at how one accesses macOS Location Services and get the geographic location of the device from Go.
To obtain the geographic location of a device on macOS, we use Apple’s Core Location framework. The framework is part of the OS, but it requires writting Objective-C (or Swift). Thanks to Go’s cgo and because Objective-C is from the family of C languages, we can write a bridge between Objective-C and Go.
This is a lengthy note. If you don’t quite feel reading and only need the working example, go directly to the Travis CI build file.
The more I delve into the world of Raspberry Pi, the more I notice that “regular and boring” things on ARM are harder than I expected.
People build and distribute software exclusively for amd64. You read another “Kubernetes something” tutorial, that went viral on Twitter, and is fancy to try it out. Still, all helm charts, or whatever the author prefered, use Docker images built exclusively for amd64.
Docker toolchain has added the support for building multi-platform images in 19.x. However, it’s available only under the “experimental” mode. The topic of building multi-platform Docker images yet feels underrepresented.
But first, what are multi-platform Docker images?
When a client, e.g. Docker client, tries to pull an image, it must negotiate the details about what exactly to pull with the registry. The registry provides a manifest that describes the digest of the requested image, the volumes the image consists of, the platform this image can run on, etc. Optionally, the registry can provide a manifests list, which, as the name suggests, is a list of several manifests bundled into one. With the manifests list in hands, the client can figure out the particular digest of the image it needs to pull.
So multi-platform Docker images are just several images, whose manifests are bundled into the manifests list.
Imagine we want to pull the image golang:1.13.6-alpine3.10. Docker client will get the manifests list from Dockerhub. This list includes digests of several images, each built for the particular platform. If we’re on Raspberry Pi, running the current Raspbian Linux, which is arm/v7, the client will pick the corresponding image’s digest. Alternatively, we could choose to pull the image arm32v7/golang:1.13.6-alpine3.10 instead, and we ended up with the same image with the digest d72fa60fb5b9. Of course, to use a single universal image name, i.e. golang, on every platform is way more convenient.
Does it mean I need to build different Docker images, for each platform I want to support?
Well, yes. This is how, official images are built.
For every platform, the image is built and pushed to the registry under the name <platform>/<image>:<tag>, e.g. amd64/golang:1-alpine. And next, a manifests list, that combines all those platform-specific images, is built and pushed with the simple name <image>:<tag>.
Docker’s BuildKit provides a toolkit that, among other nice things, allows building multi-platform images on a single host. BuildKit is used inside Docker’ buildx project, that is part of the recent Docker version.
One can use buildx, but, for this post, I wanted to try out, what would it look like to use BuildKit directly. For profefe, the system for continuous profiling of Go services, I set up Travis CI, that builds a multi-platform Docker image and pushes them to Dockerhub.
profefe is written in Go. That simplifies things, because, thanks to Go compiler, I don’t have to think about how to compile code for different platforms. The same Dockerfile will work fine on every platform.
It’s a lot happening here, but I’ll describe the most critical parts.
Let’s start with dist: bionic.
We run the builds under Ubuntu 18.04 (Bionic Beaver). To be able to build multi-platform images on a single amd64 host, BuildKit uses QEMU to emulate other platforms. That requires Linux kernel 4.8, so even Ubuntu 16.04 (Xenial Xerus) should work.
In short, we tell the component of the kernel (binfmt_misc) to use QEMU when the system executes a binaries built for a different platform. The following call in the “install” step is what’s doing that:
- docker container run --rm --privileged multiarch/qemu-user-static --reset -p yes
Under the hood, the container runs a shell script from QEMU project, that registers the emulator as an executor of binaries from the external platforms.
If you think, that running a docker container to do the manipulations with the host’s OS looks weird, well… I can’t agree more. Probably, a better approach would be to install qemu-user-static, which would do the proper setup. Unfortunately, the current package’s version for Ubuntu Bionic doesn’t do the registration as we need it. I.e. its post-install doesn’t add the "F" flag (“fix binaries”), which is crucial for our goal. Let’s just agree,that docker-run will do ok for the demonstrational purpose.
This is another “docker-run’ism”. We start BuildKit’s buildkitd daemon inside the container, attaching it to the Docker daemon that runs on the host (“privileged” mode). Next, we copy buildctl binary from the container to the host system and set BUILDKIT_HOST environment variable, so buildctl knew where its daemon runs.
Alternatively, we could install BuildKit from GitHub and run the daemon directly on the build host. YOLO.
To be able to push the images to the registry, we need to log in providing Docker credentials to host’s Docker daemon. The credentials are set as Travis CI’s encrypted environment variables ([refer to Travis CI docs])](https://docs.travis-ci.com/user/environment-variables/)).
This is the black box where everything happens. Magically!
We run buildctl stating that it must use the specified Dockerfile; it must build the images for defined platforms (I specified linux/amd64,linux/arm64,linux/arm/v7), create a manifests list tagged as the desired image (profefe/profefe:<version>), and push all the images to the registry.
buildctl debug workers ls shows what platforms does BuildKit on this host support. I listed only those I’m currently intrested with.
And that’s all. This setup automatically builds and pushes multi-platform Docker images for profefe (https://hub.docker.com/p/profefe/profefe) on a commit to project’s “master” branch on GitHub.
As I hope you’ve seen, support for multi-platform is getting easier and things that were hard a year ago are only mildly annoying now :)
If you have any comments or suggestions, reach out to me on Twitter or discuss this note on r/docker Reddit.
My home cluster is four Raspberry Pis 4 (2GB); all connected to my internet router through ethernet and
powered with 60W 6 USB-ports charger. All Pis build a small Kubernetes cluster that runs with k3s.
All by one Pis run on Raspbian Buster Lite and this setup’s been working pretty well until I’ve found out,
Aerospike, a database I required to run for a testing lab, only works on a 64-bit OS.
Luckily, Ubuntu Server has an arm64 version built for Raspberry Pi. Thus, my working plan is to switch one Pi
to Ubuntu, compile and run a single-instance Aerospike server (and any other components, that require a 64-bit OS) on this Pi, and provide a Kubernetes service in front of the DB, so other components in the cluster could access it as if it was
fully managed by Kubernetes.
After listening to the “Fireside edition” of Go Time, I questioned myself, how would I answered the questions the hosts discussed.
Because no one has asked, you are very welcome:
1. If you had two weeks to spend on a personal Go project, what would you work on?
I really want to invest more time for profefe. Specifically,
on implementing an analyser of stored profiles: the thing that would help to make sense of the data,
showing how the performance of an instance, a node, or a cluster had changed over the period of time;
how different parts of the codebase had influenced the performance of the application.
Recently Amazon has announced CodeGuru profiler (currently Java-only). From the description it feels
exactly what I pictured in my head when I started the project.
Another topic that I would like to invest more time on is the understanding of the ecosystem around/inside Kubernetes.
During the past two years, I slowed down the consumption of the DevOps/SRE topics, mostly due to the specific
state of the infrastructure in our company. But, “k8s is the new linux”, regardless of what one’s opinion on that.
Even profefe recently has ended up having a kube-profefe (a bridge between profefe and Kubernetes), contributed and maintained by other people.
2. What annoys you about Go of 2019?
The same small things that annoyed me in Go 1.4: var, new, make and “naked return”. Sure, I understand that they all ended up in the language for a reason. But I simply don’t like the “magic” of make, which works only with particular types;
the two ways of defining a variable (var or :=), or a pointer to an instance of a type (new or &T{}).
One new thing, though. Go modules’ semver imports. But I can’t say anything new about that. Probably, I just need to embrace them. Go 1.14 looks like a version where I might completely switch to modules, thanks to better handling of vendored dependencies.
3. What’s your ideal working environment?
That always surprises me. Lots of people keep saying that working from home is their ideal environment or even a factor
that influence their job offers choice. I don’t like to work at home. The only time when I feel productive when stay home is in the nights. A café or a co-working works sometimes. But I like working in a big office space. I don’t know why.
Of course, open-plan offices can be very different. Yandex’s “Red Rose” is still the best space I ever worked in. I heard they do excursions around the Moscow’s office now.
3.1. Something on pair-programming?
Since I wrote about Yandex.
Some people think pair-programming is a sort of super-power. It’s, and it’s not.
You can’t just put yourself in an environment, where someone is watching how you write the code while trying to hold a conversation about the code architecture. Pair-programming is a skill to master. But it pays off.
The pair-(trio actually)-programming sessions we did in Yandex, when we worked on bem-core, was the most significant skill boost I had during the five-plus years there.
Of course, the positive experience comes from your peers. In my case, they were people with huge baggage of knowledge and practice of working, talking, debating with each other. Like, out of nowhere, you get the understanding of what
types of questions you must ask; when it is important to spend more time on thinking and when you can make a small hack.
4. Your advice to you junior-developer self?
Don’t overthink and afraid of starting anything. Trying something by making a raw, dirty, barely-working prototype will give you way more knowledge than thinking about how to do that.
The part I like most there is different cases where compiler doesn’t allocate memory for string to []byte conversions:
For a map m of type map[string]T and []byte b, m[string(b)] doesn’t allocate (the temporary string copy of the byte slice isn’t made)
Turned out, since this wiki page was written, more similar optimisations were added to the compiler.
As it’s in Go 1.12+ the following cases are also listed in runtime/string.go:
Strings comcatenation
For the case "<" + string(b) + ">", where b is []byte no extra copying of b is needed.
Comparison
if string(b) == "foo" { ··· }
In the code above, b []byte also won’t be copied.
There are still cases where compiler can’t optimise the code for us. In some of those cases
it’s fine to do string to bytes conversion using a so called “unsafe trick” (accessing string’s underling
data directly, with out copying the data from string to bytes and vice versa). One can find several
ways of performing the trick, but none of them seems “the one that must be used”.
After years of episodic discussions, a collegue of mine assembled the list of different conserns and about
the proper way of doing it (see “unsafe conversion between string <-> []byte” topic on golang-nuts forum).
Thanks to replies from Go team, our most valid way of doing it is following:
// Refer to github.com/fmstephe/unsafeutil
type stringHeader struct {
data unsafe.Pointer
stringLen int
}
type sliceHeader struct {
data unsafe.Pointer
sliceLen int
sliceCap int
}
func StringToBytes(s string) (b []byte) {
stringHeader := (*stringHeader)(unsafe.Pointer(&s))
sliceHeader := (*sliceHeader)(unsafe.Pointer(&b))
sliceHeader.data = stringHeader.data
sliceHeader.sliceLen = len(s)
sliceHeader.sliceCap = len(s)
return b
}
func BytesToString(b []byte) (s string) {
sliceHeader := (*sliceHeader)(unsafe.Pointer(&b))
stringHeader := (*stringHeader)(unsafe.Pointer(&s))
stringHeader.data = sliceHeader.data
stringHeader.stringLen = len(b)
return s
}
The other day I received my beta access to GitHub Actions. To try them out I picked an existing pet project and created a workflow using a Go project template provided by GitHub. As it’s in September 2019, their template defines the sequence of steps: setup Go, checkout code, get dependencies, build. This is not exactly how I used to do it.
My project is a classic Go service ;) meaning: it uses vendoring and doesn’t use Go modules. So no need for “get dependencies” step. And it requires to be inside the GOPATH. With that, the provided workflow needed some adjustment.
After some trials and errors, I’ve managed to make checkout step to clone the repo into the correct destination inside the GOPATH. Here is the final workflow:
name: Run Go test
on: [pull_request]
jobs:
test:
strategy:
matrix:
go-version: [1.12.9]
runs-on: ubuntu-latest
steps:
- uses: actions/setup-go@v1
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v1
with:
path: ./src/github.com/${{ github.repository }}
fetch-depth: 5
- run: make test
env:
GOPATH: ${{ runner.workspace }}
Note, how actions/checkout@v1 above uses custom path input parameter. I set the path to ./src/github.com/${{ github.repository }}, so the project is checked out to src directory in the runners’s workspace, which I later pass as the value of GOPATH to the “make test” step. The leading dot in ./src seems very important — I’ve spent the majority of the time trying to figure out that part — refer to this issue.
One probably knows that net/http.Headers is no more than map[string][]string with extra specific methods. A usual way to initialize and populate such data-structure from an external representation is something like that:
type Header map[string][]string
func (h Header) Add(key, val string) {
if val == "" {
return
}
h[key] = append(h[key], val)
}
func main() {
h := make(Header)
h.Add("Host", "example.com")
h.Add("Via", "a.example.com")
h.Add("Via", "b.example.com")
}
From the code above, one can notice that we allocated a new slice of strings for every unique key that we added to headers. For things like HTTP headers, that’re automatically parsed for every incoming request, this bunch of tiny allocations is something we’d like to avoid.
I was curious to know if Go’s standard library cares about that.
We know that for a majority of cases, HTTP headers are an immutable key-value pair, where most of the keys have a single value. Instead of allocating a separate slice for a unique key, Go pre-allocates a continues slice for values and refers to a sub-slice of this slice for all keys.
Knowing that, we can refactor the initial Header.Add as the following:
type Header map[string][]string
func (h Header) add(vv []string, key, val string) []string {
if val == "" { ··· }
// fast path for KV pair of a single value
if h[key] == nil {
vv = append(vv, value)
h[key] = vv[:1:1]
return vv[1:]
}
// slow path, when KV pair has two or more values
h[key] = append(h[key], val)
return vv
}
func main() {
h := make(Header)
// net/textprotocol pre-counts total number of request's headers
// to allocate the slice of known capacity
vv := make([]string, 0)
vv = h.add(vv, "Host", "example.com")
vv = h.add(vv, "Via", "a.example.com")
}
