A memory-efficient data structure for geolocation history (rev2)

Unknown — Sat, 05 Nov 2022 00:00:00 +0000

In this blog post I'll cover how I built a memory-efficient data structure for storing location history data.</p>

Some Rust details will also be explained.</p>

This article is a rewrite of my previous one</a>, after taking into account feedback from Reddit</a> and further researching the topic.</p>

Motivation</h2>
My DSLR camera has no GPS built-in. Evidently, photos from it don't include geolocation-based EXIF tags.</p>
If only my location history were stored somewhere, so that I could update photos with the correct data...</p>
Unsurprisingly, Google knows where I am, where I've been and where I'm going to be</del>. They even allow me to export this data.</p>
The problem is that the exported JSON weighs ~1GB. My computer has 16GB of RAM so it can all fit in memory, but we certainly can store it more efficiently.</p>
JSON parsing</h2>
For reference, here's what the JSON looks like:</p>
{ "locations": [{ "latitudeE7": 435631892, "longitudeE7": 26848689, "accuracy": 56, "activity": [{ "activity": [{ "type": "STILL", "confidence": 100 }], "timestamp": "2014-01-09T00:48:24.424Z" }], "source": "WIFI", "deviceTag": 1348159918, "timestamp": "2014-01-09T00:48:24.751Z" }, { "latitudeE7": 435631881, // ... </code></pre> Deserializing a JSON file into a struct results in something that is certainly lighter than the JSON string.</p> The reason for this is that a string with the value "2022-10-22"</code> needs more bytes than, say, its equivalent Date</code> object.</p> The latter can simply throw away the hyphens, for example. Additionally, the number 10 can be represented with 4 bits instead of using two chars, which occupy at least 8 bits each.</p> So moving away from a textual representation is the first step in our journey.</p> The simplest way to deserialize this data is to put its contents into a String</code> and then pass it to your Json deserializer or choice. </p> The problem of this approach is that the String</code> will occupy ~1GB of RAM, even if for just a couple of seconds. </p> Ideally, we should have a stream of data and keep deserializing small chunks at a time. </p> I'm pretty sure there are some libraries out there capable of that but I decided to simply iterate over each line of the file, extract relevant data and ignore the lines I wasn't interested in.</p>Do we really need to store everything?</h2> The most straightforward structure for our purposes is a sequence of DataPoint</code>s, where DataPoint</code> has datetime</code>, latitude</code> and longitude</code>:</p> [ (DateTime(2022,10,22,10,45,32), 27.5226335, 43.552225), (DateTime(2022,10,22,10,46,21), 27.5226382, 43.552237), // ... ] </code></pre> But isn't there some redundancy? The data is clearly not random.</p> There are some assumptions we can make to help us design a better data structure:</p> Data points are sorted by time</li> Neighbor data points are very similar unless I'm on a plane, my speed is at most 100 km/h, so in a minute I'll travel less 2 km</li> on foot I'll travel just a few meters </li> </ul> </li> Most of the days I stay inside a circle of 3km of radius</li> One data point per minute is more than enough</li> Errors in position up to 15m is something I'm OK with</li> </ol> Estimating size of straightforward solution</h2> Throughout this post, let's consider 2.5 years worth of data, including 345k data points. </p> Each data point consumes 64 bits for the timestamp and we can store each coordinate with a f64</code>. </p> In total we need ~11.8 MB (if you did the math and think this off by ~50% please see this section</a>).</p> How many decimal places do we need to represent a coordinate?</h2> First let's try to understand how much each decimal place contributes to precision.</p> A quick search on Google gives us the following</a>:</p> - The first decimal place is worth up to 11.1 km - it can distinguish the position of one large city from a neighboring large city - The second decimal place is worth up to 1.1 km - it can separate one village from the next - The third decimal place is worth up to 110 m - it can identify a large agricultural field or institutional campus - The fourth decimal place is worth up to 11 m - it can identify a parcel of land </code></pre> If we want to have errors less than 15 m, we need to be able to correctly represent the latitude and longitude up to the 4th decimal place.</p> If both coordinates are off by 0.0001° then the error will be close to sqrt(11^2+11^2) ~ 15 m</code>.</p> How many bits do we need to represent time?</h2> For our purposes, 24 bits would be enough</a>.</p> However, we don't need to store timestamps. </p> Instead of having at most one data point per minute, we can have exactly one data point per minute.</p> It's cheaper to just duplicate data for missing timestamps. The duplication of data is surely unfortunate, but having to spend 24 bits for every minute of data is worse.</p> The plan</h2> Suppose our data looks like this:</p> Timestamp</th> Latitude</th> Longitude</th></tr></thead> 2022-10-15 16h13</td> 26.1234</td> 43.5678</td></tr> 2022-10-15 16h14</td> 26.1235</td> 43.5677</td></tr> 2022-10-15 16h15</td> 26.1236</td> 43.5678</td></tr> </tbody></table> We could delta-encode it so that all points (except the first one) are just the difference from the previous value:</p> Timestamp</th> Latitude</th> Longitude</th></tr></thead> +1 min</td> +0.0001</td> -0.0001</td></tr> +1 min</td> +0.0001</td> +0.0001</td></tr> </tbody></table> The data clearly becomes easier to reason about and is arguably "smaller".</p> But when using an f32</code> to represent a number, it doesn't matter if it's 26.1234</code> or 0.0001</code>: they will both occupy 32 bits. So we need a way to make small numbers occupy less memory.</p> Finally it would be nice to somehow merge the latitude and longitude columns, so that we can keep track of a single value instead of having to store 2 deltas for each minute.</p> To recap, the plan is to:</p> merge latitude and longitude into a single value</li> delta-encode the data</li> store small numbers efficiently</li> </ol> Representing 2D data with a single number</h2> The objective here is to come up with a function f(pos: (f32, f32)) -> u32</code> and an "inverse" function g(n: u32) -> (f32, f32)</code> such that error(pos, g(f(pos))) < ε</code>, for any valid pos</code>.</p> In our case, ε</code> = 15 m.</p> The most straightforward way to do it is like so:</p> In this approach, we travel a full row, move one step up/down, travel another full row and so on. Source: 3b1b</figcaption> </figure> The problem with this algorithm is that the distance between neighbour points isn't necessarily small.</p> Suppose we move one step to the right. The distance traveled will be one unit. Great!</p> Now suppose we move one step up. The distance traveled will be grid.cols</code>, which is certainly more than we'd like. This means that moving +0.0001</code> in latitude could result in a distance of thousands of units, yet we want deltas to be as small as possible.</p> A more elegant solution would be to use Hilbert curves</a>:</p> The Hilbert curve is a continuous fractal space-filling curve. Source: 3b1b</figcaption> </figure> </figure> Now, if we move one unit, the traveled distance will generally be small, regardless of the movement being horizontal or vertical.</p> For resolutions big enough, this will remain true for most points, except the ones at "boundaries" (e.g. going from a red cell to a green one).</p> This video from 3b1b</a> explains the concept of space-filling in detail. It's worth checking it out!</p> That said, I decided to go with a simpler strategy, which is close to the concept of binary space partitioning</a>.</p> Here's the idea:</p> is our point in the left hemisphere or right? if left, set first bit to 0; otherwise, 1</li> </ul> </li> is our point in the south hemisphere or north? if south, set second bit to 0; otherwise, 1</li> </ul> </li> is our point in the left half of the region in 1</code> or right? if left, set third bit to 0; otherwise, 1</li> </ul> </li> is our point in the south half of the region in 2</code> or north? if south, set fourth bit to 0; otherwise, 1</li> </ul> </li> ...</li> </ol> Here's an image depicting the process of representing the yellow circle:</p> Whenever we decide on a region in the horizontal space, the selected region becomes greener; likewise, pinker for vertical regions.</figcaption> </figure> One nice property of this algorithm is that, as we take more decisions (i.e. as resolution gets more fine-grained), we're simply adding more bits.</p> This means we can throw away less significant bits as we see fit.</p> Storing small numbers efficiently</h2> This paper</a> describes a lot of different methods.</p> I decided to use a modified version of the Simple8b algorithm because of, as you've probably guessed, its simplicity.</p> The idea is to divide a u64</code> into 2 parts: a selector</code> of 4 bits, and a data</code> region of 60 bits.</p> The selector determines how many bits each number inside data</code> will take.</p> For example, for selector</code>=15, we store 60 numbers of 1 bit each; for selector</code>=14 (= 1110 in binary), we store 30 numbers of 2 bits each:</p> 4 bits 60 bits === selector === | ====================== data ======================== 1110 01 10 00 10 11 ... 01 (= 14 in dec) | | └ third number | | └ second number | first number 30th number </code></pre> I made 2 modifications to this algorithm that allowed me to compress data even further.</p> Modification 1: representing (a lot of) zeros</h3> As mentioned above, we're adding one data point to our data structure for every minute. But what if there's simply no data for an entire month? </p> The good news is that the deltas will be a lot of zeros, if we decide to simply copy the latest data point over and over.</p> The bad news is that we'll have a lot of unnecessary zeros.</p> So I changed the algorithm so that when selector</code>=1, data</code> is the count of zeros we're trying to represent. So let's say that we have 1 billion consecutive zeros. Instead of needing 1 billion numbers to represent this data, we can simply create a Simple8b-compressed u64</code> with selector</code>=1 and data</code>=1 billion.</p> Modification 2: not tracking the count of elements</h3> Unfortunately, the Simple8b-compressed u64</code> doesn't contain the number of elements compressed.</p> For example, this test fails:</p> let input0 = [2, 76, 3, 5, 7, 2, 0, 0]; let input1 = [2, 76, 3, 5, 7, 2]; let inverse0 = simple8b::decompress(simple8b::compress(input0)); let inverse1 = simple8b::decompress(simple8b::compress(input1)); // passes assert_eq!(input0, inverse0); // fails: inverse1 = input0, with the extra zeros assert_eq!(input1, inverse1); </code></pre> This is also unfortunate because keeping track of this count (which requires at least 8 bits), defeats the purpose of compressing: we may save a few bits here and there but, with this cost of 8 bits per u64</code>, perhaps the final size will be bigger than the original.</p> In Simple8b, a zero means a lack of data, so there's no way to distinguish between trailing zeros or absence of data.</p> My change was really simple: whenever we try to store a number n</code>, we actually store n+1</code>. Conversely, when retrieving a number m</code>, we actually return m-1 = n</code>.</p> This way any zeros we find during decompression indicate that the actual data has already ended.</p> The downside of this approach is that selector</code>=15 (60 numbers of 1 bit each) becomes useless: normally this could only happen if we had a bunch of 0s and 1s. But now a 1 becomes a 2, so this can only be possible if we have a lot of 0s only. In this case selector</code>=1 will be used.</p> But we would probably not have this scenario of multiple 0s and 1s to begin with: unless I kept moving ~1 cm per minute for a long period of time. 😅</p> Representing deltas</h2> Alright, so we can now represent both coordinates with a single number and store a bunch of these numbers efficiently.</p> A position at time i</code> can then be represented as position_i = position_0 + sum(delta_i, i = 0 to i)</code>.</p> But delta_i</code> is a u64</code>, which is an unsigned integer. How can we represent negative numbers?</p> This is easy. We can simply bit shift to the left and use the least-significant bit to represent the sign:</p> fn to_unsigned_delta(signed_delta: i128) -> u64 { if signed_delta >= 0 { (signed_delta << 1) as u64 } else { ((signed_delta << 1) + 1) as u64 } } fn delta(unsigned_delta: u64) -> (u64, bool) { let is_positive = unsigned_delta % 2 == 0; let signed_delta = unsigned_delta >> 1; (signed_delta, is_positive) } </code></pre> Getting our hands dirty</h2> Finally, at its core, our data structure will do the following:</p> fn add(&mut self, time: DateTime, pos: (f32, f32)) -> Result<()> { self.add_missing_data(time, pos); let delta = calculate_delta(self.last_pos, pos); self.fail_if_error_is_too_big(delta, pos)?; // optional self.push_point(delta); self.last_pos = pos; Ok(()) } </code></pre> Just to be safe, before pushing the latest delta, I'm calculating if the haversine distance</a> is below ε</code> = 15 meters. This is more precise because the the error varies with the distance to the Equator. </p> Benchmarking</h2> After finishing everything, we can measure how many megabytes our data structure needs.</p> Let's insert data for the last couple of years into it and measure memory consumption:</p> input points: 345 k packed u64s: 56 k total size: 512 KB </code></pre> Oversized Vecs</h2> There's something odd with the numbers above: the Vec</code> should be smaller. </p> If we have 56k elements of 64 bits, the total sequence should have ~440KB, not 512KB.</p> After some experiments, I realized that the Vec</code>s were bigger than expected by 5% to 100%. In average, they were ~55% bigger.</p> I then remembered that such data structures usually double their inner buffers when there's no room for a new element.</p> Fortunately, Rust offers a Vec::shrink_to_fit()</a> operation, which we can call after finishing mutating our database.</p> After applying this change, the Vec</code> occupies the expected size.</p> Final results</h2> After fiddling with some thresholds I noticed that, to achieve a 5m precision instead of 15m, the memory overhead is negligible:</p> input points: 345 k packed u64s: 62 k total size: 485 KB </code></pre> So I decided to set the max error to 5m instead.</p> Overall we reduced memory footprint by 24x, compared to the most straightforward solution (using a stream for JSON deserialization).</p> Had we used a String</code> for storing the whole JSON first, the maximum RAM used would be 1900x bigger!</p> The final code can be found here</a>.</p> Conclusion</h2> I didn't need to invest so much bandwidth in this optimization: my PC could use 1GB of RAM for this data anyway.</p> But I learned a lot as part of the process and I ended up with results very efficient memory-wise.</p> That said, as mentioned on Reddit</a>, general-purpose compression algorithms</a> achieve similar results and simplify our code a lot. Querying is relatively more expensive, though.</p> But, depending on your needs, perhaps you can just gzip</code> everything and you're good to go!</p> A memory-efficient data structure for geolocation history Unknown — Mon, 31 Oct 2022 00:00:00 +0000 Note: please check this other post</a> instead, which is a more recent revision on this topic.</em></p> In this blog post I'll cover how I built a memory-efficient data structure for storing location history data.</p> Some Rust details will also be explained.</p> Motivation</h3> My DSLR camera has no GPS built-in. Evidently, photos from it don't include geolocation-based EXIF tags.</p> If only my location history were stored somewhere, so that I could update photos with the correct data...</p> Unsurprisingly, Google knows where I am, where I've been and where I'm going to be</del>. They even allow me to export this data.</p> The problem is that the exported JSON weighs ~1GB. My computer has 16GB of RAM so it can all fit in memory, but we certainly can store it more efficiently.</p> JSON parsing</h3> For reference, here's what the JSON looks like:</p> { "locations": [{ "latitudeE7": 435631892, "longitudeE7": 26848689, "accuracy": 56, "activity": [{ "activity": [{ "type": "STILL", "confidence": 100 }], "timestamp": "2014-01-09T00:48:24.424Z" }], "source": "WIFI", "deviceTag": 1348159918, "timestamp": "2014-01-09T00:48:24.751Z" }, { "latitudeE7": 435631881, // ... </code></pre> Deserializing a JSON file into a struct results in something that is certainly lighter than the JSON string.</p> The reason for this is that a string with the value "2022-10-22"</code> needs more bytes than, say, its equivalent Date</code> object.</p> The latter can simply throw away the hyphens, for example. Additionally, the number 10 can be represented with 4 bits instead of using two chars, which occupy at least 8 bits each.</p> So moving away from a textual representation is the first step in our journey.</p> Instead of using a JSON deserializer per se, I decided to iterate over each line of the file and ignore the lines I wasn't interested in.</p> Do we really need to store everything?</h3> A naive structure for our purposes is a sequence of DataPoint</code>s, where DataPoint</code> has datetime</code>, latitude</code> and longitude</code>:</p> [ (DateTime(2022,10,22,10,45,32), 27.5226335, 43.552225), (DateTime(2022,10,22,10,46,21), 27.5226382, 43.552237), // ... ] </code></pre> But isn't there some redundancy? The data is clearly not random.</p> There are some assumptions we can make to help us design a better data structure:</p> Data points are sorted by time</li> Neighbor data points are very similar unless I'm on a plane, my speed is at most 100 km/h, so in a minute I'll travel less 2 km</li> on foot I'll travel just a few meters </li> </ul> </li> Most of the days I stay inside a circle of 3km of radius</li> One data point per minute is more than enough</li> Errors in position up to 15m is something I'm OK with</li> </ol> How many bits do we need to represent a position?</h3> First let's try to understand how much each decimal place contributes to precision.</p> A quick search on Google gives us the following</a>:</p> - The first decimal place is worth up to 11.1 km: it can distinguish the position of one large city from a neighboring large city. - The second decimal place is worth up to 1.1 km: it can separate one village from the next. - The third decimal place is worth up to 110 m: it can identify a large agricultural field or institutional campus. - The fourth decimal place is worth up to 11 m: it can identify a parcel of land. It is comparable to the typical accuracy of an uncorrected GPS unit with no interference. </code></pre> If we want to have errors less than 15 m, then we need to be able to correctly represent the latitude and longitude up to the 4th decimal place.</p> If both coordinates are off by 0.0001° then the error will be close to sqrt(10^2+10^2) = 15 m</code>.</p> Given that neighbor data points are close to each other, we can represent deltas in position instead of absolute positions.</p> Let's first try using a single byte to represent a delta in latitude.</p> Given that we must represent the 4th decimal place precisely, we can have a range that goes from 0 to 2^7-1</code> = 127, where 0 translates to 0°; 1 to 0.0001°; and, as a consequence, 127 to 0.0127°. The remaining bit can be used to indicate if the difference is positive or negative.</p> 0.0127° of difference in both coordinates results in a distance of ~2.1km. This is good because, from assumption number 3, we can conclude that, for most days, most data points can be represented with deltas of only one byte for each coordinate.</p> This conversion simply consists of a rule of three, so I'll skip further details.</p> How many bits do we need to represent time?</h3> From assumption number 4, we can discard seconds from timestamps.</p> Let's say that we're looking at an interval of 20 years. 20 years is roughly equal to 10.5 million minutes.</p> To represent this number we need ceiling(log2(10.5 million))</code> = 24 bits.</p> Rust doesn't have a u24</code> type. Even though we could implement a u24</code> type with a tuple of (u8, u16)</code>, in the end our u24</code> would end up using 4 bytes instead of 3:</p> type struct u24 { msb: u8, lsb: u8 } fn main() { let x = u24 { msb: 1, lsb: 1 }; println!("{} bytes", x.deep_size_of()); // 4 bytes } </code></pre> This happens because of memory alignment</a>.</p> In Rust, we can circumvent this "limitation" by using the repr</code> directive:</p> #[repr(packed(1))] type struct u24 { msb: u8, lsb: u8 } fn main() { let x = u24 { msb: 1, lsb: 1 }; println!("{} bytes", x.deep_size_of()); // 3 bytes } </code></pre> But even clippy</a> complains about this, with the unaligned_references</code> warning.</p> So let's stick with a u32</code> for timestamps instead. Hey, at least we can now represent 8000+ years worth of data.</p> Getting our hands dirty</h3> At its core, our database will do the following:</p> fn add(&mut self, time: DateTime, lat: f32, lng: f32) { let point = self.low_precision_point(lat, lng); if error(point, lat, lng) > threshold { self.store_high_precision_point(time, lat, lng) } else { self.store_low_precision_point(point) } } </code></pre> Simply using the 0.0001°-based rule in the error()</code> function isn't enough because the error varies with your distance to the Equator. To be more precise, this function must calculate the haversine distance</a> instead.</p> This is more CPU-intensive but it's a cost we need to pay at the time of writing only.</p> Defining our structs</h3> Our data structure for low-precision datapoints can be a sequence of 2 1-byte positions, as mentioned above. We also need to have a reference to a high-precision point so that the final position is reference + delta.</p> Our data structure for high-precision can be a sequence of minutes plus both coordinates.</p> In the end we'll have something like this:</p> type Minutes = u32 type LatLng = (f32, f32) type LatLngDelta = (u8, u8) struct Db { high_precision_points: Vec<Minutes, LatLng>, low_precision_points: HashMap<Minutes, Vec<LatLngDelta>> } </code></pre> I'll skip the implementation details but you can check the full code here</a>.</p> Benchmarking</h3> After finishing everything, we can measure how many megabytes our data structure needs.</p> Let's insert data for the last couple of years into it and measure memory consumption:</p> input points: 345 k high-precision data points: 42 k low-precision data points: 280 k high-precision structure size: 768 KB low-precision structure size: 2583 KB sum size: 3351 KB </code></pre> In the end, less than 4 MB were occupied. </p> Oversized Vecs</h3> There's something odd with the numbers above: the Vec</code>s should be smaller. </p> If we have 42k elements of 323 bits</code> = 12 bytes each, the total sequence should have ~492KB, not 768KB.</p> After some experiments, I realized that the Vec</code>s were bigger than expected by 5 to 100%. In average, they were ~55% bigger.</p> I then remembered that such data structures usually double their inner buffers when there's no room for a new element.</p> Fortunately, Rust offers a Vec::shrink_to_fit()</a> operation, which we can call after finishing mutating our database.</p> Benchmarking</h3> Finally, here are our results:</p> input points: 345 k high-precision data points: 42 k low-precision data points: 280 k high-precision structure size: 499 KB low-precision structure size: 2339 KB sum size: 2838 KB </code></pre> Let's compare this to a naive Vec<Timestamp, f64, f64></code> implementation:</p> input_points (size_of_timestamp + 64 + 64) * average vec overhead 354k * (64+64+64) * 1.5 bits 11.8 MB </code></pre> Overall we saved around ~80% in memory!</p> After fiddling with some thresholds, I noticed that, to achieve a 5m precision instead of 15m, the memory overhead is negligible:</p> input points: 345 k high-precision data points: 46 k low-precision data points: 273 k high-precision structure size: 546 KB low-precision structure size: 2326 KB sum size: 2872 KB </code></pre> So I decided to set the max error to 5m instead.</p> Future improvements</h3> A high-precision data point occupies 32 bits for a timestamp and 32 bits for each coordinate. </p> Timestamps could be represented with 24 bits, as mentioned above. 32 bits for each coordinate is unnecessary.</p> We could use a u64</code> for everything instead: with some bit shifting, 24 bits could be reserved for the timestamp and each coordinate could use 20 bits.</p> This would decrease the size of these data points by 1-64/(32*3)</code> = 33%.</p> In addition, I wonder if we could use QuadTrees</a> somehow...</p> Conclusion</h3> This journey was clearly over-engineered.</p> But I learned a lot as part of the process and I ended up with results very efficient memory-wise.</p> Instead of asking "why?", I asked "why not?" 😂 </p> Faster monorepo workflow with materialized views Unknown — Mon, 13 Dec 2021 00:00:00 +0000 Context</h3> Monorepos have their pros and cons. </p> A plethora of in-depth articles have already been written on this subject so I won't bother writing yet another one. This one</a> summarizes the trade-offs very well.</p> In this post, I propose a solution for improving the dev experience for monorepos, by using microrepos as materialized views for subprojects and a bot as orchestrator.</p> A proof-of-concept CLI</a> for managing repositories is also provided, which can be used as base for a real-world tool set. </p> Assumptions</h2> To keep this post concise, I'll list some assumptions under which this solution was designed. </p> Depending on your workflow or the requirements of the code base you're developing, alternative solutions should be considered. </p> git must be used, as changing the SVN is too disruptive</li> the main drawback of monorepos are slow git operations in dev machines</li> if git were performant for large repos, monorepos would clearly be superior to microrepos, with few, if any, downsides </li> </ol> Proposed solution</h2> In a nutshell, devs work with materialized views an the bot propagates changes</figcaption> </figure> Repo setup</h3> 0.a. Create a monorepo mono</code> as the source of truth of our codebase. It should include all subprojects. Let's say it contains proj1</code> and proj2</code></li> 0.b. Create microrepos for proj1</code> and proj2</code>. They'll act as materialized views</li> 0.c. Protect the master</code> branch of all microrepos. Only the bot should be able to commit to master</code></li> </ul> Workflow</h3> Instead of cloning mono</code>, the dev should clone microrepos of interest. Let's say only proj2</code></li> The dev creates a branch newfeature</code> in proj2</code></li> After making the necessary changes, the dev pushes this branch to proj2</code>'s remote, not mono</code>'s</li> A bot creates a PR in mono</code>, reflecting the changes of proj2</code>'s newfeature</code></li> The dev hits the "merge pull request" button for the PR in mono</code></li> For each commit to mono</code>'s master</code>, the bot commits to the microrepos' master</code> accordingly. In this case, proj2</code>'s master will eventually include the changes from newfeature</code></li> </ol> Making changes to multiple repos at once</h2> The bot that creates the PR in mono</code> must be able to aggregate related branches from multiple microrepos.</p> In order for the bot to know if two branches are related or not, an identifier can be used. For example, if newfeature</code> was used as a branch name in proj1</code> and the feature requires changes to both projects, a namesake branch in proj2</code> must be created.</p> Why this solution is good</h2> In a nutshell, because the advantages of monorepos are kept. The only practical difference is that devs don't need to run slow git operations on their machines. </p> The good news is that all this can be abstracted away by a CLI. </p> Does this solution need to be so complex?</h2> I think so. </p> Multiple multi-billion dollar companies struggle with this problem. If there were a simple solution, I'm sure someone would already have figured it out.</p> The only simple solution (from end-user's perspective) I can think of is to have an SVN performatic for monorepos out-of-the-box. </p> Perhaps that's the case already, but a different SVN rejects our assumption 1.</code>.</p> Rejected solution</h2> Having the microrepos as source of truth and the monorepo as materialized view</h3> The problem with this solution is that changes to proj1</code> and proj2</code> must be atomic, assuming a feature requires changes to both: we either want to commit a change to both projects or drop the commit altogether. </p> git currently doesn't provide a solution for such transactions, so a mechanism for simulating atomicity would need to be designed, rendering the solution even more complex. </p> For example, a change to proj1</code> would need to be reverted in case we're not able to commit to proj2</code>. </p> As we all know, distributed systems can fail or become inconsistent for all sorts of reasons. If somehow proj2</code> got corrupted or inconsistent, it's much easier and less error-prone to fix or reconstruct its materialized view than trying to agree upon the source of truth.</p> Demo</h2> To illustrate, I could've created a GitHub bot</a>. That would exceed the time budget I set for putting this article together, though. </p> For demo purposes I've created a proof-of-concept CLI</a> that simulates the flow locally. This won't simulate the interactions with PRs, as they don't exist in a local machine, but will give us a clear idea of how this flow works. </p> In this example, all folders inside ~/github</code> represent repositories you would normally have hosted on GitHub; all folders inside ~/dev</code> represent the local clones. </p> Once this CLI is available in your $PATH</code> as git-monorepo, you can invoke it by running git monorepo</code>. </p> You can execute the commands below in your local machine if you want to follow along. The CLI prints all commands it's running for you to understand what's happening under the hood. </p> Without further ado, let's get to it.</p> 0.a. Setting up the monorepo</h3> Let's create the remote mono</code> repository:</p> mkdir -p ~/github/mono cd ~/github/mono git init mkdir proj{1,2} for i in 1 2; do echo "console.log('proj${i}')" > proj${i}/file${i}.js; done git add . git commit -am 'First commit' </code></pre> By the end of these steps, GitHub would host a monorepo like this:</p> mono/ proj1/file1.js proj2/file2.js </code></pre> 0.b. Setting up the microrepos</h3> Let's create the remote microrepos:</p> for i in 1 2; do git monorepo extract proj${i} ~/github/proj${i}; done </code></pre> The first argument is the path to the project inside mono</code>; the second argument is where the remote microrepo will live.</p> Normally, the second argument would look like https://github.com/username/proj1</code> or git@github.com:username/proj1</code></p> By the end of these steps, GitHub would host repositories like this:</p> mono/ .gitmonorepo proj1/file1.js proj2/file2.js proj1/ file1.js proj2/ file2.js </code></pre> The .gitmonorepo</code> was automatically created to keep track of the microrepos. </p> 1. Cloning a microrepo</h3> Let's clone proj1</code>:</p> mkdir -p ~/dev cd ~/dev git clone -b master ~/github/proj1 </code></pre> 2. Making changes</h3> Let's develop a new feature. </p> cd ~/dev/proj1 git checkout master git pull origin master git checkout -b newfeature echo "console.log('newchange')" >> file1.js git add . git commit -am "proj1/newfeature: change file1.js" </code></pre> 3. Pushing a change to the microrepo</h3> Let's push our changes to the remote proj1</code>:</p> git push origin newfeature </code></pre> 4. Propagating the changes to the monorepo</h3> The bot would automatically propagate the changes to mono</code>, by running something like the following:</p> cd ~/github/mono git monorepo pull newfeature </code></pre> Now, ~/github/mono/proj1/file1.js</code> should have the newchange</code> line on the newfeature</code> branch, but not in master</code>. </p> 5. Merging a PR</h3> Let's merge our branch:</p> git checkout master git merge newfeature </code></pre> Now, ~/github/mono/proj1/file1.js</code> should have the newchange</code> line on master</code>. </p> 6. Propagating the change back to the microrepo</h3> Finally, the bot would automatically update all microrepos accordingly, by running something like the following:</p> git monorepo push </code></pre> Now, ~/dev/proj1/file1.js</code> should also have the newchange</code> line on master</code>, ending the loop cycle. </p> Please note that we were able to make changes to the remote monorepo having only cloned proj1</code>. proj2</code> and mono</code> weren't cloned locally.</p> Future work</h2> We've only covered the simple, happy path so far. </p> Ideally, this system should also include:</p> a dev-friendly git wrapper for working with multiple microrepos at once</li> a UI for displaying how the microrepos and the monorepo are interacting with each other </li> different resolution strategies, in case one of the propagation changes fails for some reason</li> merge queues</li> a mechanism for replicating the monorepo locally, but using the microrepos of interest instead </li> cleanup routines, for deleting branches in microrepos whose PR in the monorepo is closed </li> fixes to these TODOs</a></li> and much more. </li> </ul> Conclusion</h2> The purpose of this article was to simply brainstorm what a more performant workflow could look like. </p> I hope that this will motivate someone out there willing to implement a system ready for real-world scenarios. </p> In case you do, I'd really appreciate if you could add a link to this post somewhere in your README.md file! :) </p> Working around Google Photos limitation Unknown — Mon, 07 Jun 2021 00:00:00 +0000 Context</h3> As of June 1, 2021, Google Photos started limiting</a> the storage for new photos and videos to 15GB/account.</p> 15GB isn't much nowadays, so I looked for alternatives. </p> Signing up for paid storage is an option which I discarded: </p> first, because I don't want to yet another lifetime subscription</li> second, because I don't want to be tied to a ecosystem. What if prices double overnight? What if I reach the storage limit and need to upgrade to the next tier?</li> </ul> That said, I love Google Photos and its machine learning wizardry.</p> A suboptimal experience, yet better than nothing, would be to:</p> upload low quality photos to Google Photos</li> upload high/original quality photos to alternative storages</li> </ul> This way, Google Photos will still keep me reminding me of past trips or allow me to search for "bridge" or "Rio de Janeiro" against my 1000s of photos. Given an image ID/date, I can download the high/original quality photo elsewhere.</p> Alternatives</h3> Whatever the solution would turn out to be, I wanted it to last at least 3.5 years. If we assume 10 photos/day in average, we're talking about ~13k photos here.</p> I ended up creating/reusing the following accounts:</p> Provider</th> Available storage</th> Photo size</th> Trust?</th></tr></thead> Google Photos</a></td> 6GB</td> 450KB</td> Yes</td></tr> Box</a></td> 7GB</td> 550KB</td> Yes</td></tr> pCloud</a></td> 10GB</td> 750KB</td> No</td></tr> Degoo</a></td> 100GB</td> 7MB</td> No</td></tr> Terabox</a></td> 1TB</td> irrelevant</td> No</td></tr> Telegram</a></td> ∞</td> irrelevant</td> Yes</td></tr> </tbody></table> I've never heard about pCloud</a>, Degoo</a> or Terabox</a> before so I'm not uploading my photos in a "human-readable" format to these services.</p> Solution</h3> I won't get into too much implementation details, but I basically used Tasker</a> and Termux</a>. I also installed the packages for file</a>, exiftool</a> and rclone</a>.</p> The pseudo-code is as follows:</p> every day at 2am: for each photo in /sdcard/DCIM: compress photo to /sdcard/Cloud/GooglePhotos/⋯.jpg compress and zip photo to /sdcard/Cloud/pCloud/⋯.jpg.7z ... zip photo to /sdcard/Cloud/Terabox/⋯.jpg.7z move photo to /sdcard/Cloud/Telegram/⋯.jpg every day at 3am, if wifi is connected: for each file in /sdcard/Cloud/<provider>: move file to <provider> </code></pre> Compression</h3> A photo is compressed for each storage provider accordingly.</p> The output image will be limited by Photo size</code> and, if Trust?</code> is false</code>, it will be 7zipped using a password.</p> The output image size is limited by playing with the properties for image resizing inside Tasker (max dimension</code> and compression quality</code>). These are estimated based on the Photo size</code> input and the original image properties, retrived by file</code>.</p> One caveat is that, for some reason, Tasker doesn't preserve the EXIF attributes when resizing images, so I needed to use exiftool</code> to overwrite the new file EXIF attributes with the original ones.</p> Uploading files to decent, real storage providers</h3> This was very straightforward and was automated using rclone</a>.</p> Uploading files to bad, real storage providers</h3> Degoo</a> and Terabox</a> don't have support for WebDAV or similar protocols. In other words, I need to open their app and manually move files.</p> I plan to do that once a month.</p> The good news is that all files will already be easily located in /sdcard/Cloud/Degoo</code>, for example.</p> Uploading files to Telegram</h3> Telegram</a> isn't a storage provider but, to my knowledge, a given chat can have infinite attachments. 🤷</p> Its API allows file uploads using a simple POST request. The response body contains a file_id</code>.</p> We can download the attachment by making a request to https://⋯.telegram.org/⋯?file_id=<file_id></code>.</p> We thus need to keep this file_id</code> in a database. My Tasker task stores this in /sdcard/Tasker/db/telegram_uploads.txt</code> and its contents look like this:</p> /Pictures/IMG_00.jpg;123456 /Pictures/IMG_01.jpg;789012 /Pictures/IMG_02.jpg;654321 </code></pre> Having this database in hand, we can even design a primitive web-based file manager, for example.</p> The database is replicated across all other storage providers, using the aforementioned methods.</p> My chat with my bot will ultimately contain thousands of messages like this</figcaption> </figure> Conclusion</h3> I'm happy with the solution because Google Photos will still keep doing its magic and photos are replicated. If one storage provider goes down (I'm sure that down the road at least one of them will), I won't lose my photos.</p> Code</h3> The Tasker tasks aren't public because they rely on a very specific setup (and because I've never met anyone in person who uses Tasker - or Termux, for that matter).</p> In the unexpected scenario I get requests to do it, I'll share them in Taskernet</a>.</p> Using a spreadsheet as a timeseries database for finance Unknown — Wed, 12 May 2021 00:00:00 +0000 Context</h3> I have already tried using personal finance apps but there's always a feature missing or an inconsistency in results. </p> Unhappy with the experience, I created a simple spreadsheet half a decade ago. It lacked features and quickly became unmaintainable.</p> I decided to build my own tool instead. I started with a React (Native) Clojurescript web/mobile app. I never finished it. </p> I created a system using Clojure and Python, paired with Grafana</a>. It worked pretty well, actually, but I never managed to deploy it in a free/cheap hosting service, because of JVM's and pandas' requirements.</p> I converted the codebase to Rust as a pet project. The memory and processing footprints became negligible. Cool! </p> But every once in a while I need to perform a simple calculation on the data, or plot an ad-hoc graph, or explore something new. Cloning repositories, editing the source code and recompiling everything isn't fun. </p> Spreadsheets are good at that. Ironically, I decided to go back to Google Sheets. But this time for real. </p> Architecture diagram</h3> Here's the diagram of the complete system:</p> </figure> I'll cover each section in more details.</p> Using Google Sheets for input</h1> One of the requirements was that, as before, input should be minimal. In this case, it should basically be an event log of money transfers: </p> The first 2 and last 2 columns are filled automatically</figcaption> </figure> An extra sheet was necessary to completely define some investments. NASDAQ:APPL</code> is all the input you need to define Apple's stock -- provided you get data externally --, but some fixed income investments have their own criteria -- e.g. rate of interest. </p> This sheet only contains data which isn't possible to obtain otherwise</figcaption> </figure> Using Google sheets for processing</h3> Handling input was the easy part. Fetching data from APIs and doing the math is complicated</strike> laborious.</p> I can't get my head around =ARRAYFORMULA(IF..., VLOOKUP(...FILTER(...(INDEX...(MATCH...)))))</code> formulae. Fortunately, Google Sheets allows using Javascript for manipulating sheets. It's not Rust</strike> my favorite language but at least it has .map()</code>, .filter()</code> and alike.</p> After so many attempts building the same system, I came up with a mental model which I didn't want to abandon so I replicated it once again.</p> To implement a timeseries database, a single spreadsheet was used. The y-axis represents time; the x-axis, different vectors:</p> This sheet contains all numeric data</figcaption> </figure> In Javascript we can read from and write to this sheet as if we were using a real timeseries database. Effectively, this made my new code look very similar to the previous ones:</p> // old influx.emit(today, 3.12) // new tsdbSheet.setValue('D67') = 3.12 // old config = readFromFile('./config.yaml') foo = config.foo // new foo = configSheet.getValue('A2') </code></pre> Abstractions</h3> These low-level spreadsheet operations were abstracted away, in order to make the rest of the code agnostic to Google Sheets. Should I decide to port it elsewhere, I'll only need to edit some parts from the TSDB</code>, EventLog</code> and Config</code> classes.</p> For example, calculating balances looks like this:</p> config.assets.forEach(asset -> { const balance = range(startDate, today).map(date => { const value = someMath(asset, date, tsdb) return [date, value] } tsdb.save(asset.id, 'balance', balance) }) </code></pre> Getting historical data for stocks isn't any different:</p> config.assets .filter(asset => asset.kind == 'stock') .forEach(stock => { const price = callSomeApi(asset.id) tsdb.save(asset.id, 'price', price) }) </code></pre> Similarly, to prevent some input, the following is executed on startup:</p> config.assets .filter(asset -> asset.reachedMaturity()) .forEach(asset -> eventLog.save('sell', asset.id, asset.maturityDate)) </code></pre> Finally, this allowed me to achieve what I already had in the early designs but with 3 benefits:</p> I can access it from anywhere, as long as the device supports Google Sheets</li> I edit tabular data, instead of a difficult-to-maintain .yml</code> file versioned in a private Github repository</li> I can quickly run an ad-hoc =SUM(FILTER(...)))</code> against my data if necessary </li> </ul> Dashboards</h1> Spreadsheet formulae are cool but I still like looking at dashboards and plotting graphs on the fly, so I reused the architecture I already had on top of Grafana. I don't want to share screenshots here but the dashboard looks like any regular Grafana one:</p> Grafana dashboards normally look like this</figcaption> </figure> For example, I can quickly write a query to compare the portfolio performance against the Ibovespa index</a>.</p> To get the performance of assets broken down by broker, sum(...) by (broker)</code> would do the trick</figcaption> </figure> (Real) timeseries database</h3> I decided to VictoriaMetrics</a> this time. It's fast and it has support for a much needed feature: backfilling. In other words, I'm able to emit values for timestamps in the past. </p> Its query engine is called MetricsQL and is a superset of PromQL</a>. </p> I'd like to highlight 3 features from MetricsQL:</p> keep_last_value</code>: only one data point per day per series is emitted. Anything above this would be redundant, since I don't day trade. However, Grafana asks for n</code> data points per day, n</code> function of the width of the panel in pixels and is normally greater than 1. Without keep_last_value</code> the plots would contain gaps. Summing metrics with gaps in different positions is erratic, because gaps are interpreted as 0.</p> </li> WITH</code> statement: this lets query aliasing, which is pretty handy. By the way, if you like WITH</code> too, you may like pipers</a>. </p> </li> range_first</code>: with this function I was able to normalize investment performances so that they all start with 1, making comparisons a breeze, as seen above. </p> </li> </ul> Exporting/importing data</h1> I added an Export</code> button to Google Sheets which exports the contents from the spreadsheets to Grafana:</p> It's easy to call custom scripts like this</figcaption> </figure> It simply serializes the in-memory data structures as JSON text and sends it in a POST request.</p> Notes on performance</h3> The Rust implementation was able to read the config file, make all requests in parallel, calculate everything and export all data to Grafana in ~5 seconds.</p> I expected my first implementation in Google Sheets to be slower, or course. But I didn't expect it to be SO SLOW. The first implementation took 45~60min to finish processing. At least that's what I estimate, because I wasn't patient enough to wait for it. </p> The problem is that sheet.setValue(...)</code> takes 500ms~3s for each cell. Also, switching between read and write modes takes about the same time. If you consider that each investment generates ~10 timeseries -- average price, balance, etc. -- and that each timeseries has 365 data points per year, then you'll see that this quickly explodes.</p> This reminded me of the problem that frameworks like React faced. Manipulating the DOM is slow, so they ended up creating a virtual representation of the DOM</a>. I did the same, but for spreadsheets. It sounds complicated, but it isn't.</p> Basically, tsdb.save(...)</code> doesn't manipulate the sheet directly but mutates an in-memory data structure instead. Later on, it sets the values for ranges instead of cells, in batch. The "public" APIs remained the same -- only the implementation details changed.</p> After these optimizations, the script takes less than a minute to complete.</p> This could be improved even further, but I don't want to make the code more complex just to save a few seconds once a month. </p> Result</h3> To be honest, it's likely that a couple of months</strike> years from now I'll decide to write everything from scratch yet again. </p> It's not you, project of mine. It's me. </p> But I'm sure of one thing: the system has never been so flexible and portable.</p> Codebase</h3> The spreadsheet and the source code are currently private because it would require some effort to make them shareable -- from privacy, security and usability standpoints. </p> Please reach out to me if you're interested in it, though. Maybe I can come up with something! 👍</p> In the meantime, I highly recommend dlombello's spreadsheets</a>, which should fit most people's needs.</p> Creating templates for CLIs Unknown — Wed, 13 Jan 2021 00:00:00 +0000 Inspiration</h3> I've been using yab</a> —a curl-like CLI— recently and I was surprised by how elegantly it solves a common problem.</p> yab calls can get quite verbose. As an example, to get a Customer</code> by its id</code> from a customer</code> microservice:</p> yab customer Customer::get \ -t /path/to/idl/some.company/customer/customer.thrift \ -P 131.144.23.5/customer:tchannel \ -r '{"request": {"id": 11}}' </code></pre> The call is pretty convoluted, but it can be simplified by storing params in a yaml file:</p> # customer_by_id.yaml service: customer procedure: "Customer::get" thrift: /path/to/idl/some.company/customer/customer.thrift peer-list: 131.144.23.5/customer:tchannel request: request: id: ${number:11} </code></pre> The file takes a set of CLI flags in their long form, e.g. --thrift</code> instead of -t</code>, and their associated values.</p> Then, the following call is equivalent to the beforementioned one:</p> yab -y customer_by_id.yaml </code></pre> Additional flags can be passed to yab to override the default values specified in the yaml file:</p> yab -y customer_by_id.yaml --number 42 </code></pre> I think this is pretty cool!</p> It can save us lots of time. It also helps sharing knowledge between team members: we could have a git repo with a bunch of these .yaml</code> files and a simple git pull</code> would allow everyone to be on the same page.</p> My immediate thought was "could we easily replicate this for other commands?"</p> Solution</h1> For the sake of simplicity, let's try to write a complex httpie</a> command. It could be an involved awk</code>, grep</code> or jq</code> command, though.</p> The final call should look like this:</p> http -v POST https://jsonplaceholder.typicode.com/posts title=foo body=bar userId=11 </code></pre> The tool I'm going to use is navi</a>. It allows you to browse through cheatsheets —that you may write yourself or download from maintainers— and execute commands. </p> navi encourages you to write .cheat</code> files which break commands down into smaller, reusable pieces:</p> % httpie # make a request to a typicode microservice http -v <method> "https://<service>.typicode.com/<endpoint>" <http-body> $ method: echo -e 'GET\nPOST\nPUT' $ service: echo -e 'jsonplaceholder\nanotherservice' $ endpoint: case "${service}:${method}" in; "jsonplaceholder:post") echo -e 'e1\ne2\ne3';; esac </code></pre> The endpoint</code> values here are incomplete for briefness and, in a real-world scenario, would probably be fetched dynamically, instead of being harcoded.</p> This .cheat</code> enables us to very quickly make a request to any endpoint in our company, given that we somehow mapped all possible values to the corresponding variables. </p> But an engineer who recently joined the team still wouldn't know what service/endpoint to request for creating a new post</code>. We could then add another cheatsheet entry for more granular commands:</p> % httpie, endpoints # create a post http -v <method> "https://<service>.typicode.com/<endpoint>" title=<title> body=<body> userId=<userId> $ method: echo 'POST' $ service: echo 'jsonplaceholder' $ endpoint: echo 'posts' $ title: echo 'foo' $ body: echo 'bar' $ userId: echo '11' </code></pre> There are a multitude of ways to invoke this cheatsheet. One of them is like this:</p>

Conclusion</h2>
Hope this post was helpful!</p>

A memory-efficient data structure for geolocation history

`Conclusion</h3> This journey was clearly over-engineered.</p> But I learned a lot as part of the process and I ended up with results very efficient memory-wise.</p> Instead of asking "why?", I asked "why not?" 😂 </p>`

Faster monorepo workflow with materialized views

Working around Google Photos limitation

Conclusion</h3>
I'm happy with the solution because Google Photos will still keep doing its magic and photos are replicated. If one storage provider goes down (I'm sure that down the road at least one of them will), I won't lose my photos.</p>

Using a spreadsheet as a timeseries database for finance

Creating templates for CLIs

Denis Isidoro

Trying to land an offer at a big tech company? These YouTubers will help you

Intro</h2>
I've recently landed an offer I was really looking forward (yay! 🎉)</p>
The least I could do was to give some shoutouts to the YouTubers which made this possible.</p>

A memory-efficient data structure for geolocation history (rev2)

Denis Isidoro

Trying to land an offer at a big tech company? These YouTubers will help you

Intro</h2> I've recently landed an offer I was really looking forward (yay! 🎉)</p> The least I could do was to give some shoutouts to the YouTubers which made this possible.</p>

Coding</h2> No mystery here: Neetcode</a>.</p> </iframe> </div> The dreaded dynamic programming questions</figcaption> </figure> He solves Leetcode problems, which is all you'll need.</p>

Conclusion</h2> Hope this post was helpful!</p>

A memory-efficient data structure for geolocation history (rev2)

A memory-efficient data structure for geolocation history

JSON parsing</h3> For reference, here's what the JSON looks like:</p>

Faster monorepo workflow with materialized views

Context</h3> Monorepos have their pros and cons. </p>

Repo setup</h3> 0.a. Create a monorepo mono</code> as the source of truth of our codebase. It should include all subprojects. Let's say it contains proj1</code> and proj2</code></li> 0.b. Create microrepos for proj1</code> and proj2</code>. They'll act as materialized views</li>

0.b. Setting up the microrepos</h3> Let's create the remote microrepos:</p>

Working around Google Photos limitation

Conclusion</h3> I'm happy with the solution because Google Photos will still keep doing its magic and photos are replicated. If one storage provider goes down (I'm sure that down the road at least one of them will), I won't lose my photos.</p>

Using a spreadsheet as a timeseries database for finance

Architecture diagram</h3> Here's the diagram of the complete system:</p> </figure> I'll cover each section in more details.</p>

Using Google Sheets for input</h1> One of the requirements was that, as before, input should be minimal. In this case, it should basically be an event log of money transfers: </p>

Creating templates for CLIs

Intro</h2>
I've recently landed an offer I was really looking forward (yay! 🎉)</p>
The least I could do was to give some shoutouts to the YouTubers which made this possible.</p>

Conclusion</h2>
Hope this post was helpful!</p>

Conclusion</h3>
I'm happy with the solution because Google Photos will still keep doing its magic and photos are replicated. If one storage provider goes down (I'm sure that down the road at least one of them will), I won't lose my photos.</p>