Introduction

This was a difficult but important book to read. The legacy of slavery and racism on our society is both overt and subtle. Much of it is ignored and glossed over but some people are trying to integrate a more historically accurate interpretation into our institutions and lives.

Thomas Jefferson

The first part of the book takes place at the farm/plantation of the founding father Thomas Jefferson. The reality is that slavery was a big part of all the founding fathers life and business, but Jefferson has some especially egregious behavior. He split families, tortured slaves and even had six (!) children with a slave that was his wife’s half sister! The estate is trying very hard to integrate this lesser told story into the traditional story and it is a difficult task. How can the misery and suffering of the slaves be reconciled with the fact that they allowed him to do his incredibly important work?

Whitney Plantation

This plantation is an independent project of a wealthy retired businessman to preserve the stories of the slaves that sustained it. It is incredible work that I am glad is being done. The reality of our history needs to be preserved and integrated, it cannot be ignored and buried. I hope to be able to visit this important contribution.

Modern Day Slavery in Angola

Angola is a maximum security prison in Louisiana that is filled with black men preforming forced labour for cents on the hour. It is modern day slavery and it highlights the corruption of the justice system as an overt means of continuing slavery. All talk of reconciliation is empty as long as this practice continues. This section was so infuriating.

Juneteenth and Manhattan

The actual events of Juneteenth were fascinating to me. The end of slavery in the South after the Civil War was a messy crazy process. I was not aware that plans to give slaves plots of land was rejected last moment. How different things might be today if it had happened.

The legacy of slavery in Manhattan is also something not talked about often. The economics of slavery were embedded throughout all of states and New York was no exception. Just because it was on the “winning” side doesn’t mean that it doesn’t bare any responsibility.

Blackford Cemetery

The Blackford cemetery is a Confederate cemetery and it is a focal point for Confederate culture. Is it possible to honor the soldiers without getting tangled in their implicit support of slavery? Is it possible to celebrate a culture when that culture justified a war to maintain slavery? It is a similar situation to post war Germany where everyone has to grapple with the fact that normal people enabled the slaughter of millions of people. What I experienced is that Germans have reframed the teaching of the history in terms of “Never Again”, i.e. we must teach this to ensure that it can never happen again. Unfortunately I don’t see this kind of hard work being done in the US.

Conclusion

I don’t think we will be able to address the legacy of slavery and racism without a clear acknowledgment and acceptance of the past. I really appreciated the way this book presented the historical blind spots of US history. Highly recommended.

Introduction

These books changed the way I see and think about our modern world. They cover what we know today about the American continent before Christopher Columbus arrived in the “New World” and the many ways the entire world was changed forever afterwards. Mostly for my own benefit I will list a bunch of the things I learned and perhaps this will encourage you to read these books as well.

Impact of Pathogens

I had heard before about the impact of various pathogens brought over by the Europeans, but I didn’t appreciate the scale. The exact population of the Americas is impossible to know, but there is lots of evidence for as high as 100 million people. These people had to deal with multiple waves of influenza, typhoid, small pox, yellow fever, malaria, etc. and over 50 years this probably killed over 95% of the population.

The consequences of this are mind boggling. The Indians (there doesn’t seem to be a better word for the people that lived in the Americas before Columbus) did controlled burns of the forests for thousands of years and this stopped. Without this burning, the ecosystem completely changed. Huge forests grew, herds of bison formed and the growth may even have been responsible for the mini ice age in the 1600s due to forests capturing so much CO2.

All attempted European settlements failed until the Indians were wiped out. There are so many stories of how ill prepared the settlers were for life in America. It wasn’t until they weren’t in direct competition with the Indians that settlement had a chance of succeeding. Also the weakened Indian tribes often formed alliances with the settlers in their own conflicts. These alliances never worked out for the Indians as the Europeans once established would systematically wipe out the Indians.

Impact of joining two separate ecosystems

The Americas contained ecosystems completely isolated from the rest of the world. As the world became a single global ecosystem there were many winners and losers. America didn’t have worms which is one of the reasons the controlled burns were so critical. Now worms are everywhere and the way they break down biological material has profound consequences for the native plants.

The Americas contained three plants that changed the world: tomatoes, corn and potatoes. It is hard to imagine cuisine today without tomatoes and to think that something as “universal” as spaghetti and tomato sauce is a very recent phenomenon. Corn is the backbone of our industrial agriculture feeding cattle and converted to corn syrup and various other food additives. The potato alone has been attributed to have allowed the human population to grow by billions. Impossible to imagine our society without these plants, not to mention chocolate or tobacco.

American agriculture

The early settlers often remarked at how healthy the Indians looked. Turns out the Indians ate a diet of corn, beans and squash that was nutritionally superior to the European diet of wheat and meat. The Indians didn’t have any pack or domesticated animals. That means all farming was done by hand and all communications had to be walked. The Europeans mistook that lack of visible farms as sign that the land was “unused”. But without oxen to pull plows giant farms aren’t feasible and the Indians had very different forms of farming and hunting. Even in the Amazon, there are strange super fertile regions that contains millions of pottery shards mixed in with the soil. We still don’t understand how this works or was possible.

Silver from the Andes

After the Incas were defeated and enslaved, the Spanish found a silver mine in the Andes that had been mined by the Incas. The ore was very pure and plentiful and soon the silver was moving around the world. It was supposed to go directly back to Spain to fund various wars.Some enterprising Spanish sailors realized they could run the silver across the Pacific, trade with the Chinese for silks and spices, cross the Pacific, Mexico, the Atlantic and make a fortune. Almost two thirds of the silver went to China and even the one third to Spain was enough to trigger massive inflation in both countries. This inflation was the proximate trigger of various regime changes.

Rubber

The industrial revolution requires three things: steel, oil and rubber. I underappreciated the critical role that rubber plays in all the gaskets, seals and tires that are part of modern machines. The rubber/latex tree is still a core part of our economy and now grows all over the world. Rubber tree farms cause all kinds of ecological disasters and the rubber boom of the late 1800s also caused massive economic upheaval. It is another example of a natural product for which we haven’t been able to make an economical substitute.

Slavery

I was not aware of the link between malaria and slavery. Malaria was so deadly to Europeans and Indians that settlement in the Americas was almost a death sentence. It was because the Africans had more natural immunity the malaria that drove much of the Atlantic slave trade. Many African slaves were in fact prisoners of war between African tribes and so many slaves had military training. Unsurprisingly many escaped their slavery and formed “free” communities. One incredible example is a state in modern day Brazil that survived for almost 90 years. It was strategically placed on a cliff side with access to water, etc. The populace was trained and they were able to resist many attacks. It is such an amazing story that I hope it becomes a movie some day.

Conclusion

This is just a fraction of the incredible history of the Americas that I am so grateful to have been able to learn about. Highly recommended.

Batch Jobs in Kubernetes

The design of Kubernetes has its origins at Google as a platform for microservices. It does support batch jobs but they do not have the same level of support as microservices. The only difference between a Pod and Job is that a Job does not restart automatically.

Pod Abstraction

The Pod abstraction assumes that the CPU and memory usage of the processes running inside a Pod are predictable and fairly constant. When a Pod’s memory usage is larger than its allocation, the assumption is that there is a bug or memory leak and the process should be killed.

Best Effort resource allocation

For Pods the default resource allocation is “Best Effort”. When no CPU or memory limits are defined the Pod is considered low priority and does not have any resource limits. During my initial explorations with K8s I created 10 Best Effort Jobs that compiled software. K8s started all the Pods on a single Node. The compile jobs quickly used up all available memory on the system and K8s started killing Jobs at random.

This behavior makes sense when the Pods are stateless and part of Deployments and con be easily moved to other nodes. Of course the compile jobs can just be restarted but it is wasteful to throw away the work in progress. In this case “Best Effort” doesn’t really seem appropriate for running batch jobs.

PodInterAntiAffinity

Using the Pod AnitAffinity with the nodename and/or labels, the K8s scheduler will spread the Pods out over a set of Nodes. This way if the nodes have enough resources “Best Effort” pods will have the best chance of not consuming too many resources on the nodes. But it isn’t a guarantee and it is difficult to predict if a Job/Pod will finish.

Burst and Guaranteed Pods

If “Best Effort” isn’t appropriate, then the next step is to give each Job a CPU and memory limit. The question then becomes what those limits should be? A Burstable Pod has lower limit and max limit. A Guaranteed Pod has a max limit. A Burstable Pod allows a form of resource over commitment. If the memory usage of the processes in the Pod ever tries to allocate more memory than the resource limit, the allocation will fail and K8s will kill the Pod.

Compressible and Uncompressible resources

The default Pod resources CPU and Memory have different behavior when the resource is exhausted. CPU over commitment is handled by the kernel scheduler and the scheduler will allocate CPU time based on the scheduler policy. A Pod that attempts to use more CPU time than allocated will be throttled. CPU is considered a compressible resource.

Memory is different because if there isn’t any memory available, attempted allocations will fail. This is considered a uncompressible resource. There isn’t a way to throttle process memory usage. The only option is swap memory which can allow allocations to proceed but it comes with problems like thrashing. With containers it gets even more complicated because by default the kernel does not account for swap memory in the memory resource usage.

Setting memory limits

So the only way to prevent a Pod from being killed when it uses too much memory is to set the memory allocation high enough. This is where the predictability of microservices makes figuring out the max memory limit easier.

Finding a memory limit for Yocto builds

But what about large and unpredictable Yocto builds? Yocto provides infinite configuration options and also supports two forms of build parallelization: jobs and parallel packages. These options speed up the build but make the the package build order non deterministic. SState can be used to speed up builds but it makes predicting memory usage even more difficult because it is impossible to know ahead of time which packages will be rebuilt.

Swap?

What about using swap to add memory temporarily to a Pod that has used all of its allocation? K8s does not support swap and will require that swap be disabled. As far as I can tell the main issue is around how to account for the swap memory. Swap cannot just be added to memory of the machine because it is much slower. The current kube tools do not track swap usage and the kernel does not enable swap memory accounting by default for performance reasons. Swap can increase performance by removing unused memory pages to disk and giving applications more RAM. However a Pod using swap could thrash the entire node causing failures of Pods that are not using too many resources.

There is work underway to add swap alpha support to K8s 1.22 but it has many limitations and may be restricted to “Best Effort” Pods. Exactly how swap will be managed and accounted for are still open questions. At some point swap may be a part of the solution but it isn’t feasible now.

Workarounds

Without a way to make memory compressible there are only workarounds and tradeoffs.

• reducing parallelism for large packages like tensorflow or chromium does keep memory usage down at the cost of slowing down the build. When a build fails due to resource allocation failure we track the current packages being compiled from the process list and use that to identify potential problem packages.
• Tracking memory usage of a Pod using monitoring systems like Prometheus allows us to track changes in memory usage over time. It also gives us a better picture of how the memory usage of Yocto build changes over the course of the build. Ideally the processes causing peak memory usage can be identified and reduced to keep memory usage more stable allowing for better utilization of the nodes resources. The Prometheus alerting system can also be used to warn when builds cross memory usage boundaries like 90% which may give us time to deploy workarounds before builds fail.
• A build that failed due to a failed memory allocation could potentially be restarted with lower parallalization and/or changed memory limits. This is tricky because our builds use local disks with HostPath and the build Pod would need to be rescheduled to the node with the in progress build files. K8s 1.21 has added improved support for local volumes. Local volume management in K8s deserves a post of its own.
• The make tool has an option –load-average which tells make to no longer spawn new jobs if the load average is above a specific value. This doesn’t work for Pods because load is system wide and uses CPU but concept of feedback into make or ninja is interesting. Since a process can use the /proc filesystem to monitor the memory usage of a container is might be possible to have these tools reduce the number of spawned jobs based on current container memory usage.

Conclusion

It is the combination of the following:

• memory being uncompressible
• K8s/container memory limits are hard limits without second chances
• Yocto builds have unpredictable memory consumption

that makes this a very difficult problem. The only “proper” solution would be to make the builds have more predictable memory usage. This would require a feedback mechanism to make/ninja/bitbake to adjust the number of running processes based on the current container memory usage.