Rendered at 23:07:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dlopes7 1 days ago [-]
Very interesting read, these bugs that only happen after a lot of time in production are the worse to troubleshoot
ReactiveJelly 1 days ago [-]
So a stronger type system in the boundary between Go and Linux would have caught this?
fn-mote 1 days ago [-]
My reading is that this is entirely Go’s fault. In Go, there was a union of two different types without metadata to distinguish the two. The disambiguation was done by looking to see if the data looked like a certain pointer.
I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)
inigyou 1 days ago [-]
It's 64 bits on both but only 32 bits were read on 32-bit systems. Two 64-bit values that were different were considered the same because only 32 bits were checked.
nasretdinov 1 days ago [-]
Go was born into 64-bit era, I remember seeing many more bugs that only occur on 32-bit systems. I'm pretty certain the observation in the article is correct that no one at Google tests Go on anything apart from 64-bit ARM/x86 Linux
typical182 1 days ago [-]
FWIW, every commit for the Go project is tested on multiple 32-bit architectures, including pre-commit tests and post-commit tests.
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
orphereus 1 days ago [-]
Very cool site. Makes me want to do some system programming project just to make a site with all the tests like this.
znpy 23 hours ago [-]
interesting that they build for solaris as well
budman1 1 days ago [-]
Isn't this something that software developers do everyday?
apitman 1 days ago [-]
The way I look at these types of difficult debugging problems has completely changed.
If it's a system I want to understand more deeply and I have the time, I try to force myself to do it by hand. If not, the agent does it.
The trap is that it's easy to convince myself that I never have the time.
budman1 1 days ago [-]
Not being a user of agents, my question is 'does the agent always find the root cause and remove the defect'?
The way I got good at finding the hard ones was finding hard ones. Without that practice, I would be helpless.
apitman 2 hours ago [-]
I've found agents very effective at both idebtifying root causes from vague descriptions, and explaining both the problem and the fix to me.
It's not a panacea though. Sometimes it comes up with bad causes and/or solutions.
k4rli 1 days ago [-]
Reason why it's super hard for juniors at the moment. For "agents" to produce good results, the operator must have the experience to direct it in the right direction and properly validate the output.
I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.
BlackRabbit1 2 days ago [-]
Not directly related to your embedded Go bug, but maybe helpful:
What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.
DarmokTanagra 2 days ago [-]
I wonder what other 32 bit bugs are waiting to be discovered.
I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)
You can see a list here:
https://build.golang.org/
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
If it's a system I want to understand more deeply and I have the time, I try to force myself to do it by hand. If not, the agent does it.
The trap is that it's easy to convince myself that I never have the time.
The way I got good at finding the hard ones was finding hard ones. Without that practice, I would be helpless.
It's not a panacea though. Sometimes it comes up with bad causes and/or solutions.
I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.
What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.