因为一时也想不到什么很好的写作顺序，就大致按着自己脑中的关键词和时间线来回忆吧。

科比去世

没错，这个居然是脑海里第一个跳出来的关键词，惭愧。

其实我已经好久没看球了，我也向来算不上什么体育运动爱好者。喜欢打篮球的原因大抵是初中高中实在太无聊，需要一些不会被老师家人反对的兴趣爱好来释放压力。

但篮球也确实给我那些年带来了无数欢乐。还记得高中我还叫嚣着幻想湖人继续夺冠，一喊就是三年，连我们班不看球的人都知道有我这号脑残粉天天在那叫嚣（笑）。

然而今年科比去世了，湖人夺冠了，我早不看球了，一晃10年了。很是唏嘘。

前短时间还看到朋友圈有好友分享安东尼回掘金的新闻，甚至回想起初中那会每天放学在公交车上用同学的手机靠2G网看腾讯体育新闻的日子，很是怀念。

我觉得我爱科比的原因非常简单，fancy。我觉得直到今天我依然对很多fancy但不那么实用的东西有莫名的好感（such as PL research，笑）。

科比的去世真的带走了心里很多的回忆。年初刚发生这件事的时候我还被一些同事笑话，毕竟和今年其他事比起来这好像实在上不了台面。

来美国前我就想着要在美国实现的三个小心愿：

1. 看一场科比的球
2. 听一场Linkin Park的演唱会
3. 听一场Green Day的演唱会

这些都是我最真实的青春。然而我来的前一年科比退役了，我来美国之前Chester自杀了。好像我什么都慢了一步。希望自己还能有希望再朋克一把吧（笑）。

另外一个让我难过的是竹内结子的去世。本来想单独分一个section来几年一下我迷恋日剧多年最喜欢的女神。不过想来想去想说的话也大抵相同。有些生命就这样突然消逝了，伴随着一些很美好的记忆也就这样在脑海中埋藏起来，我还是要好好活下去。

Master毕业

这大概可以算我可以骄傲一下的事。来A&M的三年我大概一直在纠结要不要在导师手下直接拿PhD，感谢一些不愉快的事情让我看清楚了老板的本质，也下定决心要离开。

做决定的时候大概是2月，然后一直疲于工作到4月中旬，花了一个月左右从零把论文答辩都搞定了。那段时间人真的非常不好，每天都是焦虑到睡不着和对未来的纠结。

有一种自己很久的努力都喂狗了的气愤。好在结果是好的，我毕业了，老板在短暂的生气过后也没有为难我。我自觉得做了一个不错的答辩，也通过写论文好好捋了捋这3年做过的事，并从中找到了一些自己的价值。

很惭愧自己和那些真正的PL大佬比太弱了,我所做的科研在我看来都没有入PL的门。曾经一度打开推特看到一堆看不懂的讨论就无限自闭。

不过焦虑是没有用的，尤其在和Master了断了以后我也想的很开了。不论做什么事，时间的鸿沟是最难跨过的，与其焦虑，期待有捷径可走，不如就按照自己的节奏把时间花在自己感兴趣的事上。正视自己的弱，但不要过分贬低自己的工作，才是正确的态度。

我大概还是会走的慢一点，毕竟我是一个花了3年才毕业Master。不过不妨碍我更加自信地在这条路上走下去。

科研

其实这一年做过的科研可以说trival。毕竟我毕业以后老板直接把我从一些GitHub repo里移除了（笑）。

但最让我骄傲的是我们的SC paper被接受了。这大概是我3年在组里唯一觉得solid的工作，毕竟是自己从零写出来的，从代码到实验到论文，和Brad不知道熬了多少个夜。终于觉得自己做出了一点微小的贡献。

此外很感谢今年的ICFP mentoring program。让我在毕业以后还有项目可做，还认识了一些很nice的同事和教授。

然后自己莫名其妙的还混到了operations committe。我也不知道自己当时怎么想的，希望能帮到更多和我一样被折磨过的年轻人吧。

虽然我知道这是废话，但通过mentoring program，我也亲自确认了一件事：这个世界上就是有热爱科研和编程，并且为人师表的教授的。

我希望能变成这样的人，并且不希望变成我前老板那样的人，period。

工作（创业？）

我是真的不想多提起这段经历，这就是我过去两年的痛苦源泉。

你付出了200%的努力，然后被人当成廉价劳动力和工具随便使唤。你做出了真实的贡献，被人当成你欠他的债。

千言万语汇成一句话，慎重和老板创业。

顺便也回想起了本科愉快的创业经历。让我意识到做一个优秀的tech leader真的太需要在技术力之外的品质了。

一个聪明的混蛋是不可能成功的。虽然这可能在电影里很酷，但真的不要高估自己的智商，也不要把他人对你的善意当成理所当然。

同时让我认识到的是科研这座孤岛。一个优秀的研究者，尤其是当上faculty之后，是很难有渠道摄入批评的。

在系里，大多教授都是和你不同方向的，难以评价你的工作，平时低头不见抬头见，也没有人会做这种吃力不讨好的事。

你的学生可能忌惮于你的权利而不敢批评，或者你干脆就把学生的话当成耳旁风。

在领域里，大家都利益相关，又很少能知道group内部的真实情况，所以聊天大多是互相吹捧。

由此，一个科研工作者很可能陷入自我膨胀中，而不能客观审视自己成果的局限性。

我不知道怎么去改变这些问题，但我觉得人不能，至少不应该去否定时间。

尊重他人经年累月得出的成果，敬畏你不了解的知识，我也只能这样要求自己。

杂，和一些不成熟的经验

今年是在美国的第4年，很多朋友都回国，或准备回国了。

想着自己还要在美国呆上至少5年，心里确实很寂寞。

写到这里就不知道说什么了，希望大家都好吧。我能期盼的就是接下来的若干年里我每年都有时间回一趟国。

这一年里的大多不高兴也大多是由于老板引起的。

我觉得如果有缘人会读到这里，并且恰巧要申请PhD，我想分享一下关于我对于和导师相处的经验：

1. 不要只选择中国导师。是的，也许你会更容易拿到offer，但从我身边的数据看，中国导师就是普遍要更harsh（注意，不是push）。正常的批评应该虚心接受，但无止境但贬低和负面情绪一定是有害的。请千万不要在被导师贬低的时候陷入无限的自我怀疑。更不要产生负罪感。保持自己的节奏，不要无止境的将时间投入到科研中，不要中了一些内心险恶的人的圈套。
2. Don’t be soft。要敢于和老板抗争，要敢于表达自己的不满。要明白，很多老板在PUA你的时候，也是在对你的底线进行一种试探。要在适当的时候清晰的表达你的不满，但不要以愤怒的形式表达出来。一切就事论事，以便于在日后依然保持健康的师生关系。
3. 在进组之前一定要尽可能和组里的学生取得联系，以此侧面了解情况。更重要的是，对于现在的学生说的话要降低期待。只要不是强烈推荐，那几乎可以肯定这个老板有对学生不好的地方。然后再根据自己的情况做取舍。
4. 学生之间背景差异比较大的导师往往会更包容，which means 更nice。
5. 用自己的实力和态度来打动老板，不要靠帮老板搬砖来企图讨好老板。

最后

终于在临近新年的时候收获了一些好消息。

当然还有很长的路要走，把自己的所有愿望放在心里，然后期待2021。

It is commonly known that writing concurrent programs can introduce a kind of bug that normally does not exist in single-threaded programs. Many people refer to them as “Races”, or “Race Conditions”.

You may also hear about people talking about a kind of bug called “Data Races”. For example, the simulation program for the COVID-19 pandemic was questioned about their correctness because of suspected potential “data races”. The discussions have caused quite some drama online.

So are data races the same as race conditions, given that they both share “race” in their names and have similar code patterns? This confuses a lot of people, and even some experienced developers.

Simply put, race conditions and data races are different. One is not even a subset of the other, which is a common misunderstanding.

Race condition, according to its wiki page:

… is the condition of an electronics, software, or other system where the system’s substantive behavior is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when one or more of the possible behaviors is undesirable.

This definition implies race condition is not inherently a bug. Race conditions are inevitable as long as concurrent events exist in the system. Consider the following example:

1
2
3
4
5
6
7

// x is a shared variable
// Thread 1
x = 1
print x

// Thread 2
x = 999

There is clearly a race condition going on, where the result of print x is dependent on the sequence of assignments to x. However, whether it is a bug really depends on what we expect to be printed out, i.e., if the behavior is “desired”.

On the other hand, data race is clearly defined as a bug. Data race occurs when two memory accesses:

• are accessing the same piece of memory;
• are performed concurrently;
• have at least one write;
• are not protected by synchronization or locks.

The definition of “concurrently” is language-dependent. For example, the C++ Standard defines two actions as potentially concurrent if:

• they are performed by different threads, or
• they are unsequenced, and at least one is performed by a signal handler.

The second rule is idiosyncratic to C++. The Java Language Specification gives a different definition.

Now re-examining the example above, we find it is also a “data race”. The concurrent accesses of x are unprotected, so it is a BUG!

A valid fix for this bug is to simply protect each access with a lock:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

// x is a shared variable
// l is a global lock

// Thread 1
lock(l)
x = 1
unlock(l)

lock(l)
print x
unlock(l)


// Thread 2
lock(l)
x = 999
unlock(l)

Now, all data races in this example are gone, but a race condition still exists. The critical sections (the code segment between a pair of lock and unlock) are still unsequenced. The code can still be buggy, say if printing out “999” is undesirable.

Benign or non-benign? Where lies the evilness?

Let’s suppose printing out “999” is indeed a buggy behavior.

Clearly, the buggy printing of “999” is not introduced by the data race but the race condition. To get rid of the bug, we can either a) join thread 2 before the write operation to x in thread 1; or b) put the two operations on x in thread 1 into the same critical section. Both fixes ensure there could be no unexpected operation between x = 1 and print x.

Now One observation you might have is, fixing data races does not necessarily eliminate the race conditions. If so, what is the harm actually brought by a data race? Why we want to get rid of them?

Before we step into the evil world of data races, let’s first take a step back to understand a concept: “benign data race”. There is a widely held and strong perception that data races are acceptable in certain contexts, and thus are commonly used in existing code.

This turns out to be very wrong. Not only are those benign data races technically incorrect and harmful, but new bugs may be introduced by future re-compilation if the code is meant to be portable.

I present some common C/C++ “benign” patterns from a paper by Hans-J. Boehm below, and briefly explain how those common patterns could go wrong:

1. Double-checked locking

1
2
3
4
5
6
7
8
9

if (!init_flag) {
    lock(); 
    if (!init_flag) {
        my_data = ...;
        init_flag = true;
    }
    unlock(); 
} 
tmp = my_data;

This is a very famous pattern (known as DCLP), and is also well-known to be incorrect. The code is potentially buggy if compiler optimization or hardware reordering is applied (which they are, in most real cases).

For example, there’s nothing to prevent the compiler from advancing init_flag = true before the setting of my_data. Assuming there are two threads executing the code above, the second thread may end up reading an uninitialized value (because init_flag is set to true before my_data get initialized).

A general solution for this pattern does not exist until Java 1.5 (the volatile keyword) and C++11 (the memory fence). Without these, we have to protect the whole pattern with a lock, which in turn brings a performance degradation.

For more details about DCLP, you can check the wiki, this page for Java, and Scott and Andrei’s paper about C++.

2. Reading imprecise value is allowed

There are cases where a read from a thread does not care about if another thread has updated the value or not, i.e., either the old value or the new value results in a valid computation. In such cases, are data races benign?

The answer, of course, is NO. Simply because data races will potentially cause the read operation to obtain a meaningless value that doesn’t correspond to either the old value nor the new value.

1
2
3
4
5
6
7
8
9
10
11
12

// Read global
int my_counter = counter;
int (* my_func) (int);
if (my_counter > my_old_counter) {
    ...
    my_func = ...;
}
...
if (my_counter > my_old_counter) {
    ... 
    my_func(...) 
}

Consider the code pattern on the above, where my_counter reads from a global variable counter without protection. The global counter may get updated by some other threads running in parallel. Here the developer may think reading an old value of counter is fine due to the if (my_counter > my_old_counter) check.

However, if the compiler thinks the register of my_counter needs to be spilled between the two if-checks, then it is legit to re-read my_counter‘s value again to avoid spilling, as shown on the below. Now under this optimized code, the two if-checks may not yield the same result, which can lead to unexpected behavior when calling my_func.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// Read global
int my_counter = counter;
int (* my_func) (int);
if (my_counter > my_old_counter) {
    ...
    my_func = ...;
}
...
// ANOTHER read global
int my_counter = counter;
if (my_counter > my_old_counter) {
    ... 
    my_func(...) 
}

If we go a bit further to the hardware level. Assuming we want to maintain an imprecise global 64-bit integer counter. We make the counter++ concurrent and unprotected. If your system only supports 32-bit indivisible writes, then once a thread is updating the counter value from 2^32 - 1 to 2^32, another thread could read a value whose higher bits are already updated but lower bits are not. This will make the counter value totally meaningless.

3. Incorrect ad hoc synchronizations

Ad hoc (or user constructed) synchronizations should not use ordinary operations. Otherwise, we will generally face issues similar to what is mentioned above.

1
2
3
4

// `flag` is intended to be an ad hoc lock
while (!flag) {
    ...
}

1
2
3
4
5

// compiler optimized code
tmp = flag
while (!tmp) {
    ...
}

For example, the busy-waiting loop above can be optimized to something like the second code snippet, due to register spilling. The optimized loop might run indefinitely under concurrent execution, yet it is equivalent to the example on top under sequential code.

Or if we want to use a busy-waiting loop as a memory barrier, like the example below:

1
2

while (!flag) {}; // waiting for `data` to be updated
tmp = data;

Compiler or hardware could potentially reverse the order of the two instructions. Thus, this ad hoc memory barrier is mostly meaningless.

4. Redundant write

Assume we have a function f(x):

1
2
3
4
5
6
7

// count is a shared variable between threads
f(x) {
    ...;
    for (p = x; p != 0; p = p -> next) {
        if (p->data < 0) count++;
    }
}

Notice that count is a shared variable, and it’s only updated when p has negative value. Now if we use this function in the following way:

1
2
3
4
5
6
7

// Thread 1
count = 0;
f(positives);

// Thread 2
f(positives);
count = 0;

The argument positives is a linked list whose nodes only contain positive value. Hence, there’s only one data race between the two count = 0 instructions. You may wonder, how could this possibly go wrong, since all we are doing is writing the same value to count repeatedly.

However, things become different if the compiler inlines f(positives) and promote the variable count to a thread-local register (this is legitimate because count is written unconditionally). Now the actual instructions being executed by each thread become something like below:

1
2
3
4
5
6
7
8
9

// Thread 1
count = 0; // step 2
reg1 = count // step 4
count = reg1 // step 6

// Thread 2
reg2 = count; // step 1
count = reg2; // step 3
count = 0; // step 5

Now under this specific interleaving, we find it is possible for count to remain its original value instead of being zeroed out.

5. Disjoint bit manipulation

This kind of “benign” data races are described as:

… data races between two memory operations where the programmer knows for sure that the two operations use or modify different bits in a shared variable.

Those kinds of manipulation are not generally benign. Updating bits usually contains a) Reading the whole value; b) Updating the relevant bits; and c) Writing the value back. We can clearly see specific interleaving may cause some updates to be invalid.

Even in the case when there is only 1 thread updating the bits, we can come up with the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

x.bits1 = 1;
switch(i) {
    case 1:
        x.bits2 = 1;
        ...;
        break;
    case 2:
        x.bits2 = 1;
        ...;
        break;
    case 3:
        ...;
        break;
    default:
        x.bits2 = 1;
        ...;
        break;
}

Our original code is shown above. x is a global variable whose bits represent different flags, and bits1 and bits2 are two bit fields of x.

Notice that in our original code, case 3 does not update x.bits2 and it happens to be the branch we are using (assume this is a fact compiler does not know, so that the other branches will not be pruned). Now if we have a second thread only reading the value of x.bits2. This seems to be a perfectly safe situation.

However, existing compilers may consider adjacent bit-fields as belonging to the same memory location (here in our case, x.bits1 and x.bits2). The write to x.bits1 will make the compiler think the target memory location is data-race free, thus justify its optimizations on x.bits2. This leads to the optimized code shown on below, where the potential updates on x.bits2 are advanced to shrink the code size (if we have more branches that update x.bits2). Now if we are still using case 3, the read from other thread will conflict with the write on case 3, causing the second thread to read invalid values of x.bits2.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

x.bits1 = 1;
tmp = x.bits2
x.bits2 =1
switch(i) {
    case 1:
        ...;
        break;
    case 2:
        ...;
        break;
    case 3:
        x.bits2 = tmp;
        ...;
        break;
    default:
        ...;
        break;
}

To sum up, the evilness of data races mostly lies in compiler optimizations, hardware reordering, and underlying memory models. Most compiler optimizations assume the program is data-race free, thus compilers may introduce undefined behaviors when data races exist. The worst thing about undefined behavior is it could lead to any actions (such as launching a missile?) instead of just entering an invalid path of your program.

If you are interested in reading more discussion related to data races, there’s another blog explaining this.

How do I prevent my program from launching missiles unexpectedly?

If you make it here, hopefully you are not terrified by what can be caused by data races. Just to be fair, there could be “benign” data races under more concrete contexts, such as knowing the specific architecture, compiler version, optimization passes used and the memory models, and the code is not intended to be portable.

Nevertheless, I strongly believe developers should pay more attention to potential data races just like how they treat compiler warnings (I feel terribly sorry for you if you don’t).

Properly protecting concurrent shared memory accesses not only prevent programs from all those subtle bugs we presented above, but also significantly increase the readability for your code by specifying their concurrent semantics. Besides, there are tools like ThreadSanitizer, Intel Inspector, and Coderrect Racedetector to efficiently reveal potential data races at different programming phases.

Let’s just not making our software program to be a missile launcher, :=).

1
2
3
4
5
6
7
8
9
10
11
12

def yin(cb1):
yield '@'
def yang(cb2):
yield '*'
yield from cb1(cb2)
yield from yang(yang)

def puzzle():
yield from yin(yin)

for x, _ in zip(puzzle(), range(50)):
print(x, end='')

1

@*@**@***@****@*****@******@*******@********@*****

Each time the prgram will output an extra * symbol, therefore our universe is created.

This puzzle is a famous one in functional programming area (probably?). Notably solved by utilizing call/cc (continuation).

As we all know, python’s generator is no more than a simplified version of continuation. And that’s where the solution comes from.

An interesting question here is why there’s an increment on * symbol.

The key to understanding this is noticing the context carried with each generator. To make everything clear, we use notation like yin_yang~yin. The name befoere _ indicates the generator name we are currently calling, the sequence after _ indicates the value of cb1 in the nested context, from the nearest to the furthest.

Here we only need to care about cb1 (callback 1), since it is the place where magic happens. For cb2, since we only call yang(yang), it just duplicates itself every time.

Then we can interpret the process in the following way:

1
2
3
4
5
6
7
8

yin(yin) \\ @
-> yang_yin(yang_yin) \\ *
-> yin(yang_yin) \\ @
-> yang_yang~yin(yang_yang~yin) \\ *
-> yang_yin(yang_yang~yin) \\ *
-> yin(yang_yang~yin) \\ @
-> yang_yang~yang~yin(yang_yang~yang~yin)
...

We can see how the context grows. The key here is, even though we are always passing yang to cb1, the context carried with each yang is different.

If this explaination is not clear enough for you, a comprehensive scheme version explaination is available here.

Haven’t written blog for a looooooong time and so many things happened since my graduation.

Started joining Jeff’s lab recently, life will start to become regular (hopfully?).

This time I’ll try to write my blog in English, just to record my life a little bit.