Reflection / ๋ฐ˜์ถ”

I originally planned the mobile as a piece on metacognition, as it apparently is. Then I was about to accompany it with an essay about how we deliberate and reach a decision, which I studied during my PhD.

But while I was making it, I also realized that I had ideas like this, and others, from when I was little. I felt like I’ve only made it now because now I have more experience executing such ideas.

In that sense, the mobile could be also about growth: a larger self looking back on the little selves. Indeed, metacognition, or evaluation of one’s own thinking, can be one way for learning and growth for humans and machines. For example, if you were sure you would be invited to a friend’s birthday and if you were not, you would wonder about the reason and might learn more about what happened to the friend or to your friendship. That’s different from when you were unsure about getting the invitation to begin with, in which case being not invited wouldn’t mean much.ย Therefore, the sense of being sure, or confidence, is a form of metacognition that can help learning. Machines use confidence to learn as well: agreement between the graded sense of confidence and the all-or-none outcomes can beย mathematically expressed as “cross entropy“, which is a standard measureย used in training machines.

Back to the mobile, I debated whether to use wires or paper, but chose paper because each figure is planar. I glued several sheets together to reinforce them. If someone wants it in a more permanent form, I’d like to try 3D printing it.

์›๋ž˜๋Š” ๋ฉ”ํƒ€-์ธ์ง€์— ๊ด€ํ•œ ๋ชจ๋นŒ๋กœ ๊ณ„ํšํ–ˆ๋‹ค. ์ง€๊ธˆ๋„ ๊ทธ๋ ‡๊ฒŒ ๋ณผ ์ˆ˜ ์žˆ๋‹ค. ์„ค๋ช…์œผ๋กœ๋Š” ์šฐ๋ฆฌ๊ฐ€ ์–ด๋–ป๊ฒŒ ๊ณ ๋ฏผํ•˜๊ณ  ๊ฒฐ๋ก ์— ๋„๋‹ฌํ•˜๋Š”์ง€์— ๋Œ€ํ•ด, ๋‚ด๊ฐ€ ๋ฐ•์‚ฌ๊ณผ์ • ๋•Œ ์—ฐ๊ตฌํ–ˆ๋˜ ๋‚ด์šฉ์„ ๊ณ๋“ค์—ฌ ์“ธ ์ƒ๊ฐ์ด์—ˆ๋‹ค.

ํ•˜์ง€๋งŒ ๋ชจ๋นŒ์„ ๋งŒ๋“œ๋Š” ๋™์•ˆ, ์–ด๋ฆด ๋•Œ๋„ ์ด๋Ÿฐ ์•„์ด๋””์–ด๋ฅผ ํฌํ•จํ•ด ์—ฌ๋Ÿฌ ์•„์ด๋””์–ด๋“ค์ด ์žˆ์—ˆ๋˜ ๊ฒƒ์ด ๊ธฐ์–ต๋‚ฌ๋‹ค. ์ด์ œ์•ผ ์ด ๋ชจ๋นŒ์„ ๋งŒ๋“ค๊ฒŒ ๋œ ๊ฒƒ์€, ์ด์ œ์„œ์•ผ ๊ทธ ์•„์ด๋””์–ด๋ฅผ ์‹คํ˜„ํ•  ์ˆ˜ ์žˆ๊ฒŒ ๋˜์–ด์„œ๋ผ๋Š” ์ƒ๊ฐ์ด ๋“ค์—ˆ๋‹ค.

๊ทธ๋Ÿฐ ์˜๋ฏธ์—์„œ, ์ด ๋ชจ๋นŒ์€ ์„ฑ์žฅ์— ๊ด€ํ•œ ์ž‘ํ’ˆ์ผ ์ˆ˜๋„ ์žˆ๊ฒ ๋‹ค: ํฐ ์ž์‹ ์ด ์ž‘์€ ์ž์‹ ๋“ค์„ ๋ฐ˜์ถ”ํ•˜๋Š” ๊ฒƒ์ด๋‹ค. ๋ฉ”ํƒ€-์ธ์ง€, ํ˜น์€ ์Šค์Šค๋กœ์˜ ์ƒ๊ฐ์— ๋Œ€ํ•œ ํŒ๋‹จ์€ ์‹ค์ œ๋กœ ์‚ฌ๋žŒ์ด๋‚˜ ๊ธฐ๊ณ„๊ฐ€ ๋ฐฐ์šฐ๊ณ  ์„ฑ์žฅํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋œ๋‹ค. ์˜ˆ์ปจ๋Œ€ ์–ด๋–ค ์นœ๊ตฌ์˜ ์ƒ์ผํŒŒํ‹ฐ์— ์ดˆ๋Œ€๋  ๊ฑฐ๋ผ๊ณ  ํ™•์‹ ํ–ˆ๋Š”๋ฐ ์ดˆ๋Œ€๋ฐ›์ง€ ๋ชปํ–ˆ๋‹ค๋ฉด, ๊ทธ ์นœ๊ตฌ๋‚˜ ์นœ๊ตฌ์™€์˜ ์šฐ์ •์— ๋Œ€ํ•ด ๋‹ค์‹œ ์ƒ๊ฐํ•ด๋ณด๊ณ  ๋ญ”๊ฐ€๋ฅผ ๋” ๋ฐฐ์šธ ์ˆ˜ ์žˆ์„ ๊ฒƒ์ด๋‹ค. ํ•˜์ง€๋งŒ ์ดˆ๋Œ€๋ฐ›์„์ง€ ์—ฌ๋ถ€๊ฐ€ ์• ์ดˆ์— ๋ถˆํ™•์‹คํ–ˆ๋‹ค๋ฉด, ์ดˆ๋Œ€๋ฅผ ๋ชป ๋ฐ›์•„๋„ ๋ณ„ ๋œป์ด ์—†์—ˆ๋‹ค๊ณ  ์—ฌ๊ธธ ๊ฒƒ์ด๋‹ค. ์ด๋ ‡๊ฒŒ ์ž์‹ ๊ฐ์€ ๋ฐฐ์›€์„ ๋„์šธ ์ˆ˜ ์žˆ๋Š” ๋ฉ”ํƒ€์ธ์ง€์˜ ํ•œ ํ˜•ํƒœ์ด๋‹ค. ์ž์‹ ๊ฐ์€ ๊ธฐ๊ณ„๋“ค์˜ ํ›ˆ๋ จ์—๋„ ์“ฐ์ธ๋‹ค:ย  ์ž์‹ ๊ฐ๊ณผ ์‹ค์ œ ๊ฒฐ๊ณผ ์‚ฌ์ด์˜ ์ผ์น˜๋„๋Š” ์ˆ˜ํ•™์ ์œผ๋กœ “ํฌ๋กœ์Šค ์—”ํŠธ๋กœํ”ผ“๋กœ ํ‘œํ˜„๋  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์ด๋Š” ๊ธฐ๊ณ„๋“ค์„ ํ›ˆ๋ จํ•  ๋•Œ ์ผ์ƒ์ ์œผ๋กœ ์“ฐ์ด๋Š” ์ฒ™๋„์ด๋‹ค.

๋ชจ๋นŒ๋กœ ๋Œ์•„์™€์„œ, ์ฒ ์‚ฌ๋ฅผ ์“ธ์ง€ ์ข…์ด๋ฅผ ์“ธ์ง€ ๊ณ ๋ฏผํ•˜๋‹ค๊ฐ€, ๊ฐ ์ธ๋ฌผ์˜ ๋””์ž์ธ์ด ํ‰๋ฉด์ ์ด๊ธฐ ๋•Œ๋ฌธ์— ์ข…์ด๋ฅผ ์“ฐ๊ธฐ๋กœ ํ–ˆ๋‹ค. ์ข…์ด ๋ช‡ ์žฅ์„ ํ•จ๊ป˜ ๋ถ™์—ฌ์„œ ๋‹จ๋‹จํ•˜๊ฒŒ ์„ธ์šธ ์ˆ˜ ์žˆ๋„๋ก ๋งŒ๋“ค์—ˆ๋‹ค. ๋” ํŠผํŠผํ•œ ๋ฒ„์ „์„ ์›ํ•˜๋Š” ์‚ฌ๋žŒ์ด ์žˆ๋‹ค๋ฉด 3D ํ”„๋ฆฐํŒ…์œผ๋กœ ๋งŒ๋“ค์–ด ๋ณด๊ณ  ์‹ถ๋‹ค.

YK 2019.

Fixinโ€™ ResNet / ๋ ˆ์Šค๋„ท ๊ณ ์น˜๊ธฐ

Diggin’ into neural nets.

A ResNet is a relatively new type of deep artificial neural networks that can recognize objects – the kind that finds objects in your images on Google Photos.

This illustration mixes images of a stack of sandwiches with a ResNet. The protagonist is adding a “skip connection” (blue wire) between neural layers (slices of bread). Such skip connections, inspired by biology, are the characteristic of ResNets. The skip connections allowed ResNets to be deeper than its predecessors (have a tall stack), and helped them to recognize more complex images.

Why is it good for a network to be deep? And why do skip connections help? I find it a stretch to explain them with the analogy of a sandwich, so I defer the answers to later posts. Let’s say for now that the ResNet has skip connections repeated every 2-3 layers, so it’s easy to make it deeper by stacking the same structure multiple times. That makes it look like the stacked sandwiches (see Figure 3 of the original paper).

Well, this was my first attempt at a neuroscience/AI-inspired illustration. The analogy leaves a lot to be desired, but hopefully it will get better as I try more. At least this fulfills my new year’s resolutionโ€”to post about neuroscienceโ€”on the new year’s day! ๐Ÿ™‚

์‹ ๊ฒฝ๋ง ์† ํŒŒ๊ณ ๋“ค๊ธฐ.

๋ ˆ์Šค๋„ท์€ ๋น„๊ต์  ์ƒˆ๋กœ์šด ์‹ฌ์ธต ์‹ ๊ฒฝ๋ง์œผ๋กœ, ๊ตฌ๊ธ€ ํฌํ† ์—์„œ์ฒ˜๋Ÿผ ๋ฌผ๊ฑด๋“ค์„ ์•Œ์•„๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ทธ๋ฆผ์—์„œ๋Š” ๋†’์ด ์Œ“์€ ์ƒŒ๋“œ์œ„์น˜์™€ ๋ ˆ์Šค๋„ท์˜ ์ด๋ฏธ์ง€๋ฅผ ํ•ฉํ–ˆ์Šต๋‹ˆ๋‹ค. ์ฃผ์ธ๊ณต์€ “๊ฑด๋„ˆ๋›ฐ๋Š” ์—ฐ๊ฒฐ”(ํŒŒ๋ž€ ์ „์„ )์„ ์‹ ๊ฒฝ๋ง ์ธต (๋นต) ์‚ฌ์ด์— ์ถ”๊ฐ€ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ๊ฑด๋„ˆ๋›ฐ๋Š” ์—ฐ๊ฒฐ์€ ์‹ค์ œ ๋‡Œ์—์„œ ์˜๊ฐ์„ ์–ป์€ ๊ตฌ์กฐ๋กœ, ๋ ˆ์Šค๋„ท์˜ ๊นŠ์ด๋ฅผ ๋”ํ•˜๋Š” ๋ฐ ๋„์›€์„ ์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค. (์ธต์„ ๋†’์ด ์Œ“์„ ์ˆ˜ ์žˆ๊ฒŒ ํ•ด์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค.)

์™œ ์‹ ๊ฒฝ๋ง์ด ๊นŠ์œผ๋ฉด ์ข‹์€์ง€, ์™œ ๊ฑด๋„ˆ๋›ฐ๋Š” ์—ฐ๊ฒฐ์ด ๋„์›€์ด ๋˜๋Š”์ง€, ์ƒŒ๋“œ์œ„์น˜์˜ ๋น„์œ ๋กœ ์„ค๋ช…ํ•˜๊ธฐ์—๋Š” ํ•œ๊ณ„๊ฐ€ ์žˆ์–ด์„œ, ๋Œ€๋‹ต์€ ์ดํ›„ ๊ผญ์ง€๋“ค๋กœ ๋ฏธ๋ฃจ๋„๋ก ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ๋Š” ๋ ˆ์Šค๋„ท์ด ๊ฑด๋„ˆ๋›ฐ๋Š” ์—ฐ๊ฒฐ์„ 2-3์ธต๋งˆ๋‹ค ๋ฐ˜๋ณตํ•ด์„œ ๊ฐ€์ง€๊ณ  ์žˆ์–ด์„œ, ์ƒŒ๋“œ์œ„์น˜๋ฅผ ์Œ“๋“ฏ์ด ๊ฐ™์€ ๊ตฌ์กฐ๋ฅผ ๋ฐ˜๋ณต๋งŒ ํ•˜๋ฉด ์‰ฝ๊ฒŒ ๊นŠ์–ด์ง€๋„๋ก ํ•  ์ˆ˜ ์žˆ๋‹ค๋Š” ์ ๋งŒ ์–ธ๊ธ‰ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. (์›๋ž˜ ๋…ผ๋ฌธ์˜ ๊ทธ๋ฆผ 3์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.)

๋‡Œ๊ณผํ•™/AI์— ๊ด€ํ•œ ์ฒซ ๊ผญ์ง€์˜€์Šต๋‹ˆ๋‹ค. ๋น„์œ ๊ฐ€ ์•„์ง ์–ด์ƒ‰ํ•˜์ง€๋งŒ, ๋” ์‹œ๋„ํ•˜๋ฉด์„œ ๊ฐœ์„ ํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ ์–ด๋„ ๋‡Œ๊ณผํ•™์— ๊ด€ํ•ด ์จ ๋ณด๊ฒ ๋‹ค๋Š” ์ œ ์ƒˆํ•ด ๋‹ค์ง์„ ์ƒˆํ•ด ์ฒซ๋‚ ์— ์‹ค์ฒœํ•œ๋‹ค๋Š” ์˜๋ฏธ๋Š” ์žˆ๊ฒ ๋„ค์š”. ๐Ÿ™‚