So a couple of weeks ago the internet, well the parts that I like to visit blew up with a very special fly, well its brain to be more precise. And I just had to get onto the bandwagon as well. And this is what this blog will be all about. Me playing with a fly’s brain. Now before I get into the details let me take a step back and explain exactly what happened and why this is so amazing. The fly brain used in this project comes from the FlyEM team at HHMI’s Janelia Research Campus, who worked with collaborators including the University of Cambridge and Google. They took the brain and nerve cord of a single male fruit fly, cut it into extremely thin layers and imaged every layer with an electron microscope. Software and a large team of human proofreaders then traced each neuron through those images and recorded every synapse where one neuron connects to another. This gave them a full wiring diagram, known as a connectome, of well over a hundred thousand neurons and many millions of connections. They released it publicly online, so anyone can download the complete nervous system of a real insect, stimulate it in different ways and observe how the activity spreads through the network.
Before the fly there was the worm. OpenWorm is an open-source project started in 2011 with the goal of building the first virtual organism in a computer. Its subject is C. elegans, a tiny nematode worm with just 302 neurons, which makes it the first animal whose entire nervous system was mapped from electron microscope images. A worldwide community of scientists, engineers and volunteers has been turning that wiring diagram into a working simulation that includes the worm’s neurons, muscles and body. All of the project’s code, data and models are released, so anyone can download the worm, run it and experiment with it.
Below is an animation of the worm’s neural network controlling a virtual body as it crawls around inside a simulated lab environment. The body and its surroundings exist only in the computer, but the wiring that drives them comes from the nervous system of a real worm.
But we are not here to talk about worms. We are here to talk about flies. Let me introduce you to Splat as I will call him from now on.
In this blog I’ll take you inside a fruit fly’s head and explore its brain in fine detail, from the individual neurons down to the synapses that connect them.
So, a little bit more about the fly’s actual brain that we will be looking at today. This fly brain connectome contains 166,700 neurons linked by over 25.5 million directed connections, built from 124 million synaptic contacts. Of these, 74,070 neurons have their cell body (soma) on the left side of the brain and 74,235 on the right, with a further 392 sitting on the midline. Vision starts with 3,377 photoreceptors (R1-R6), which feed into 8,884 lamina relay neurons that pass visual signals deeper into the brain. Connections carry a predicted sign based on neurotransmitter: 15.18 million are excitatory, 9.81 million inhibitory, and 588,262 of unclear polarity. By comparison, a human brain has roughly 86 billion neurons and 100 trillion synapses, a tiny fly brain packed into a few hundred thousand cells against a human brain’s tens of billions. And as you can see from the photo below the brain that we will be working with is rather small, insert joke about my sister here.
So you may ask, how was this exactly accomplished? How was a fly’s brain digitized? First the fly’s brain and nerve cord were carefully dissected out and preserved with chemicals that fix every cell in place. The tissue was then stained with heavy metals, which make cell membranes show up clearly under an electron microscope, and set in a hard block of resin. A heated diamond knife cut this block into slabs a few hundredths of a millimeter thick. Each slab went into a special microscope that uses a focused beam of ions to shave away a layer of the surface only a few nanometers thick. An electron beam then scans the newly exposed surface and takes an image before the next layer is shaved off. This repeats over and over, layer by layer, until the whole slab has been imaged. Across all the slabs this produced an enormous number of images, which together capture the fly’s nervous system at a resolution fine enough to see individual synapses.
Below is one of these images. You can see the detail of the resolution that was obtained via the electron microscope.
although if you would view this image full-size you would actually be able to see each neuron individually. Below is a short animation of the full resolution of the detailed that was captured with the electron microscope. If you look at it long enough, you will be able to see the path that each one of the neurons followed. Keep in mind this resolution is much much much much smaller than a human hair for comparison.
And as always, if you do something, it’s worth overdoing it. So I took it upon myself and Claude to take the sliced images and generate a 3D model from it that you can see below.This is a similar method being used to generate 3D models from CT scans that you can get done in a hospital. You can use the mouse to rotate the 3D model and adjust the sliders for density and even slice the model up in further sections to see every detail. Please note that this probably only works on a computer running a browser like Chrome and not on a cell phone.
Below is a short video showing all the slices that was used to generate the 3D model of the fly’s brain. Just looking at this still astounds me since the entire fly is less than a couple of millimeters big and the brain is probably only one millimeter big in total.
Again, keeping with everything’s worth overdoing, I also try to apply the same technique to all the slices of the fly’s brain.You will see my attempt was less than successful, which give you an indication of how much experience and knowledge you need to have in order to make a project like this successful. Nonetheless, you are still able to make out some details in my version, however, I think it looks more like abstract art than anything that can be used in a scientific environment.
So you might wonder how did the professionals actually map out all the neural pathways of the brain if all you have to work off is a bunch of images. Well, very high level, it is basically like connecting the dots and following paths. You start off with a black dot and you follow the black dot through every slice until you have reached the end of that specific path. Now rinse and repeat a couple of thousand times and you got yourself a fully digitized brain.
Once every neuron had been traced, the next job was to divide the brain into regions. Neuroscientists split the fly’s nervous system into areas called neuropils. These are dense tangles of nerve fibers and synapses where particular groups of neurons meet and connect, and each one is linked to particular jobs such as processing smells, handling vision or controlling movement. Labeling every neuron and synapse with the region it belongs to turns a huge wiring diagram into something closer to a map, which makes it much easier to see how the different parts of the brain work together.
The animation below builds up the brain in stages, showing 512, 2048, 8192 and finally 30000 neurons. The full dataset contains about 166700 neurons connected by over 124 million synapses, but drawing all of them at once just produces a solid blob where you can’t see anything inside, so I stopped at 30000. Even that is only a fraction of the whole nervous system, and every one of those neurons was traced through the electron microscope images, along with each of its branches and every synapse it makes. Mapping something this detailed in a structure smaller than a poppy seed would have been impossible without modern computing power and AI software that can follow neurons through all those images, with human proofreaders checking and correcting the results.
Just for interest and because it looks really cool here’s a rendering I found of all the neurons represented as a solid 3D model. As you can see it’s so dense you can’t even see anything inside.
And because I know you are dying to know, here’s a short high level description of some of the main areas of the brain. The nervous system is split into two main parts. At the front is the brain itself, which sits in the fly’s head. Much of it is taken up by two large optic lobes, one behind each eye, because a fly relies heavily on vision and needs a lot of processing power to handle fast-moving images while it flies. Between them is the central brain, which handles smell, learning, memory and decision making. A thin bundle of nerve fibers then runs through the fly’s narrow neck and connects the brain to the ventral nerve cord, the smaller part at the back. The nerve cord sits in the fly’s thorax, close to the legs and wings, and works roughly like a spinal cord. It carries commands down from the brain and controls walking, flying and other movements locally, close to the muscles that do the work.
And now for the really cool part. Below is a video of the fly’s brain starting off at a low number of neurons and then slowly increased. This is the first view you’ll see of the live simulation of the brain. So you’ll also see the neurons being fired in real time as the model rotates.
Please just note, it seems that YouTube always defaults to the lowest resolution for videos. And since these videos does contain quite a lot of detail, I would encourage you to manually switch to the highest resolution possible.
As mentioned above, this is a live simulation of the brain and below you can see only the active neurons and connections as they fire in real time. At the moment this is mostly random firings as the fly at this point does not really have much input and the electrical signals you see is basic noise at this point. Imagine one of those old CRT TVs where you see the static noise on the screen. That’s what you see in the fly’s brain at the moment.
And now for the real fun stuff. What I did with the brain once I downloaded it to my local computer. I took the fly’s brain and connected it to a virtual fly’s body in a virtual environment. I then connected the virtual flies body parts to the corresponding sections in the fly’s brain. For example, if a specific section controlling the fly’s left leg would trigger the virtual body of the fly would then move as a result. Same goes for the wings and the rest of the body including the eyes. The eyes was a bit more tricky. I had to do some custom work there. I created two virtual cameras which in turn is a bunch of tiny little cameras that is then connected to each one of the neural pathways linked to that specific tiny eye resulting in the original type of vision a fly would be used to.
I also mapped out the neural activity of the fly over time. I didn’t map out each section of the brain, just the neural activity over all being experienced by the fly at that specific point. as I did not want to do a doctorate In this field, I was happy to only map the overall brain activity and not each section although that would be really interesting and I might do it in a future further version 2 of this project.
Below is a short video of the fly simulation running in real time.It’s really interesting to see how the neural activity decreases when there’s less for the fly to actually see and interact with. I have also limited the fly’s ability to, you know, fly. So it’s more of a walk than a fly…Ba dum tissss…
Okay, but serious. At the moment the fly is not as we would call it fully conscious. This is more unconsciously reacting to its environment. Similar that you would reach out and catch a ball that you were unaware of until you caught it in your hand. to actually make the fly fully “conscious” I would need to spend many many more weeks connecting each part of the fly’s brain to the correct area. For example, gravity, sense, smell, etc.
Here’s an interesting view from the fly’s perspective. This is not 100% as the fly would see it as the eyes is not overlapping. However, I didn’t want to spend too much time getting the camera angles perfectly aligned since there were quite a few of them as you can see. Initially this was just for troubleshooting to make sure the fly actually sees correctly. However, afterwards it’s actually very interesting to have a fly’s perspective of the world.
Here I only showed the fly and the electrical signals of the brain. So you can actually see it’s spiking if it’s looking at colorful objects or if it’s only looking at a wall you’ll see the neural activity go down.
Now as I mentioned, I was not the only one who downloaded the fly’s brain and other people can get really creative with what they can do with Splat. I’m not going into the right and wrong of being able to have a full fly brain running inside your computer. Because in theory if you can simulate every neuron and synapse of the fly’s brain, is that fly actually alive or not? Is this the first step to uploading a human to the cloud? And would that human be alive? Well, that would be up to you and I guess the people who will be uploaded to the cloud in the next 10, 15, 20 years. Personally, I think that would be really cool to be able to live forever and think at the speed of light. But that’s just me.
Okay, enough with the philosophy. Below are some examples of what other people put Splat through. This video here shows Brian connecting the fly to a drone and having the fly control the drone. So, for the fly’s point of view, it is actually flying around in a warehouse. Imagine sitting on a lovely piece of rotting apple and suddenly waking in a warehouse, not knowing where you are and flying around trying to figure out how this happened.
Another use case is somebody put the fly’s brain in another simulated environment. In this case it is a world model that is rendered in real time by an AI. That is why it looks so realistic. When the fly wants to fly left the AI simply generates the video of the virtual camera also turning to the left and vice versa. I personally like the one on the left where the fly can fly around in a fruit market. I presume that might be heaven for the fly.
Below somebody put the fly’s brain in front of a computer to play the stock market. I’m not going to comment on how well the flight actually did.
Here somebody connected the fly to a virtual robot and the fly could walk around that then translated into the virtual robot walking around.
Obviously, lots of people put the fly inside game environments. Below you can see Grand Theft Auto 1 being played by the fly. However, other people put the fly in Doom, Minecraft and many other games as well.
And yes, somebody put the fly inside a car to see if it was able to drive a car. From the video that I watched, the fly seems to be very aggressive and was laying on the horn constantly because the other cars was driving too slow.
If you would like to learn more about the scientists and organizations behind this extraordinary work, start with Janelia Research Campus in Ashburn, Virginia, USA (https://www.janelia.org/). Opened in 2006, Janelia is part of the Howard Hughes Medical Institute, a nonprofit biomedical research organization founded in 1953 that supports fundamental scientific discovery (https://www.hhmi.org/). Janelia brings together researchers working in neuroscience, microscopy and computational analysis to investigate how brains are organized and how networks of neurons produce behavior. Its FlyEM project team focuses specifically on reconstructing the wiring of the fruit fly nervous system, using electron microscopy to identify individual neurons and the synapses connecting them (https://www.janelia.org/project-team/flyem). The complete male fruit fly central nervous system map was produced through a collaboration between FlyEM, the University of Cambridge, the MRC Laboratory of Molecular Biology and Google Research. The official project website provides access to the research, interactive exploration tools and comparisons between male and female neural circuitry (https://male-cns.janelia.org/). For readers who want to explore the underlying data themselves, the downloads page includes links to microscopy images, neuron reconstructions, synapse information and connectivity datasets (https://male-cns.janelia.org/download/). Those interested in the software behind the work can also browse the team’s public GitHub repositories, which contain tools for processing microscopy data, reconstructing neurons and analyzing their connections (https://github.com/janelia-flyem).
And last but not least below is a link to my GitHub for the fruit fly environment I created. You would still need to download the actual fly brain yourself though.
