neur2201.fw.png


1. Introduction


This video clip from a Fox News report shows the interview of Michio Kaku (American theoretical physicist and popularizer of science) about the latest advancements in neural decoding, or as he preferred to coin it, mind reading! In this interview, Professor Kaku outlines the use of functional magnetic resonance imaging (fMRI) in neural decoding and focuses particularly on future implications of such findings, especially the construction of a “dictionary of thought”, or decoding program that would quickly and accurately read thoughts.

Although briefly, Professor Kaku did brush over limitations of current technologies, however we felt that he didn't show this accurately and as our research into neural decoding suggests, there is far more ground to be covered in order to make “mind reading” a real possibility. Professor Kaku did mention the possibility of making interrogation far easier and less unsavory as a possible application of mind reading, but we felt that his scope was rather limited, and have proposed our own future implications and associated ethical issues.

This media piece was of particular interest to us because it brings up a huge topic in neuroscience with significant and exciting applications. The most practical of which, included the study of fMRI in relation to responses to orientative stimuli towards building a "dictionary of thought" and ultimately opening up a variety of applications in communication. Similarly, the applications of brain machine interfaces by using electroencephalography have already proven fruitful in developing prosthetic limbs for amputees and paraplegics to aid them in simple everyday tasks.


Media piece























2. Neuroscientific Context


Background
Neural decoding refers to the field in neuroscience that is concerned with reconstruction of sensory stimuli from information that has already been encoded by the brain. This refers to the ability to predict what sensory stimuli the subject is receiving based on recorded electrical activity of action potentials in neurons in the brain. Progress in neural decoding has largely looked at deciphering visual input from methods like functional magnetic resonance imaging (fMRI) in conjunction with the use of 3D voxel models to represent neural activity in the primary visual cortex.

2.1 Measuring neural activity and neuroimaging

The most common methods of neuroimaging and measuring neural activity in neural decoding are electroencephalographies (EEG) and fMRI. However, fMRI has a large advantage over EEG in terms of visual representation of active regions of the brain when subjected to perceive visual stimuli. EEG also lacks accuracy in terms of quantitative statistical pattern recognition for more local neural activities, whilst voxel representation for neural ensembles in fMRI have been as accurate as 80% which leaves 20% of inaccuracy being attributed to noise in neural activity and temporal resolution (Haynes & Rees, 2006).

Electroencephalography

Electroencephalography utilises multiple electrodes attached on a subject's scalp to measure the summated post synaptic potentials within neurons and along the scalp. The use of EEG to detect spiking activity known as local field potentials is crucial to distinguishing between different stimuli. Local field potentials (LFP) or "slow waves" have been studied in relation to neural decoding as they have shown to be sensitive to integrative processes and carry information about cortical networks (Belitski et. al, 2010). This suggests that visual information processed by orientation cortical columns in the striate cortex can be detected by electrodes and can be interpreted into which LFPs are representative of certain stimuli. Despite interest in looking at LFP with electroencephalography, very few studies have been conducted to investigate stimulus information in sensory cortical LFP due to a lack of accuracy in using scalp recorded EEG.

LFP.png
Figure 1 - (a) Stimulus information in the form of LFP frequency bands. (b) Frequency of LFP plotted as a function against stimulus duration.

Intracranial EEG (iEEG) or Electrocorticography (ECoG)

A more accurate form of EEG is intracranial EEG or electrocorticography where micro-electrodes are inserted subdurally onto the surface of the brain. ECoGs sample neuronal activity from a smaller cortical area than other conventional EEG methods and contain higher frequency gamma rhythms. As a result, ECoG's are more accurate, have greater temporal resolution than traditional scalp electrode EEGs, but are ethically a more invasive procedure (Lebedev & Nicolelis, 2006).

Although firing modulations in neurons are highly variable in a single neuron and firing patterns are distinct in the execution in particular movements, averaging across many trials (to reduce variability) and studying statistical frequencies has acquired consistent firing patterns for those movements. Not only can ECoGs detect the same LFP's as scalp electrode EEGs but they can also be practically used in brain machine interfaces (BMI) in conjunction with prosthetic limbs. Brain machine interfaces connect tissues of the brain to machines and aim to assist or repair impaired motor and sensory function in the body such as those of an amputee or paraplegic (Andersen, Musallam & Pesaran, 2004).

Functional Magnetic Resonance Imaging

A visual representation of already encoded neural information of sensory stimuli can be achieved through a method used frequently in neuroimaging - functional magnetic resonance imaging. fMRI gives the experimenter the ability to construct a subject’s visual field map or retinotopy (Wandell, Dumoulin & Brewer, 2007). A retinotopy is more specifically a form of topography or mapping that represents visual information in neurons at the visual field. fMRI is used for the functional purpose of blood-oxygen level dependent (BOLD) contrast to measure the effect of neural activity on blood flow and energy use by cells in the brain. BOLD contrast capitalises on the different magnetic fields expressed in deoxyhemoglobin and oxyhemoglobin during periods of neural activity known as hemo-dynamic responses (HDR) in the brain. These hemodynamic responses are correlated to local field potentials from EEG readings which refer to the electrical potentials in the extracellular matrix around neurons. This suggests that neural activity in one area of the primary visual cortex is likely to be linked to the visual input from specific stimuli as opposed to random spiking (Logothetis, 2003).

Figure 2.1 - A retinotopy constructed by fMRI representing BOLD contrast of haemodynamic responses in the primary visual cortex.
Figure 2.1 - A retinotopy constructed by fMRI representing BOLD contrast of haemodynamic responses in the primary visual cortex.

From an image produced by fMRI, blood flow or neural activity is visualized through a contrasting spatial unit called a voxel. Voxels are the smallest units of MRI construction and corresponds to a single pixel in an MRI image. The basis of neural decoding in relation to fMRI and 3D voxel models is that high contrast stimuli (high spatial resolution) of varying orientation should be able to stimulate corresponding and responsive voxels in the primary visual cortex (Sun et. al, 2013). One of the many limitations of interpreting voxels is that they are representative of not only high contrasting stimuli, but also visual “noise” and extraneous stimuli such as subtle head motions and input outside the focus of the fovea i.e. in the periphery of the visual field.In the presence of different stimuli (Kamitani and Tong, 2005). Voxels are used as independent pieces of data and are statistically summated and averaged to be representative of the most common neural spikes in response to orientative stimuli.
Figure 2.2 - A 3x3 box of 3mm by 3mm voxels representative of activity in neuronal ensembles
Figure 2.2 - A 3x3 box of 3mm by 3mm voxels representative of activity in neuronal ensembles

2.2 Methods of neural decoding


Visual decoding


Background


Visual decoding refers to the retrieval of visual information from previously perceived information. According to Kamitani & Tong (2005), human perception is assumed to be a product of neural encoding of fundamental features of external world. These features and perceived data are being processed and encoded into information for the cognitive brains of humans. The fundamental features include orientation, colour and motion. In order to decode visual information from the visual system, different techniques have been employed. In early studies, fMRI pattern analysis was used in decoding visual information.

Figure 3 - Pathways of visual processing
Figure 3 - Pathways of visual processing

Orientation

Orientation is one of the most fundamental and earliest visual features to be processed. Orientation cortical columns in the primary visual cortex or the striate cortex have been known to encode specific orientation of stimuli – especially their angle of orientation. Each neuron is responds to a specific type of orientation direction (Haynes & Rees, 2006). Neurons in these columns tend to fire action potentials or become excited when the corresponding orientation of the stimuli is transduced at the retinal level.

Motion

Another fundamental feature of visual system is the motion of perceived objects. The extrastriate visual area, medial temporal (MT/V5) and medial superior temporal cortex (MST) are involved in motion processing (Kourtzi & Kanwishe, 2000). Previous studies showed motion detection triggers high levels of responses. Specific neurons have been found to respond to certain directions of stimuli, known as selective columns (Zimmermann, Goebel, De Martino, van de Moortele, Feinberg, Adriany, Yacoub, E, 2011).

Colour

Colour is also represented in cortical neurons, in which studies has been done on visual areas V1, V2, V3, V4 and VO1. According to Brouwer and Heeger (2009), each colour stimulus is associated with patterns of activities that is assumed to be a representation in the cortical neuron. To achieve visual colour decoding, multivariate techniques such as conventional pattern classification, a forward model of idealized color tuning, and principal component analysis (PCA) are adapted. V1 is found to provide most accurate prediction.





3. Applications of Neural decoding


Brain-Computer Interfaces (BCI)

Neural decoding has a vast array of uses being able to decode human thoughts gives almost limitless potential that can be abused or harnessed to give back function to those who have lost function of limbs. Brain computer interfaces (BCI) provide the means to translate electro-chemical nerve signals into meaningful electrical signals that can be utilized to perform an action that will significantly improve the quality of life for these individuals.

Figure 4 - A prosthetic arm machine interface with motor neurons from the primary motor cortex.
Figure 4 - A prosthetic arm machine interface with motor neurons from the primary motor cortex.

Brain computer interfaces can be divided roughly into two categories; Non- invasive and invasive (Lebedev and Nicolelis, 2006). Non-invasive means of interfacing with the brain typically revolve around the use of EEG, which provides decent temporal resolution but poor temporal resolution. fMRIs are not generally used in more direct forms of BCI as they require the subject to be static and also inside the machine limiting its usefulness for purposes such as neural prosthetics. Invasive methods typically revolve around the use of iEEG or ECoG and other forms of chronic electrode implants that while providing far superior temporal and spatial resolution are left within the patient for the long term and require surgery for implantation.

In actuality BCI's are not actually interfacing between a single nerve cell and an electrode, but rather an electrode that records cortical activity in the form of oscillations in frequency and encodes them into meaningful control signals that can be arbitrary programmed into a device like a prosthetic arm (Taylor and Schwartz, 2001; Lebedev & Nicolelis, 2006). Furthermore, the use of devices like prosthetic arms are not as simple as converting neural activity into their corresponding actions. The use of BCI requires learning and skill acquired from conditioned alpha waves and mu rhythms. These are synchronised patterns of activity that control voluntary movement. Evidence supporting this comes from studies in monkeys that showed firing rates of cortical neurons could be conditioned (the modification of behaviour by its consequences) by non-muscular communication or control (Wolpaw, Birbaumer, Farland, Pfurtscheller & Vaughan, 2002).

Untitled.png
Figure 4.1 - BCI conditioning of prosthetic devices in a monkey

4. Ethical Issues


The functional magnetic resonance imaging (fMRI) has proved itself to be a very useful tool in allowing neuroscientists the power to “read minds” (Ladd & Berry, 2013). However with great power comes great responsibility. Usage and further development of fMRI has brought up several ethical issues, concerning the participants and the researchers. Furthermore the advances seen in neuroscience are challenging long held views of the mind, ones behaviour and their relationship with society (Illes & Bird, 2006).

With present technology only ever increasing; the amount of detail obtained from an fMRI will also only
increase (Tong & Pratte, 2012). This brings upon the concerns of the participants privacy and the confidentiality of the results from an fMRI. Whereas in experiments where participants have given consent and are asked questions; fMRI has the potential to invade the mental privacy of participants, without them even knowing (Ladd & Berry, 2013). A simple medical check-up for tumours could lead up to the discovery that the patient has the “thoughts” and “traits” of a serial killer (Tong & Pratte, 2012). Now comes the question of confidentiality; should this medical check-up result be left as it is, or should the “extra” results gained from the fMRI be used to draw conclusions that the patient has the potential to commit murder and should be therefore be under surveillance or even imprisoned.

In the consideration of the ethical issues of neuroscience, namely the usage of fMRI scans; not only the ethics but also the legal and social impacts should also be considered (Illes, 2010). Further advances in technology will make the results of an fMRI more and more accurate; giving the results from an fMRI the potential to be used as a form of evidence in the court. This could lead to court orders demanding fMRI scans of both the victim and the perpetrator; which in turn could violate one’s mental privacy as the evidence to be used in court could trump over consent (Ladd & Berry, 2013).

Regarding the confidentiality of the results from an fMRI scan, these results could have a large social impact on the patient; the viewing of their mental states and the potential medical results lie well within neuroethics. As seen in one case of an MRI scan in the 1980s, the patient was found to have a brain anomaly which had potential medical significance; however the researchers were not medically trained and they had no protocol to deal with such an incidental finding. Whether or not the MRI results was a real threat, the researchers chose to remain silent as disclosure of that information would have ended the patient's career as a pilot (Illes, 2010).




5. Critical Analysis


Prof. Michio Kaku illustrated on FOX News a future of mind-reading, claiming it would be a gateway for revealing human minds, thus, mind reading. Is it well-founded, solid science or just some science fiction? In this session, the news material will be critiqued on several aspects.

Media Background

The material used in the study is extracted from a Fox news session. Fox News is well known for its poor reputation in the journalism industry, it has been consistently criticized by media, professionals and individuals for their misleading news reporting strategy. Its poor news reporting is targeted at the general public in the US and some overseas countries, which could be a potential threat for scientific literacy and its relay to the general public.

Unprofessional Interview

The reporter Bill Hemmer, whether intentionally or unintentionally, is misleading the audience with unsubstantiated claims by suggesting Prof Kaku is involved in neural decoding research by introducing Kaku as ‘foreseeing the valid future development of neural decoding technology’. This demonstrates a poor example of guest invitation for specialized scientific knowledge. Prof Michio Kakua is Theoretical Physicist. Although physics has covered and has been one of the most important foundations for scientific development, inviting a physicist to shed insight for future neuroscientific development is inappropriate. Kaku although being a respected scientist in the field of physics, does not imply he has a thorough understanding of the development of neuroscience technology and the limitations of both technical and ethical issues during research.

Misconception - Fact, fiction or opinion?

Hemmer and Kaku mentioned different concepts in the media, some of which showed significant misconceptions to audience.

When Kaku was asked about the gradual development of science fiction to fact, although having a legitimate explanation on the current technology of fMRI and the development on resolving the alphabet which requires mainly visual decoding, he has jumped a significant leap suggesting a future development of a dictionary of thought. This explanation has misled the audience on the misconception of the nature of visual decoding and state of mind even if they have different approaches of investigation.

Another issue that should be noted is that Hemmer has misled the audience by suggesting that scientists have successfully performed mind-reading by showing pictures to experimental subjects. This misleads the audience to believe that there is a technology of mind reading all other thoughts instead of imaginative thought of the picture. Kaku answered that computers recognise the patterns of blood chemical flow to certain regions but has failed to explain such recognition processes. This contributes to Hemmer's misconception and also has not provided a clear explanation in order to explain the actual experiment being done. In fact, the experiment should be viewed as a visual imagery task, in which pictures are shown in a perception task for a recording of brain pattern by fMRI and therefore does not imply there is any plausible possibility of even the future concept of the dictionary of thought which would be able to suit every participant.

Kaku's follow-up answer proved his lack of understanding on current technological limitations, as he mentioned the future possibilities of the interrogation of terrorists using fMRI scanning. For the current or foreseeable future, it is unlikely to have neural reading of complex thoughts as Kaku has mentioned. This is due to the current stage in technology where there are limits to single items that require a 'pre-scanning process', poor resolution and also the limitation of variation of an individual's neuroanatomy and functionality. Although resolution could be improved over time, it still raises a humanitarian concern that could potentially lead to a neuroethical issue.

The dictionary of thought as mentioned by Kaku is very likely to be limited by individual brain differences, not only in gross anatomical levels but also at the molecular level. Neural plasticity differences might also lead to significant errors that would not enable the dictionary of thought to be well defined, simply by recording patterns of blood flow in brain. Although BOLD contrast in the brain might provide cues for certain categorical information, for example, responding to faces by activation of Fusiform Face Area (FFA), it does not provide a precise prediction of what exactly is being thought.

Future Implications of Current Technology

Instead of the implications on interrogation, or as court verdict material, fMRI neural decoding provides a better understanding of memory formation, perception, state of mind and aspects of consciousness. Future development focusing on improvement of resolution could possibly provide a more detailed understanding on brain processing information, which could possibly lead to further medical advancements.

Conclusion

FOX news once again has provided an excellent example on how poor journalism can be. First, the poor selection of guest interviewee on a topic the guest is not familiar with, that provides evidence that the audience could easily be misled by a non-expert that lack enough understanding on neuroscience. Second, questions asked by Hemmer showed his inability to understand original scientific journal articles and to portray to an audience, an understanding of mind reading technology. Thirdly, the nature of visual decoding and state of mind decoding should not be mixed up and that further explains the reason why an expert in neuroscience should be interviewed instead. In conclusion, such irresponsible media reports could possibly provide concerning misunderstandings towards neuroscience development by the general public, which could lead to controversies on ethical or privacy issues.



6. Appendix


The media was selected on the basis of being an appealing topic with controversial issues being discussed. It was one of many topics that we had all suggested and unanimously decided to endeavour. This notion was supported with the large amounts of research being done in this field for future technological and applicable advancements. The articles cited and referred to in the wiki were all chosen based on relevance, validity of techniques used in each experiment and reliability of conclusions across each article.

This task was reached though discussion within the group over a lunch meeting at a Thai restaurant after the Thursday lab. After having arrived at the conclusion that our proposed topics had either been done by other groups, or were far too crass to pull off. Someone mentioned Edward Snowden and the invasion of privacy issues surrounding the abuse of power; another person mentioned recent research from DARPA into brain-computer interfaces. From that point the topic drifted to if the government could would they read our minds? Spurred on by our lack of reasonably interesting alternative topics, as well as our love of science fiction and delicious Pad Thai. We decided on the subject of neural decoding as it was the closest thing we had to mind reading.


Summary of Reviewer Comments

Overall the reviewers' comments were very helpful, they helped us notice things that were in desperate need of changing or adding as well as reminding us of some things we had forgotten to mention or include. After considering each piece of advice that the reviewers had to offer, we made a brief list of things that needed fixing. We needed to rethink our approach to explaining visual encoding, ensuring we focussed on things like fMRI accuracy and retinotopy; we need to better explain our figures in order to make them easier to understand; we need more referencing in some areas; some issues with formatting need addressing to make the page more user friendly and we need to ensure our spelling and grammar are correct throughout.


Application of reviewer comments

With explaining visual encoding and how this might become synonymous with mind reading, we changed our approach slightly and gave more focus on forming links between our information on neural imaging and the concept of “mind reading”. We added better explanations of our figures to make them easier to understand. With formatting, some specific ways in which we made the page more user friendly include moving the video further up the page and constructing a table of contents. With referencing, we made a real effort to back up all of our claims including those for the critique of Fox News’s quality of journalism.

To Richard,

Discussion on the wiki space was not utilized as much as it could have been. For maximum transparency, here is a link to our facebook group.

Kind regards,

Anthony Deng – z3418031, Man Fai Ho – z3372323, Richard Lim – z3418068, John Nguyen – z3420078, Gerard Weber – z3373160


https://www.facebook.com/groups/201323793362995/





7. References
Andersen, R.A., Musallam, S. & Pesaran, B. 2004. Selecting the Signals for a Brain-Machine Interface. Current Opinion in Neurobiology. 14, 720-726.

Belitski, A., Panzeri, S., Magri, C., Logothetis, N.K. & Kayser, Christoph. 2010. Sensory information in local field potentials and spikes from visual and auditory cortices: time scales and frequency bands. Journal of Computational Neuroscience. 29, 533-545.

Brouwer, G. J., & Heeger, D. J. (2009). Decoding and reconstructing color from responses in human visual cortex. The Journal of Neuroscience, 29(44), 13992-14003.

Haynes, J.D. & Rees, G. 2006. Decoding mental states from brain activity in humans. Nature Reviews - Neuroscience. 7, 523-534.

Illes, J. (2010). Empowering brain science with neuroethics. The Lancet ,376(9749), 1294-1295.

Illes, J., Bird, S. J. (2006). Neuroethics: a modern context for the ethics in neuroscience. Trends Neurosci, 9, 511-517.

Kamitani, Y. & Tong, F. 2005. Decoding the visual and subjective contents of the human brain. Nature Neuroscience. 8(5), 679-685.

Kourtzi, Z., & Kanwisher, N. (2000). Activation in human MT/MST by static images with implied motion. Journal of cognitive neuroscience, 12(1), 48-55.


Ladd, S., Berry, J. L. (2013). The Potential Role of fMRI in Lie Detection. http://www.eradimaging.com/site/article.cfm?ID=792

Lebedev, M. A., & Nicolelis, M. A. (2006). Brain–machine interfaces: past, present and future. TRENDS in Neurosciences, 29(9), 536-546.

Logothetis, N.K. 2003. The Underpinning of the BOLD Functional Magnetic Resonance Imaging Signal. The Journal of Neuroscience. 23(10), 3963-3971.

Sun, P., Gardner, J.L., Costagli, M., Ueno, K., Waggoner, R.A., Tanaka, K. & Cheng, K. 2013. Demonstration of Tuning to Stimulus Orientation in the Human Visual Cortex: A High-Resolution fMRI study with a Novel Continuous and Periodic Stimulation Paradigm. Cerebral Cortex, 23(7), 1618-1629.

Taylor, D. M., Tillery, S. I. H., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296(5574), 1829-1832.

Tong, F., Pratte, M.S. (2012). Decoding Patterns of Human Brain Activity. Annu. Rev. Psychol. 63, 483-509

Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453(7198), 1098-1101.

Wandell, B.A., Dumoulin, S.O. & Brewer, A.A. 2007. Visual Field Maps in Human Cortex. Neuron. 56(2), 366-383

Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G. & Vaughan, T.M. 2002. Brain-computer interfaces for communication and control. Clinical Neurophysiology. 113, 767-791.

Zimmermann, J., Goebel, R., De Martino, F., van de Moortele, P. F., Feinberg, D., Adriany, G., ... & Yacoub, E. (2011). Mapping the organization of axis of motion selective features in human area MT using high-field fMRI. PLoS One,6(12), e28716.



Group Members
Anthony Deng – z3418031
Man Fai Ho – z3372323
Richard Lim – z3418068
John Nguyen – z3420078
Gerard Weber – z3373160


Very cool - I like it (plus the Physics Prof has the whole mad scientist hair going on). Be clear in spelling out what this finding means and what its significance is, but a consideration of future technology, along the lines you have outlined would also be good.

Approved!


2013-08-15 12.07.52.jpg



Group Roles
1. Introduction and background of neural decoding
- Gerard Weber
2. Methods of neural decoding and analysis
- John Nguyen
3. Measuring neural activity (neuroimaging including fMRI, EEG) and analysis
- Richard Lim
4. Ethics involved with “mind reading”
- Anthony Deng
5. Critical analysis of media and future implications
- Man Fai Ho

Deadlines
Submission of proposal – August 12th.
First draft – September 9th.
Proofreading and editing (after critical review) –September 16th - September 22nd.
Final wiki – September 23rd.

*Minute attached as PDF