## Visual Cryptography And Stenography

 This article is about Visual Cryptography. Visual Cryptography is a technique that allows information (images, text, diagrams …) to be encrypted using an encoding system that can be decrypted by the eyes. It does not require a computer to decode.

Source: http://www.datagenetics.com/blog/november32013/

The technique I’m going to describe is attributed to two great mathematicians: Moni Naor and Adi Shamir, in 1994. In this implementation, I’m going to show how to split a secret message into two components. Both parts are necessary to reconstruct and reveal the secret, and the possession of either one, alone, is useless in determining the secret.

 The basis of the technique is the superposition (overlaying) of two semi-transparent layers. Imagine two sheets of transparency covered with a seemingly random collection of black pixels.

Individually, there is no discernable message printed on either one of the sheets. Overlapping them creates addition interference to the light passing through (mathematically the equivalent of performing a Boolean OR operation with the images), but still it just looks like a random collection of pixels.

 Mysteriously, however, if the two grids are overlaid correctly, at just the right position, a message magically appears! The patterns are designed to reveal a message.

### Demonstration

Let’s look at couple of examples of this in action, then we’ll describe how the technique works.

Below you will see two random looking rectangles of dots. One is fixed in the center, and the other you can drag around the canvas. As the rectangles intersect, the images merge. If you align the rectangles perfectly, a hidden message will appear. There are three hidden message to see in this demonstration, once you’ve decoded one, click on the square button in the bottom left to advance to the next.

To give you feedback, once the images are perfectly aligned, the advance button will go blank with a red border (don’t worry, your computer will not self-destruct in five seconds)

### How does it work?

 First we take a monochrome image for the source. Pixels in the image are either white or black. To the right is the source for the first example we saw above.
 Next we sub-divide each pixel into four smaller subpixels. We need to shade these four subpixels to represent the source image, then subjectively divide them between the two cypher images we are to create.

We need to distribute the shading such that, if you have just one of the cypher images, it is impossible to determine what is on the other cypher image, and thus, impossible to decrypt the image.

What we do is look at the color of each pixel in the original source image. If the original pixel in the image is set (black), we fill in all four sub pixels then distribute them two per cypher layer. We flip a coin to determine which pattern we place on which layer (so that it is random). It does not matter which pair of pixels goes on which layer, when they are combined, all four pixels will be black.

Conversely, if the source image pixel is white, we shade in just two pixels. This time, however, we make sure that the same pixels are shaded on both layers. In this way, when the two cypher images are combined, only two pixels are shaded. As before, we flip a coin to determine which chiral set we go with, and make sure the same image appears on both layers.

 The result of this process is two images (both four times as large as the original) which when combined result in an image with half the contrast of the original. The black of the source remains black in the combined cypher, but the white in the source is changed to a randomly mottled half-tone gray. Luckily this is still sufficiently high enough contrast for the secret message to be easily read.

Someone who has possession of only one of the cypher images will be able to determine the (2 x 2) pattern of each pixel but has no idea if the corresponding pixel cluster on the other image is the same (white space), or opposite (black pixel). Every grid of (2 x 2) sub pixels on both layers contains exactly two pixels.

 Of course, the two pixels selected do not have to follow checker-board pattern I used above. As long as two are shaded at random, and the rules followed as to whether the same, or complementary, pixels are shaded on the other layer, the system will work.

Here is a short animation of a some of these (2 x 2) pixel sub-blocks sliding over each other:

Pretty cool, huh? Well hold on, it gets cooler …

### Moni Naor, Adi Shamir, and more people …

The original paper by Naor and Shamir talks about how to implement this system in a more generic way. For instance, instead of splitting the image into just two cypher texts, why don’t we split the image between n-cyphertexts; all of which are needed to be combined to reveal the final image? (Or possibly a subset of any k images out of these n).

If you are interested in reading more, you can find a reprint of the original paper here.

As an example, here are some (3 x 3) sub-elements that could be used to distribute an image over four cypher images, all of which are needed to be combined to reveal the secret images:

The top line shows the subpixels used to represent a black pixel in the original images, and the bottom line a white pixel.

• Any single share contains exactly five black subpixels.

• Any stacked pair contains exactly seven black subpixels.

• Any stacked triplet contains eight black subpixels.

 However, when all four in each row are combined, the top row contains nine subpixels (all black), whilst the lower row contains only eight (allowing light to shine through and creating the contrast necessary to read the image). You can see from this how the colluding of any two or any three people is not enough to reveal the secret. (Mathematically it’s possible to do this with eight, not nine, sub-pixels , but there’s no easy way to sub divide and pack a square array with eight!)

## Deeper down the rabbit hole: Visual Steganography

 We can use this technique to do something even cooler! Imagine that, in addition to the two source images, we have a third secret image we want to encode. Let’s say we want to produce two cypher images that look ‘innocent’, but secretly hide the third. The generated two cypher images could be printed on transparencies and made to look like legitimate images of no consequence. However, these images, when combined in just the correct way, could be used reveal a third message. The technology of hiding images inside other images is called Steganography.

I’ve done this below. Trust me, you’re not going to believe this at first. You’re going to be convinced that there is some ‘behind the scenes’ script at work that changes the image. I assure you this is not the case. You’re still not going to believe me!

Below, on the left, is my name, encoded from a monochrome image, and also containing partial details of a third hidden image. On the right is the word ‘Fish’, similarly encoded. Now, drag the right image over to the left image and watch what happens when they overlap perfectly. Wham! How cool is that?

### How does this magic work?

The hidden image we are encoding has black pixels and white pixels. As before, we sub divide each pixel into (2 x 2) subpixels. When the two images are combined, we want to represent the black pixels of the hidden image by having all four subpixels black. We’ll represent the white pixels has having three subpixels black. This is sufficient contrast for the hidden image to be seen.

For each black or white pixel in the hidden image, there are four possible combinations of black and white pixels of the two source images. For the two source images, we’re going to say that any three black subpixels represents black in that source image, and any two pixels represents white.

Examples of all eight permutations of source, image 1 and image 2 are depicted below:

When the hidden image pixel is BLACK:

• The combined two cypher images (OR) have to have all four subpixels set.

• When both source images also have a black pixel, this is easy. Both cypher images need to have three out of the four subpixels set. The only constraint is that the missing subpixel is not the same on both layers. One subpixel is randomly selected on the first layer, and one is randomly select from the other three on the second layer.

• When the first image has a black pixel (requiring three subpixels set), and the second image has a white pixel (requiring two subpixels set), as above, first, a random single subpixel is selected on the black layer to remove. Next two subpixels are randomly selected on the second layer with the constraint that one of the selected subpixels is the same as the gap in the first layer. In this way, when the two are combined, four black subpixels are displayed.

• The opposite happens when the first layer is white, and the second layer is black.

• Finally, if both source pixels are white (requiring just two subpixels set), two subpixels are selected at random on the first layer, and the inverse of this selection used for the second layer.

When the hidden image pixel is WHITE:

• The combined two cypher images (OR) have to have any three subpixels set.

• When both source images have a black pixel, this is easy. Both cypher images need to have three out of the four subpixels set, and these need to be the same subpixels. Three subpixels are randomly selected and these are set on both of the cypher image layers.

• When the first image has a black pixel (requiring three subpixels set), and the second image has a white pixel (requiring two subpixels set), as above, first, three random subpixels are selected on the first layer. Next one of these three subpixels is randomly selected for removal and this pattern is used on the second layer.

• The opposite happens when the first layer is white, and the second layer is black.

• Finally, if both source pixels are white (requiring two subpixels set), two are selected at random on the first layer, then one of these is duplicated on the second layer, and a second random subpixel is selected on the second layer (from the two white subpixels not selected on the first layer). Both layers have two subpixels, and when combined, there are three subpixels visbile.

### Other potential uses of the concept

The ability to give an answer, and potentially mask a true answer to a question, tangentially, reminds me of a technique used to get truthful representations in surveys where the subject is potentially embarrassing or where there is incentive to not give a truthful answer.

 Imagine you are conducting a survey with the aim of measuring certain characteristics of your audience, and the subject of some of the questions is sensitive (for example, questions about political preference, sexual orientation, whether you have committed fraud, or cheated, or made a mistake that has cost your company thousands of dollars). People might have a motivation to give a non-truthful answers, possible from embarrassment, peer pressure, or fear. Also, paranoid people might not want to give truthful answers for fear that, even if the survey is anonymous, answers to other questions might be enough to allow an individual to be distinctly identified and thus his answers to the sensitive questions determined.
 The solution? Give the people taking your questionnaire a coin. When the question appears e.g. “Have you ever made a mistake that has cost your company thousands of dollars?”, ask the subject to flip a coin. If the coin comes up HEADS, tell the person to answer the question truthfully. If the coin comes up TAILS, tell the person to flip the coin again and if the coin lands HEADS to answer ”Yes” and if the second flip comes up TAILS to answer the question ”No”.

Any person looking at the survey results and seeing a ”Yes” on an answer will not know if any single person’s answer is truthful, or the result of a coin flip. Any person can be free of embarrassment as none of his/her peers will know either.

 The law of large numbers, however, will allow a good estimate of the number of people “Who have made a costly mistake”, because you’d be able to subtract the number of expected fake ”Yes” answers, then scale up the remainder of the answers.

# Steganography

Not to be confused with Stenography.

Steganography (, ) is the practice of concealing a file, message, image, or video within another file, message, image, or video. The word steganography combines the Greek words steganos (στεγανός), meaning “covered, concealed, or protected”, and graphein (γράφειν) meaning “writing”.

The first recorded use of the term was in 1499 by Johannes Trithemius in his Steganographia, a treatise on cryptography and steganography, disguised as a book on magic. Generally, the hidden messages appear to be (or be part of) something else: images, articles, shopping lists, or some other cover text. For example, the hidden message may be in invisible ink between the visible lines of a private letter. Some implementations of steganography that lack a shared secret are forms of security through obscurity, whereas key-dependent steganographic schemes adhere to Kerckhoffs’s principle.[1]

The advantage of steganography over cryptography alone is that the intended secret message does not attract attention to itself as an object of scrutiny. Plainly visible encrypted messages—no matter how unbreakable—arouse interest, and may in themselves be incriminating in countries where encryption is illegal.[2] Thus, whereas cryptography is the practice of protecting the contents of a message alone, steganography is concerned with concealing the fact that a secret message is being sent, as well as concealing the contents of the message.

Steganography includes the concealment of information within computer files. In digital steganography, electronic communications may include steganographic coding inside of a transport layer, such as a document file, image file, program or protocol. Media files are ideal for steganographic transmission because of their large size. For example, a sender might start with an innocuous image file and adjust the color of every 100th pixel to correspond to a letter in the alphabet, a change so subtle that someone not specifically looking for it is unlikely to notice it.

## History

The first recorded uses of steganography can be traced back to 440 BC when Herodotus mentions two examples in his Histories.[3] Demaratus sent a warning about a forthcoming attack to Greece by writing it directly on the wooden backing of a wax tablet before applying its beeswax surface. Wax tablets were in common use then as reusable writing surfaces, sometimes used for shorthand.

In his work Polygraphiae Johannes Trithemius developed his so-called “Ave-Maria-Cipher” that can hide information in a Latin praise of God. “Auctor Sapientissimus Conseruans Angelica Deferat Nobis Charitas Potentissimi Creatoris” for example contains the concealed word VICIPEDIA.[4]

## Techniques

### Physical

Steganography has been widely used, including in recent historical times and the present day. Known examples include:

• Hidden messages within wax tablet—in ancient Greece, people wrote messages on wood and covered it with wax that bore an innocent covering message.
• Hidden messages on messenger’s body—also used in ancient Greece. Herodotus tells the story of a message tattooed on the shaved head of a slave of Histiaeus, hidden by the hair that afterwards grew over it, and exposed by shaving the head. The message allegedly carried a warning to Greece about Persian invasion plans. This method has obvious drawbacks, such as delayed transmission while waiting for the slave’s hair to grow, and restrictions on the number and size of messages that can be encoded on one person’s scalp.
• During World War II, the French Resistance sent some messages written on the backs of couriers in invisible ink.
• Hidden messages on paper written in secret inks, under other messages or on the blank parts of other messages
• Messages written in Morse code on yarn and then knitted into a piece of clothing worn by a courier.
• Messages written on envelopes in the area covered by postage stamps.
• In the early days of the printing press, it was common to mix different typefaces on a printed page due to the printer not having enough copies of some letters in one typeface. Because of this, a message could be hidden using two (or more) different typefaces, such as normal or italic.
• During and after World War II, espionage agents used photographically produced microdots to send information back and forth. Microdots were typically minute (less than the size of the period produced by a typewriter). World War II microdots were embedded in the paper and covered with an adhesive, such as collodion. This was reflective, and thus detectable by viewing against glancing light. Alternative techniques included inserting microdots into slits cut into the edge of post cards.
• During WWII, Velvalee Dickinson, a spy for Japan in New York City, sent information to accommodation addresses in neutral South America. She was a dealer in dolls, and her letters discussed the quantity and type of doll to ship. The stegotext was the doll orders, while the concealed “plaintext” was itself encoded and gave information about ship movements, etc. Her case became somewhat famous and she became known as the Doll Woman.
• Jeremiah Denton repeatedly blinked his eyes in Morse Code during the 1966 televised press conference that he was forced into as an American POW by his North Vietnamese captors, spelling out “T-O-R-T-U-R-E”. This confirmed for the first time to the U.S. Military (naval intelligence) and Americans that the North Vietnamese were torturing American POWs.
• Cold War counter-propaganda. In 1968, crew members of the USS Pueblo intelligence ship held as prisoners by North Korea, communicated in sign language during staged photo opportunities, informing the United States they were not defectors, but captives of the North Koreans. In other photos presented to the US, crew members gave “the finger” to the unsuspecting North Koreans, in an attempt to discredit photos that showed them smiling and comfortable.

### Digital messages

Image of a tree with a steganographically hidden image. The hidden image is revealed by removing all but the two least significant bits of each color component and a subsequent normalization. The hidden image is shown below.

Image of a cat extracted from the tree image above.

Modern steganography entered the world in 1985 with the advent of personal computers being applied to classical steganography problems.[5] Development following that was very slow, but has since taken off, going by the large number of steganography software available:

• Concealing messages within the lowest bits of noisy images or sound files.
• Concealing data within encrypted data or within random data. The message to conceal is encrypted, then used to overwrite part of a much larger block of encrypted data or a block of random data (an unbreakable cipher like the one-time pad generates ciphertexts that look perfectly random without the private key).
• Chaffing and winnowing.
• Mimic functions convert one file to have the statistical profile of another. This can thwart statistical methods that help brute-force attacks identify the right solution in a ciphertext-only attack.
• Concealed messages in tampered executable files, exploiting redundancy in the targeted instruction set.
• Pictures embedded in video material (optionally played at slower or faster speed).
• Injecting imperceptible delays to packets sent over the network from the keyboard. Delays in keypresses in some applications (telnet or remote desktop software) can mean a delay in packets, and the delays in the packets can be used to encode data.
• Changing the order of elements in a set.
• Content-Aware Steganography hides information in the semantics a human user assigns to a datagram. These systems offer security against a nonhuman adversary/warden.
• Blog-Steganography. Messages are fractionalized and the (encrypted) pieces are added as comments of orphaned web-logs (or pin boards on social network platforms). In this case the selection of blogs is the symmetric key that sender and recipient are using; the carrier of the hidden message is the whole blogosphere.
• Modifying the echo of a sound file (Echo Steganography).[6]
• Steganography for audio signals.[7]
• Image bit-plane complexity segmentation steganography
• Including data in ignored sections of a file, such as after the logical end of the carrier file.

#### Digital text

• Making text the same color as the background in word processor documents, e-mails, and forum posts.
• Using Unicode characters that look like the standard ASCII character set. On most systems, there is no visual difference from ordinary text. Some systems may display the fonts differently, and the extra information would then be easily spotted, of course.
• Using hidden (control) characters, and redundant use of markup (e.g., empty bold, underline or italics) to embed information within HTML, which is visible by examining the document source. HTML pages can contain code for extra blank spaces and tabs at the end of lines, and colours, fonts and sizes, which are not visible when displayed.
• Using non-printing Unicode characters Zero-Width Joiner (ZWJ) and Zero-Width Non-Joiner (ZWNJ).[8] These characters are used for joining and disjoining letters in Arabic and Persian, but can be used in Roman alphabets for hiding information because they have no meaning in Roman alphabets: because they are “zero-width” they are not displayed. ZWJ and ZWNJ can represent “1” and “0”.

#### Social steganography

In communities with social or government taboos or censorship, people use cultural steganography—hiding messages in idiom, pop culture references, and other messages they share publicly and assume are monitored. This relies on social context to make the underlying messages visible only to certain readers.[9][10] Examples include:

• Hiding a message in the title and context of a shared video or image
• Misspelling names or words that are popular in the media in a given week, to suggest an alternate meaning

### Network

All information hiding techniques that may be used to exchange steganograms in telecommunication networks can be classified under the general term of network steganography. This nomenclature was originally introduced by Krzysztof Szczypiorski in 2003.[11] Contrary to typical steganographic methods that use digital media (images, audio and video files) to hide data, network steganography uses communication protocols’ control elements and their intrinsic functionality. As a result, such methods are harder to detect and eliminate.[12]

Typical network steganography methods involve modification of the properties of a single network protocol. Such modification can be applied to the PDU (Protocol Data Unit),[13][14][15] to the time relations between the exchanged PDUs,[16] or both (hybrid methods).[17]

Moreover, it is feasible to utilize the relation between two or more different network protocols to enable secret communication. These applications fall under the term inter-protocol steganography.[18]

Network steganography covers a broad spectrum of techniques, which include, among others:

• Steganophony — the concealment of messages in Voice-over-IP conversations, e.g. the employment of delayed or corrupted packets that would normally be ignored by the receiver (this method is called LACK — Lost Audio Packets Steganography), or, alternatively, hiding information in unused header fields.[19]
• WLAN Steganography – transmission of steganograms in Wireless Local Area Networks. A practical example of WLAN Steganography is the HICCUPS system (Hidden Communication System for Corrupted Networks)[20]

### Printed

Digital steganography output may be in the form of printed documents. A message, the plaintext, may be first encrypted by traditional means, producing a ciphertext. Then, an innocuous covertext is modified in some way so as to contain the ciphertext, resulting in the stegotext. For example, the letter size, spacing, typeface, or other characteristics of a covertext can be manipulated to carry the hidden message. Only a recipient who knows the technique used can recover the message and then decrypt it. Francis Bacon developed Bacon’s cipher as such a technique.

The ciphertext produced by most digital steganography methods, however, is not printable. Traditional digital methods rely on perturbing noise in the channel file to hide the message, as such, the channel file must be transmitted to the recipient with no additional noise from the transmission. Printing introduces much noise in the ciphertext, generally rendering the message unrecoverable. There are techniques that address this limitation, one notable example is ASCII Art Steganography.[21]

### Using puzzles

The art of concealing data in a puzzle can take advantage of the degrees of freedom in stating the puzzle, using the starting information to encode a key within the puzzle / puzzle image.

For instance, steganography using sudoku puzzles has as many keys as there are possible solutions of a sudoku puzzle, which is 6.71×1021. This is equivalent to around 70 bits, making it much stronger than the DES method, which uses a 56 bit key.[22]

Discussions of steganography generally use terminology analogous to (and consistent with) conventional radio and communications technology. However, some terms show up in software specifically, and are easily confused. These are most relevant to digital steganographic systems.

The payload is the data covertly communicated. The carrier is the signal, stream, or data file that hides the payload—which differs from the channel (which typically means the type of input, such as a JPEG image). The resulting signal, stream, or data file with the encoded payload is sometimes called the package, stego file, or covert message. The percentage of bytes, samples, or other signal elements modified to encode the payload is called the encoding density, and is typically expressed as a number between 0 and 1.

In a set of files, those files considered likely to contain a payload are suspects. A suspect identified through some type of statistical analysis might be referred to as a candidate.

## Countermeasures and detection

Detecting physical steganography requires careful physical examination—including the use of magnification, developer chemicals and ultraviolet light. It is a time-consuming process with obvious resource implications, even in countries that employ large numbers of people to spy on their fellow nationals. However, it is feasible to screen mail of certain suspected individuals or institutions, such as prisons or prisoner-of-war (POW) camps.

During World War II, prisoner of war camps gave prisoners specially treated paper that would reveal invisible ink. An article in the 24 June 1948 issue of Paper Trade Journal by the Technical Director of the United States Government Printing Office, Morris S. Kantrowitz, describes, in general terms, the development of this paper. They used three prototype papers named Sensicoat, Anilith, and Coatalith. These were for the manufacture of post cards and stationery provided to German prisoners of war in the US and Canada. If POWs tried to write a hidden message, the special paper rendered it visible. The U.S. granted at least two patents related to this technology—one to Kantrowitz, U.S. Patent 2,515,232, “Water-Detecting paper and Water-Detecting Coating Composition Therefor,” patented 18 July 1950, and an earlier one, “Moisture-Sensitive Paper and the Manufacture Thereof”, U.S. Patent 2,445,586, patented 20 July 1948. A similar strategy is to issue prisoners with writing paper ruled with a water-soluble ink that runs in contact with water-based invisible ink.

In computing, steganographically encoded package detection is called steganalysis. The simplest method to detect modified files, however, is to compare them to known originals. For example, to detect information being moved through the graphics on a website, an analyst can maintain known clean-copies of these materials and compare them against the current contents of the site. The differences, assuming the carrier is the same, comprise the payload. In general, using extremely high compression rates makes steganography difficult, but not impossible. Compression errors provide a hiding place for data, but high compression reduces the amount of data available to hold the payload, raising the encoding density, which facilitates easier detection (in extreme cases, even by casual observation).

## Applications

### Use in modern printers

Main article: Printer steganography

Some modern computer printers use steganography, including HP and Xerox brand color laser printers. These printers add tiny yellow dots to each page. The barely-visible dots contain encoded printer serial numbers and date and time stamps.[23]

### Example from modern practice

The larger the cover message (in binary data, the number of bits) relative to the hidden message, the easier it is to hide the latter. For this reason, digital pictures (which contain large amounts of data) are used to hide messages on the Internet and on other communication media. It is not clear how common this actually is. For example: a 24-bit bitmap uses 8 bits to represent each of the three color values (red, green, and blue) at each pixel. The blue alone has 28 different levels of blue intensity. The difference between 11111111 and 11111110 in the value for blue intensity is likely to be undetectable by the human eye. Therefore, the least significant bit can be used more or less undetectably for something else other than color information. If this is repeated for the green and the red elements of each pixel as well, it is possible to encode one letter of ASCII text for every three pixels.

Stated somewhat more formally, the objective for making steganographic encoding difficult to detect is to ensure that the changes to the carrier (the original signal) due to the injection of the payload (the signal to covertly embed) are visually (and ideally, statistically) negligible; that is to say, the changes are indistinguishable from the noise floor of the carrier. Any medium can be a carrier, but media with a large amount of redundant or compressible information are better suited.

From an information theoretical point of view, this means that the channel must have more capacity than the “surface” signal requires; that is, there must be redundancy. For a digital image, this may be noise from the imaging element; for digital audio, it may be noise from recording techniques or amplification equipment. In general, electronics that digitize an analog signal suffer from several noise sources such as thermal noise, flicker noise, and shot noise. This noise provides enough variation in the captured digital information that it can be exploited as a noise cover for hidden data. In addition, lossy compression schemes (such as JPEG) always introduce some error into the decompressed data; it is possible to exploit this for steganographic use as well.

Steganography can be used for digital watermarking, where a message (being simply an identifier) is hidden in an image so that its source can be tracked or verified (for example, Coded Anti-Piracy), or even just to identify an image (as in the EURion constellation).

### Alleged use by intelligence services

In 2010, the Federal Bureau of Investigation alleged that the Russian foreign intelligence service uses customized steganography software for embedding encrypted text messages inside image files for certain communications with “illegal agents” (agents under non-diplomatic cover) stationed abroad.[24]

### Distributed steganography

There are distributed steganography methods,[25] including methodologies that distribute the payload through multiple carrier files in diverse locations to make detection more difficult. For example, U.S. Patent 8,527,779 by cryptographer William Easttom (Chuck Easttom).

### Online challenge

The online mechanism Cicada 3301 incorporates steganography with cryptography and other solving techniques since 2012.[26]

## Synesthesia / Cypher

Interesting human condition. A phenomena where colors are asociated with letters or sounds or tones.

Cypher:

I once thought about “writing in color” making an alphabet based on colors and thus creating a cypher that way. One could use #values as well.

Instead of color one can also use musical notes or frequencies to create a similar cypher.

Relating colors, tones and letters is an interesting exercise for me in the near future.

It can be in a way related  to visual cryptography and is a interesting way to encode knowledge in my opinion.

How someone with synesthesia might perceive (not “see”) certain letters and numbers. Synesthetes see characters just as others do (in whichever color actually displayed) but simultaneously perceive colors as associated with or evoked by each one.
Classification and external resources
Specialty Neurology
ICD10 R44.8
MeSH C562460

Synesthesia (also spelled synæsthesia or synaesthesia; from the Ancient Greek σύν syn, “together”, and αἴσθησις aisthēsis, “sensation“) is a neurological phenomenon in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway.[1][2][3][4] People who report such experiences are known as synesthetes.

Difficulties have been recognized in adequately defining synesthesia:[5][6] Many different phenomena have been included in the term synesthesia (“union of the senses”), and in many cases the terminology seems to be inaccurate. A more accurate term may be ideasthesia.

In one common form of synesthesia, known as grapheme → color synesthesia or color-graphemic synesthesia, letters or numbers are perceived as inherently colored.[7][8] In spatial-sequence, or number form synesthesia, numbers, months of the year, and/or days of the week elicit precise locations in space (for example, 1980 may be “farther away” than 1990), or may appear as a three-dimensional map (clockwise or counterclockwise).[9][10]

Only a fraction of types of synesthesia have been evaluated by scientific research.[11] Awareness of synesthetic perceptions varies from person to person.[12]

Although synesthesia was the topic of intensive scientific investigation in the late 19th and early 20th centuries, it was largely abandoned by scientific research in the mid-20th century.[13] Psychological research has demonstrated that synesthetic experiences can have measurable behavioral consequences, and functional neuroimaging studies have identified differences in patterns of brain activation.[8] Many find that synesthesia aids the creative process.[citation needed] Psychologists and neuroscientists study synesthesia not only for its inherent appeal, but also for the insights it may give into cognitive and perceptual processes that occur in synesthetes and non-synesthetes alike.

## CharacteristicsEdit

There are two overall forms of synesthesia: projecting synesthesia and associative synesthesia. People who project will see actual colors, forms, or shapes when stimulated, as is commonly accepted as synesthesia; associators will feel a very strong and involuntary connection between the stimulus and the sense that it triggers. For example, in the common form chromesthesia (sound to color) a projector may hear a trumpet and see an orange triangle in space while an associator might hear a trumpet and think very strongly that it sounds “orange”.

Some synesthetes often report that they were unaware their experiences were unusual until they realized other people did not have them, while others report feeling as if they had been keeping a secret their entire lives.[11] The automatic and ineffable nature of a synesthetic experience means that the pairing may not seem out of the ordinary. This involuntary and consistent nature helps define synesthesia as a real experience. Most synesthetes report that their experiences are pleasant or neutral, although, in rare cases, synesthetes report that their experiences can lead to a degree of sensory overload.[14]

Though often stereotyped in the popular media as a medical condition or neurological aberration, many synesthetes themselves do not perceive their synesthetic experiences as a handicap. To the contrary, some report it as a gift—an additional “hidden” sense—something they would not want to miss. Most synesthetes become aware of their distinctive mode of perception in their childhood. Some have learned how to apply their ability in daily life and work. Synesthetes have used their abilities in memorization of names and telephone numbers, mental arithmetic, and more complex creative activities like producing visual art, music, and theater.[11]

Despite the commonalities which permit definition of the broad phenomenon of synesthesia, individual experiences vary in numerous ways. This variability was first noticed early in synesthesia research.[15] Some synesthetes report that vowels are more strongly colored, while for others consonants are more strongly colored.[14] Self reports, interviews, and autobiographical notes by synesthetes demonstrate a great degree of variety in types of synesthesia, intensity of synesthetic perceptions, awareness of the perceptual discrepancies between synesthetes and non-synesthetes, and the ways synesthesia is used in work, creative processes, and daily life.[11][16]

Synesthetes are very likely to participate in creative activities.[17] It has been suggested that individual development of perceptual and cognitive skills, in addition to one’s cultural environment, produces the variety in awareness and practical use of synesthetic phenomena.[12][16]

## FormsEdit

Synesthesia can occur between nearly any two senses or perceptual modes, and at least one synesthete, Solomon Shereshevsky, experienced synesthesia that linked all five senses.[medical citation needed] Types of synesthesia are indicated by using the notation x → y, where x is the “inducer” or trigger experience, and y is the “concurrent” or additional experience. For example, perceiving letters and numbers (collectively called graphemes) as colored would be indicated as grapheme → color synesthesia. Similarly, when synesthetes see colors and movement as a result of hearing musical tones, it would be indicated as tone → (color, movement) synesthesia.

While nearly every logically possible combination of experiences can occur, several types are more common than others.

### Grapheme-color synesthesiaEdit

From Wednesday is Indigo Blue.[3] Note this example’s upside-down clock face.

In one of the most common forms of synesthesia, individual letters of the alphabet and numbers (collectively referred to as graphemes) are “shaded” or “tinged” with a color. While different individuals usually do not report the same colors for all letters and numbers, studies with large numbers of synesthetes find some commonalities across letters (e.g. A is likely to be red).[14]

As a child, Pat Duffy told her father, “I realized that to make an R all I had to do was first write a P and draw a line down from its loop. And I was so surprised that I could turn a yellow letter into an orange letter just by adding a line.” Another grapheme synesthete says, “When I read, about five words around the exact one I’m reading are in color. It’s also the only way I can spell. In elementary school I remember knowing how to spell the word ‘priority’ [with an “i” rather than an “e”] because … an ‘e’ was out of place in that word because ‘e’s were yellow and didn’t fit.”[18]

### ChromesthesiaEdit

Main article: Chromesthesia

Another common form of synesthesia is the association of sounds with colors. For some, everyday sounds such as doors opening, cars honking, or people talking can trigger seeing colors. For others, colors are triggered when musical notes and/or keys are being played. People with synesthesia related to music may also have perfect pitch because their ability to see/hear colors aids them in identifying notes or keys.[citation needed]

The colors triggered by certain sounds, and any other synesthetic visual experiences, are referred to as photisms.

According to Richard Cytowic,[3] chromesthesia is “something like fireworks”: voice, music, and assorted environmental sounds such as clattering dishes or dog barks trigger color and firework shapes that arise, move around, and then fade when the sound ends. Sound often changes the perceived hue, brightness, scintillation, and directional movement. Some individuals see music on a “screen” in front of their faces. For Deni Simon, music produces waving lines “like oscilloscope configurations – lines moving in color, often metallic with height, width and, most importantly, depth. My favorite music has lines that extend horizontally beyond the ‘screen’ area.”

Individuals rarely agree on what color a given sound is. B flat might be orange for one person and blue for another. Composers Liszt and Rimsky-Korsakov famously disagreed on the colors of music keys.

### Spatial sequence synesthesiaEdit

Those with spatial sequence synesthesia (SSS) tend to see numerical sequences as points in space. For instance, the number 1 might be farther away and the number 2 might be closer. People with SSS may have superior memories; in one study, they were able to recall past events and memories far better and in far greater detail than those without the condition. They also see months or dates in the space around them. Some people see time like a clock above and around them.[unreliable medical source?][19][20]

### Number formEdit

Main article: Number form

A number form from one of Francis Galton’s subjects (1881).[9] Note how the first 12 digits correspond to a clock face.

A number form is a mental map of numbers that automatically and involuntarily appears whenever someone who experiences number forms thinks of numbers. Number forms were first documented and named in 1881 by Francis Galton in “The Visions of Sane Persons”.[21]

### Auditory-tactile synesthesiaEdit

In auditory → tactile synesthesia, certain sounds can induce sensations in parts of the body. Auditory → tactile synesthesia may sometimes originate from birth or be acquired sometime later in life.[citation needed] It is one of the least common forms of synesthesia.[22]

### Ordinal linguistic personificationEdit

Ordinal-linguistic personification (OLP, or personification for short) is a form of synesthesia in which ordered sequences, such as ordinal numbers, days, months and letters are associated with personalities (Simner & Hubbard 2006). Although this form of synesthesia was documented as early as the 1890s (Flournoy 1893; Calkins 1893) researchers have, until recently, paid little attention to this form (see History of synesthesia research). Ordinal personification normally co-occurs with other forms of synesthesia such as grapheme-color synesthesia.

### MisophoniaEdit

Main article: Misophonia

Misophonia is a neurological disorder in which negative experiences (anger, fright, hatred, disgust) are triggered by specific sounds. Richard Cytowic suggests that misophonia is related to, or perhaps a variety of, synesthesia.[23] Miren Edelstein and her colleagues have compared misophonia to synesthesia in terms of connectivity between different brain regions as well as specific symptoms. They formed the hypothesis that “a pathological distortion of connections between the auditory cortex and limbic structures could cause a form of sound-emotion synesthesia.”[24]

### Mirror-touch synesthesiaEdit

This is a rare form of synesthesia where individuals feel the same sensation that another person feels (such as touch). For instance, when such a synesthete observes someone being tapped on their shoulder, the synesthete involuntarily feels a tap on their own shoulder as well. People with this type of synesthesia have been shown to have higher empathy levels compared to the general population. This may be related to the so-called mirror neurons present in the motor areas of the brain, which have also been linked to empathy.[25]

### Lexical-gustatory synesthesiaEdit

This is another rare form of synesthesia where certain tastes are experienced when hearing words. For example, the word basketball might taste like waffles. The documentary ‘Derek tastes like earwax’ gets its name from this phenomenon, in references to pub owner James Wannerton who experiences this particular sensation whenever he hears the name spoken.[26][27] It is estimated that 0.2% of the population has this form of synesthesia.[28]

### Other formsEdit

Other forms of synesthesia have been reported, but little has been done to analyze them scientifically. These include associating people or emotion with colors, sounds to objects, letters to objects, and many more.

## CauseEdit

Little is known how synesthesia develops. The first studies of synesthesia in children and its development are currently undergoing.

Based on the findings that synesthesia is not a phenomenon of crossed senses but has the properties of ideasthesia, it was proposed[29] that synesthesia develops during childhood at the time at which children are for the first time intensively engaged with abstract concepts. This hypothesis—referred to as semantic vacuum hypothesis—explains why the most common forms of synesthesia are grapheme-color, spatial sequence and number form: These are usually the first abstract concepts that educational systems require children to learn.

## MechanismEdit

Regions thought to be cross-activated in grapheme-color synesthesia (green=grapheme recognition area, red=V4 color area).[30]

Dedicated regions of the brain are specialized for given functions. Increased cross-talk between regions specialized for different functions may account for the many types of synesthesia. For example, the additive experience of seeing color when looking at graphemes might be due to cross-activation of the grapheme-recognition area and the color area called V4 (see figure).[30] This is supported by the fact that grapheme-color synesthetes are able to identify the color of a grapheme in their peripheral vision even when they cannot consciously identify the shape of the grapheme.[30]

An alternate possibility is disinhibited feedback, or a reduction in the amount of inhibition along normally existing feedback pathways.[31] Normally, excitation and inhibition are balanced. However, if normal feedback were not inhibited as usual, then signals feeding back from late stages of multi-sensory processing might influence earlier stages such that tones could activate vision. Cytowic and Eagleman find support for the disinhibition idea in the so-called acquired forms[3] of synesthesia that occur in non-synesthetes under certain conditions: temporal lobe epilepsy, head trauma, stroke, and brain tumors. They also note that it can likewise occur during stages of meditation, deep concentration, sensory deprivation, or with use of psychedelics such as LSD or mescaline, and even, in some cases, marijuana.[3] However, synesthetes report that common stimulants, like caffeine and cigarettes do not affect the strength of their synesthesia, nor does alcohol.[3]:137–40

A very different theoretical approach to synesthesia is that based on ideasthesia. According to this account, synesthesia is a phenomenon mediated by the extraction of the meaning of the inducing stimulus. Thus, synesthesia may be fundamentally a semantic phenomenon. Therefore, to understand neural mechanisms of synesthesia the mechanisms of semantics and the extraction of meaning need to be understood better. This is a non-trivial issue because it is not only a question of a location in the brain at which meaning is “processed” but pertains also to the question of understanding—epitomized in e.g., the Chinese room problem. Thus, the question of the neural basis of synesthesia is deeply entrenched into the general mind–body problem and the problem of the explanatory gap.[32]

## Diagnostic criteriaEdit

Although often termed a “neurological condition,” synesthesia is not listed in either the DSM-IV or the ICD since it most often does not interfere with normal daily functioning.[33] Indeed, most synesthetes report that their experiences are neutral or even pleasant.[14] Like perfect pitch, synesthesia is simply a difference in perceptual experience.

Reaction times for answers that are congruent with a synesthete’s automatic colors are faster than those whose answers are incongruent.[3]

The simplest approach is test-retest reliability over long periods of time, using stimuli of color names, color chips, or a computer-screen color picker providing 16.7 million choices. Synesthetes consistently score around 90% on reliability of associations, even with years between tests.[1] In contrast, non-synesthetes score just 30–40%, even with only a few weeks between tests and a warning that they would be retested.[1]

The automaticity of synesthetic experience. A synesthete might perceive the left panel like the panel on the right.[30]

Grapheme-color synesthetes, as a group, share significant preferences for the color of each letter (e.g. A tends to be red; O tends to be white or black; S tends to be yellow etc.)[14] Nonetheless, there is a great variety in types of synesthesia, and within each type, individuals report differing triggers for their sensations and differing intensities of experiences. This variety means that defining synesthesia in an individual is difficult, and the majority of synesthetes are completely unaware that their experiences have a name.[14]

Neurologist Richard Cytowic identifies the following diagnostic criteria for synesthesia in his first edition book. However, the criteria are different in the second book:[1][2][3]

1. Synesthesia is involuntary and automatic.
2. Synesthetic perceptions are spatially extended, meaning they often have a sense of “location.” For example, synesthetes speak of “looking at” or “going to” a particular place to attend to the experience.
3. Synesthetic percepts are consistent and generic (i.e. simple rather than pictorial).
4. Synesthesia is highly memorable.
5. Synesthesia is laden with affect.

Cytowic’s early cases mainly included individuals whose synesthesia was frankly projected outside the body (e.g. on a “screen” in front of one’s face). Later research showed that such stark externalization occurs in a minority of synesthetes. Refining this concept, Cytowic and Eagleman differentiated between “localizers” and “non-localizers” to distinguish those synesthetes whose perceptions have a definite sense of spatial quality from those whose perceptions do not.[3]

## EpidemiologyEdit

Depending on the study, researchers have suggested 1 in 2,000 people have some form of synesthesia, while others have reported 1 in 300 or even as many as 1 in 23. One problem with statistics is that some individuals will not self-classify as they do not realize that their perceptions are different from those of everyone else.[30]

Grapheme-color, chromesthesia, or anything color-related, appear to be the most common forms of synesthesia, they have a prevalence rate of 64.4% in the synesthesia population. Some studies have found that color related grapheme can account for 86%. Time related words-colour synesthesia is the second most common with a prevalence rate of 22%-62%. Music-color is also prevalent at 18%, some studies found that music-color was shown in 41% of patients. Some of the rarest are reported to be auditory-tactile, mirror-touch, and lexical-gustatory. [34]

There is research to suggest that the likelihood of having synaesthesia is greater in people with autism.[35]

## HistoryEdit

The interest in colored hearing dates back to Greek antiquity, when philosophers asked if the color (chroia, what we now call timbre) of music was a quantifiable quality.[36] Isaac Newton proposed that musical tones and color tones shared common frequencies, as did Goethe in his book, “Theory of Color.”[citation needed] There is a long history of building color organs such as the clavier à lumières on which to perform colored music in concert halls.[37][37][38]

The first medical description of “colored hearing” is in an 1812 German thesis by the German physician Sachs.[39] The “father of psychophysics,” Gustav Fechner, reported the first empirical survey of colored letter photisms among 73 synesthetes in 1876,[40][41] followed in the 1880s by Francis Galton.[9][42][43] C.G.Jung refers to “colour hearing” in his Symbols of Transformation in 1912.[44] Research into synesthesia proceeded briskly in several countries, but due to the difficulties in measuring subjective experiences and the rise of behaviorism, which made the study of any subjective experience taboo, synesthesia faded into scientific oblivion between 1930 and 1980.

As the 1980s cognitive revolution made inquiry into internal subjective states respectable again, scientists returned to synesthesia. Led in the United States by Larry Marks and Richard Cytowic, and later in England by Simon Baron-Cohen and Jeffrey Gray, researchers explored the reality, consistency, and frequency of synesthetic experiences. In the late 1990s, the focus settled on grapheme → color synesthesia, one of the most common[14] and easily studied types. Synesthesia is now the topic of scientific books and papers, Ph.D. theses, documentary films, and even novels.

Since the rise of the Internet in the 1990s, synesthetes began contacting one another and creating web sites devoted to the condition. These early grew into international organizations such as the American Synesthesia Association, the UK Synaesthesia Association, the Belgian Synaesthesia Association, the Canadian Synesthesia Association, the German Synesthesia Association, and the Netherlands Synesthesia Web Community.

## Society and cultureEdit

### Artistic investigationsEdit

Vision by Carol Steen; Oil on Paper; 15×12-3/4″ 1996. A representation of a synesthetic photism experienced during acupuncture.
Main article: Synesthesia in art

Synesthetic art historically refers to multi-sensory experiments in the genres of visual music, music visualization, audiovisual art, abstract film, and intermedia.[11][13][45][46][47][48] Distinct from neuroscience, the concept of synesthesia in the arts is regarded as the simultaneous perception of multiple stimuli in one gestalt experience.[49]

Neurological synesthesia has been a source of inspiration for artists, composers, poets, novelists, and digital artists. Nabokov writes explicitly about synesthesia in several novels.[citation needed] Kandinsky (a synesthete) and Mondrian (not a synesthete) both experimented with image-music congruence in their paintings. Scriabin composed colored music that was deliberately contrived and based on the circle of fifths, whereas Messiaen invented a new method of composition (the modes of limited transposition) specifically to render his bi-directional sound-color synesthesia. For example, the red rocks of Bryce Canyon are depicted in his symphony Des canyons aux étoiles (“From the Canyons to the Stars”). New art movements such as literary symbolism, non-figurative art, and visual music have profited from experiments with synesthetic perception and contributed to the public awareness of synesthetic and multi-sensory ways of perceiving.[11]

Contemporary artists with synesthesia, such as Carol Steen[50] and Marcia Smilack[51] (a photographer who waits until she gets a synesthetic response from what she sees and then takes the picture), use their synesthesia to create their artwork. Brandy Gale, a Canadian visual artist, experiences an involuntary joining or crossing of any of her senses – hearing, vision, taste, touch, smell and movement. Gale paints from life rather than from photographs and by exploring the sensory panorama of each locale attempts to capture, select, and transmit these personal experiences.[52][53][54]

### Literary depictionsEdit

Synesthesia is sometimes used as a plot device or way of developing a character’s inner life. Author and synesthete Pat Duffy describes five ways in which synesthetic characters have been used in modern fiction.[55][56]

1. Synesthesia as Romantic ideal: in which the condition illustrates the Romantic ideal of transcending one’s experience of the world. Books in this category include The Gift by Vladimir Nabokov.
2. Synesthesia as pathology: in which the trait is pathological. Books in this category include The Whole World Over by Julia Glass.
3. Synesthesia as Romantic pathology: in which synesthesia is pathological but also provides an avenue to the Romantic ideal of transcending quotidian experience. Books in this category include Holly Payne’s The Sound of Blue.
4. Synesthesia as psychological health and balance: Painting Ruby Tuesday by Jane Yardley, and A Mango-Shaped Space by Wendy Mass.
5. Synesthesia as young adult literature and science fiction: Ultraviolet by R.J. Anderson, and “One Is Not A Lonely Number” by Evelyn Krieger (YM Books, 2010).

Many literary depictions of synesthesia are not accurate. Some say more about an author’s interpretation of synesthesia than the phenomenon itself.[citation needed]

### Notable casesEdit

Identifying synesthesia in the historical record is fraught with error unless (auto)biographical sources explicitly give convincing details.

There are many famous synesthetes, most of whom are artists, writers, or musicians. David Hockney perceives music as color, shape, and configuration and uses these perceptions when painting opera stage sets (though not while creating his other artworks). Russian painter Wassily Kandinsky combined four senses: color, hearing, touch, and smell.[1][3] Vladimir Nabokov described his grapheme-color synesthesia at length in his autobiography, Speak, Memory, and portrayed it in some of his characters.[57] Synesthetic composers include Duke Ellington,[58] Nikolai Rimsky-Korsakov,[59] and Olivier Messiaen, whose three types of complex colors are rendered explicitly in musical chord structures that he invented.[3][60] Physicist Richard Feynman describes his colored equations in his autobiography, What Do You Care What Other People Think?[61]

Other notable synesthetes include musicians Billy Joel,[62]:89, 91 Itzhak Perlman,[62]:53 Alexander Frey, Ida Maria,[63] Brian Chase[64][65] and Patrick Stump; actress Stephanie Carswell (credited as Stéphanie Montreux); inventor Nikola Tesla;[66] electronic musician Richard D. James aka Aphex Twin (who claims to be inspired by lucid dreams as well as music); and classical pianist Hélène Grimaud. Drummer Mickey Hart of The Grateful Dead wrote about his experiences with synaesthesia in his autobiography Drumming at the Edge of Magic.[citation needed] Pharrell Williams, of the groups The Neptunes and N.E.R.D., claims to experience synesthesia[67][68] and used it as the basis of the album Seeing Sounds. Singer/songwriter Marina and the Diamonds experiences music → color synesthesia and reports colored days of the week.[69]

Some artists frequently mentioned as synesthetes did not, in fact, have the neurological condition. Alexander Scriabin‘s 1911 Prometheus, for example, is a deliberate contrivance whose color choices are based on the circle of fifths and appear to have been taken from Madame Blavatsky.[3][70] The musical score has a separate staff marked luce whose “notes” are played on a color organ. Technical reviews appear in period volumes of Scientific American.[3] On the other hand, his older colleague Nikolai Rimsky-Korsakov (who was perceived as a fairly conservative composer) was, in fact, a synesthete.[71]

French poets Arthur Rimbaud and Charles Baudelaire wrote of synesthetic experiences, but there is no evidence they were synesthetes themselves. Baudelaire’s 1857 Correspondances introduced the notion that the senses can and should intermingle. Baudelaire participated in a hashish experiment by psychiatrist Jacques-Joseph Moreau and became interested in how the senses might affect each other.[11] Rimbaud later wrote Voyelles (1871), which was perhaps more important than Correspondances in popularizing synesthesia. He later boasted “J’inventais la couleur des voyelles!” (I invented the colors of the vowels!).[citation needed]

Daniel Tammet wrote a book on his experiences with synesthesia called Born on a Blue Day.[72]

Joanne Harris, author of Chocolat, is a synesthete who says she experiences colors as scents.[73] Her novel Blueeyedboy features various aspects of synesthesia.

## ResearchEdit

Tests like this demonstrate that people do not attach sounds to visual shapes arbitrarily. Which shape would you call “Bouba” and which “Kiki?”

Research on synesthesia raises questions about how the brain combines information from different sensory modalities, referred to as crossmodal perception or multisensory integration.

An example of this is the bouba/kiki effect. In an experiment first designed by Wolfgang Köhler, people are asked to choose which of two shapes is named bouba and which kiki. 95% to 98% of people choose kiki for the angular shape and bouba for the rounded one. Individuals on the island of Tenerife showed a similar preference between shapes called takete and maluma. Even 2.5 year-old children (too young to read) show this effect.[74] Recent research indicated that in the background of this effect may operate a form of ideasthesia.[75]

Researchers hope that the study of synesthesia will provide better understanding of consciousness and its neural correlates. In particular, synesthesia might be relevant to the philosophical problem of qualia,[4][76] given that synesthetes experience extra qualia (e.g. colored sound). An important insight for qualia research may come from the findings that synesthesia has the properties of ideasthesia,[5] which then suggest a crucial role of conceptualization processes in generating qualia.[29]

### Technological applicationsEdit

Synesthesia also has a number of practical applications, one of which is the use of ‘intentional synesthesia’ in technology.[77]

#### Synesthesia and virtual realityEdit

One type of application is the pain-reducing virtual reality program.[78] In existing programs, the main purpose is to reduce pain when undergoing a specific treatment by shifting the attention from the experienced pain to the virtual program in which the patient is participating. By using artificial synesthesia and combining various senses, this can help to enhance the control of a person’s attention, which can be used to improve and direct sensory distraction from the perceived pain.

For example, many treatments for burn pain and wounds may increase patients’ anxiety, which increases perceived pain. Shifting attention from pain and anxiety is therefore an important part of the treatment process.[79] Virtual reality has proven to be very effective in managing this acute pain in several medical settings by shifting patients’ attention from their experienced pain to the program in which they have been introduced. It appears to be far more effective than other distraction techniques, like playing video games.[80] More specifically, the convergence of many sense modalities (e.g. sound, sight, and touch) gives patients the perception of being immersed in the virtual environment, which helps them endure the pain while relying less on pharmacological therapy.

#### The VoiceEdit

Peter Meijer developed a sensory substitution device called The vOICe (the capital letters “O,” “I,” and “C” in “vOICe” are intended to evoke the expression “Oh I see”). The vOICe is a privately owned research project, running without venture capital, that was first implemented using low-cost hardware in 1991.[81] The vOICe is a visual-to-auditory sensory substitution device (SSD) preserving visual detail at high resolution (up to 25,344 pixels).[82] The device consists of a laptop, head-mounted camera or computer camera, and headphones. The vOICe converts visual stimuli of the surroundings captured by the camera into corresponding aural representations (soundscapes) delivered to the user through headphones at a default rate of one soundscape per second. Each soundscape is a left-to-right scan, with height represented by pitch, and brightness by loudness.[83] Default resolution of the soundscape is 176×64. Therefore, it is roughly comparable to a retinal implant or brain implant with 10,000 electrodes.

The process of converting greyscale camera images into soundscapes works according to three simple rules. The first is ‘left and right’ in which left-to-right scanning results in hearing the stereo pan from left to right correspondingly. If there is a visual pattern on the left, the user hears a sound on the left, and similarly for the right. The second rule is ‘up and down’: every scan provides a pitch that indicates elevation. The higher the position of the visual pattern, the higher the pitch. The third and final rule is ‘dark and light’: loudness corresponds to brightness. The louder the sound, the brighter the visual pattern. Silence indicates no light stimuli, the loudest sounds represent white light, and everything in between is a shade of grey.

For example, a straight bright line on a dark background, running from the top left to the bottom right, would sound like a tone steadily decreasing in pitch; a dot would sound like a short beep; and two dots would sound like two short beeps. Since real-life images are much more complex, there is also much more to hear through this device. While converting the visual pattern into a sound, the device uses a predictable real-time audio and video processing algorithm, allowing users to listen to and then interpret the visual information captured by a digital video camera. The vOICe compensates for the loss of vision by converting information from the lost sensory modality into stimuli in a remaining modality.[84]

This could lead to synthetic vision with truly visual sensations through crossmodal sensory integration through training and education. It requires a certain amount of time and effort to become proficient at differentiating objects, identifying objects, and locating them in space. Users are therefore advised to start training in a safe, familiar home environment in order to integrate the novel stimuli with other senses.

One of the remaining questions in this ongoing research concerning the vOICe is to what extent the use of a sensory substitution system can lead to visual sensations through forms of induced artificial synesthesia.

#### EyeborgEdit

The Eyeborg is a device developed by Adam Montandan that incorporates the auditory and visual spectra. It makes it possible for people with color-blindness to hear colors. This device was inspired by naturally occurring synesthesia.[85]

## ReferencesEdit

1. [page needed] Cytowic, Richard E. (2002). Synesthesia: A Union of the Senses (2nd edition). Cambridge, Massachusetts: MIT Press. ISBN 0-262-03296-1. OCLC 49395033.
2. ^ a b [page needed] Cytowic, Richard E. (2003). The Man Who Tasted Shapes. Cambridge, Massachusetts: MIT Press. ISBN 0-262-53255-7. OCLC 53186027.
3. [page needed] Cytowic, Richard E; Eagleman, David M (2009). Wednesday is Indigo Blue: Discovering the Brain of Synesthesia (with an afterword by Dmitri Nabokov). Cambridge: MIT Press. ISBN 0-262-01279-0.
4. ^ a b [page needed] Harrison, John E.; Simon Baron-Cohen (1996). Synaesthesia: classic and contemporary readings. Oxford: Blackwell Publishing. ISBN 0-631-19764-8. OCLC 59664610.
5. ^ a b Nikolić D (2009). “Is synaesthesia actually ideaesthesia? An inquiry into the nature of the phenomenon” (PDF). Proceedings of the Third International Congress on Synaesthesia, Science & Art, Granada, Spain, April 26–29.
6. ^ Simner J (2012). “Defining synaesthesia”. British Journal of Psychology (Review) 103 (6): 1–15. doi:10.1348/000712610X528305. PMID 22229768.
7. ^ Rich AN, Mattingley JB (January 2002). “Anomalous perception in synesthesia: a cognitive neuroscience perspective”. Nature Reviews Neuroscience (Review) 3 (1): 43–52. doi:10.1038/nrn702. PMID 11823804.
8. ^ a b Hubbard EM, Ramachandran VS (November 2005). “Neurocognitive mechanisms of synesthesia” (PDF). Neuron (Review) 48 (3): 509–20. doi:10.1016/j.neuron.2005.10.012. PMID 16269367.
9. ^ a b c Galton F (1880). “Visualized Numerals”. Nature 21 (543): 494–5. doi:10.1038/021494e0.
10. ^ Seron X, Pesenti M, Noël MP, Deloche G, Cornet JA (August 1992). “Images of numbers, or “When 98 is upper left and 6 sky blue””. Cognition 44 (1–2): 159–96. doi:10.1016/0010-0277(92)90053-K. PMID 1511585.
11. [page needed] van Campen, Cretien (2007). The Hidden Sense: Synesthesia in Art and Science. Cambridge, Massachusetts: MIT Press. ISBN 0-262-22081-4. OCLC 80179991.
12. ^ a b Campen, Cretien van (2009) “The Hidden Sense: On Becoming Aware of Synesthesia” TECCOGS, vol. 1, pp. 1-13.[1]
13. ^ a b Campen C (1999). “Artistic and psychological experiments with synesthesia”. Leonardo 32 (1): 9–14. doi:10.1162/002409499552948.
14. [page needed] Sagiv, Noam; Robertson, Lynn C (2005). Synesthesia: perspectives from cognitive neuroscience. Oxford: Oxford University Press. ISBN 0-19-516623-X. OCLC 53020292.
15. ^ [page needed] Flournoy, Théodore (2001). Des phénomènes de synopsie (Audition colorée). Adamant Media Corporation. ISBN 0-543-94462-X.
16. ^ a b [broken citation] Dittmar, A. (Ed.) (2007) Synästhesien. Roter Faden durchs Leben? Essen, Verlag Die Blaue Eule.
17. ^ Dailey A, Martindale C, Borkum J (1997). “Creativity, synesthesia, and physiognomic perception”. Creativity Research Journal 10 (1): 1–8. doi:10.1207/s15326934crj1001_1.
18. ^ [unreliable source?] “Slashdot Discussion”. 2006-02-19. Retrieved 2006-08-14.
19. ^ [unreliable medical source?]Do sequence-space synaesthetes have better spatial imagery skills? Maybe not, The National Center for Biotechnology Information
20. ^ [unreliable source?] A Mind That Touches the Past, Sciencemag.org
21. ^ Galton F (1881). “The visions of sane persons” (PDF). Fortnightly Review 29: 729–40. Retrieved 2008-06-17.
22. ^ Naumer MJ, van den Bosch JJ (July 2009). “Touching sounds: thalamocortical plasticity and the neural basis of multisensory integration”. J. Neurophysiol. 102 (1): 7–8. doi:10.1152/jn.00209.2009. PMID 19403745.
23. ^ [page needed] Cytowic, Richard E. (2002). Synesthesia: A Union of the Senses (2nd edition). Cambridge, Massachusetts: MIT Press. ISBN 0-262-03296-1. OCLC 49395033
24. ^ Edelstein, Miren, David Brang, Romke Rouw, and Vilayanur S. Ramachandran. “Misophonia: Physiological Investigations and Case Descriptions.” Frontiers in Human Neuroscience 7 (2013): n. pag. National Center for Biotechnology Information. US National Library of Medicine, 25 June 2013. Web. 5 Dec. 2013.
25. ^ Heyes, Cecelia. “Where Do Mirror Neurons Come From?” Neuroscience and Biobehavioral Reviews (2009): 576-77. CognitiveScience. University of Oxford, 7 Nov. 2009. Web. 30 Jan. 2015.
26. ^ “Derek Tastes of Ear Wax”. Top Documentary Films. Retrieved 2 February 2015.
27. ^ “BBC – Science & Nature – Horizon”. bbc.co.uk. Retrieved 2 February 2015.
28. ^ Simner, Julia (2009). Encyclopedia of Neuroscience. Springer Berlin Heidelberg. ISBN 978-3-540-23735-8.
29. ^ a b Mroczko-Wąsowicz, A., Nikolić D. (2014) Semantic mechanisms may be responsible for developing synesthesia. Frontiers in Human Neuroscience 8:509. doi: 10.3389/fnhum.2014.00509
30. [non-primary source needed][dead link]Ramachandran VS and Hubbard EM (2001). “Synaesthesia: A window into perception, thought and language” (PDF). Journal of Consciousness Studies 8 (12). pp. 3–34.
31. ^ Grossenbacher PG, Lovelace CT (January 2001). “Mechanisms of synesthesia: cognitive and physiological constraints”. Trends Cogn. Sci. 5 (1): 36–41. doi:10.1016/S1364-6613(00)01571-0. PMID 11164734.
32. ^ Jeffrey A. Gray, David M. Parslow, Michael J. Brammer, Susan Chopping, Goparlen N. Vythelingum, Dominic H. Fytche. “Evidence Against Functionalism from Neuroimaging of the Alien Colour Effect in Synaesthesia.” Cortex. Volume 42, Issue 2, 2006. pg. 317. 15, Feb. 2015
33. ^ Hubbard, Edward (June 2007). “Neurophysiology of synesthesia”. NCBI. PubMed. Retrieved 15 Feb 2015.
34. ^ Safran, Avinoam; Sanda, Nicolae (24 Dec 2014). “Colour synesthesia. Insight into perception, emotion, and consciousness”. Current Opinion In Neurology 28 (1): 36-44. doi:10.1097/WCO.0000000000000169. PMID 25545055. Retrieved 5 August 2015.
35. ^ Baron-Cohen S, Johnson D, Asher J, Wheelwright S, Fisher SE, Gregerson PK, Allison C, “Is synaesthesia more common in autism?”, Molecular Autism, Nov 20 2013
36. ^ [page needed] Gage, J.Colour and Culture. Practice and Meaning from Antiquity to Abstraction. (London:Thames & Hudson, 1993).
37. ^ a b Peacock, Kenneth. “Instruments to Perform Color-Music: Two Centuries of Technological Experimentation,”Leonardo 21, No. 4 (1988) 397–406.
38. ^ [page needed] Jewanski, J. & N. Sidler (Eds.). Farbe – Licht – Musik. Synaesthesie und Farblichtmusik. Bern: Peter Lang, 2006.
39. ^ Mahling, F. (1926) Das Problem der `audition colorée’: Eine historisch-kritische Untersuchung. Archiv für die gesamte Psychologie, 57, 165–301.
40. ^ Fechner, G. (1876) Vorschule der Aesthetik. Leipzig: Breitkopf und Hartel. Website: [2]
41. ^ Campen, Cretien van (1996). De verwarring der zintuigen. Artistieke en psychologische experimenten met synesthesie. Psychologie & Maatschappij, vol. 20, nr. 1, pp. 10–26.
42. ^ Galton F (1880). “Visualized Numerals”. Nature 21 (533): 252–6. doi:10.1038/021252a0.
43. ^ [page needed] Galton F (1883). Inquiries into Human Faculty and Its Development. Macmillan. Retrieved 2008-06-17.
44. ^ Jung, C.G. The Tansformation of Libido in “Symbols of Transformation“, CW5, London 1912/1956, Routledge & Kegan Paul, para.237.
45. ^ Berman G (1999). “Synesthesia and the Arts”. Leonardo 32 (1): 15–22. doi:10.1162/002409499552957.
46. ^ [page needed] Maur, Karin von (1999). The Sound of Painting: Music in Modern Art (Pegasus Library). Munich: Prestel. ISBN 3-7913-2082-3.
47. ^ [page needed] Gage, John D. (1993). Colour and culture: practice and meaning from antiquity to abstraction. London: Thames and Hudson. ISBN 0-500-27818-0.
48. ^ [page needed] Gage, John D. (1999). Color and meaning: art, science, and symbolism. Berkeley: University of California Press. ISBN 0-520-22611-9.
49. ^ [page needed] Campen, Cretien van (2009) Visual Music and Musical Paintings. The Quest for Synesthesia in the Arts. In: F. Bacci & D. Melcher. Making Sense of Art, making Art of Sense. Oxford: Oxford University Press.
50. ^ Steen, C. (2001). Visions Shared: A Firsthand Look into Synesthesia and Art, Leonardo, Vol. 34, No. 3, Pages 203–208 doi:10.1162/002409401750286949
51. ^ Marcia Smilack Website Accessed 20 Aug 2006.
52. ^ “Coastal Synaesthesia: Paintings and Photographs of Hawaii, Fiji and California by Brandy Gale – Gualala Arts Center exhibit: January, 2015”. gualalaarts.org. Retrieved 2 February 2015.
53. ^ EG. “The Wondrous Sensory Spectrum of Brandy Gale”. FORA.tv. Retrieved 2 February 2015.
54. ^ brandygale.com
55. ^ Duffy, P.L. (2006). Images of Synesthetes and their Perceptions of Language in Fiction. 6th Annual Meeting of the American Synesthesia Association. University of South Florida.
56. ^ Duffy PL, Simner J (2010). “Synaesthesia in fiction”. Cortex 46 (2): 277–278. doi:10.1016/j.cortex.2008.11.003. PMID 19081086.
57. ^ [page needed] Nabokov, Vladimir. 1966. Speak, Memory: An Autobiography Revisited. New York: Putnam.
58. ^ [page needed] Ellington, as quoted in George, Don. 1981. Sweet man: The real Duke Ellington. New York: G.P. Putnam’s Sons. Page 226.
59. ^ according to the Russian press: Yastrebtsev V. “On N.A.Rimsky-Korsakov’s color sound- contemplation.” Russkaya muzykalnaya gazeta, 1908, N 39–40, p. 842–845 (in Russian), cited by Bulat Galeyev (1999).
60. ^ see Samuel, Claude. 1994 (1986). Olivier Messiaen: Music and Color. Conversations with Claude Samuel. Translated by E. Thomas Glasow. Portland, Oregon: Amadeus Press.
61. ^ [page needed] Feynman, Richard. 1988. What Do You Care What Other People Think? New York: Norton. P. 59.
62. ^ a b Seaberg, M. (2011). Tasting the Universe. New Page Books. ISBN 978-1-60163-159-6.
63. ^ Cairns, Dan (2008-02-24). “Times Online interview”. The Times (London). Retrieved 2008-07-24.
64. ^ Forrest, Emma (March 30, 2009). “Emma Forrest meets New York’s favourite art-punk rockers Yeah Yeah Yeahs”. guardian.co.uk (London: The Guardian). Retrieved 2009-05-07.
66. ^ Tesla, Nikola. “The Strange Life of Nikola Tesla” (PDF). pitt.edu. Retrieved 4 September 2012.
67. ^ [unreliable source?] It just always stuck out in my mind, and I could always see it. I don’t know if that makes sense, but I could always visualize what I was hearing… Yeah, it was always like weird colors.” From a Nightline interview with Pharrell
68. ^ “Synesthetes: “People of the Future””. Psychology Today. March 3, 2012. Retrieved May 15, 2014.
69. ^ Loose Women | Marina and the Diamonds – ITV Lifestyle ITV – 27 April 2010 – Retrieved 28 April 2010.
70. ^ [page needed] Dann, Kevin T. (1998). Bright colors falsely seen: synaesthesia and the search for transcendental knowledge. New Haven, Conn: Yale University Press. ISBN 0-300-06619-8.
71. ^ This is according to an article in the Russian press, Yastrebtsev V. “On N.A.Rimsky-Korsakov’s color sound- contemplation.” Russkaya muzykalnaya gazeta, 1908, N 39-40, pp. 842–845 (in Russian), cited by Bulat Galeyev (1999).
72. ^ [page needed] Tammet, Daniel (2007). Born on a Blue Day. Free Press. ISBN 978-1416535072.
73. ^ “Chocolat author Joanne Harris talks about her latest novel Blue Eyed Boy”. Metro. 7 Apr 2010.
74. ^ [non-primary source needed] Maurer D, Pathman T, Mondloch CJ (May 2006). “The shape of boubas: sound-shape correspondences in toddlers and adults”. Dev Sci 9 (3): 316–22. doi:10.1111/j.1467-7687.2006.00495.x. PMID 16669803.
75. ^ Gómez Milán E., Iborra O., de Córdoba M.J., Juárez-Ramos V., Artacho Rodríguez, Rubio J.L. (2013). “The Kiki-Bouba effect: A case of personification and ideaesthesia”. The Journal of Consciousness Studies 20 (1-2): 84–102.
76. ^ Gray JA, Chopping S, Nunn J et al. (2002). “Implications of synaesthesia for functionalism: Theory and experiments”. Journal of Consciousness 9 (12): 5–31.
77. ^ Suslick, Kenneth S (December 2012). “Synesthesia in science and technology: more than making the unseen visible”. Current Opinion in Chemical Biology 16 (5-6): 557–563. doi:10.1016/j.cbpa.2012.10.030. PMC 3606019. PMID 23183411.
78. ^ Reif, John. “Advancing Attention Control Using VR-Induced Artificial Synesthesia” (PDF). Retrieved 4 February 2014.
79. ^ Hoffman, Hunter G.; Doctor, Jason N.; Patterson, David R.; Carrougher, Gretchen J.; Furness III, Thomas A. (March 2000). “Virtual reality as an adjunctive pain control during burn wound care in adolescent patients”. Pain 85 (1-2): 305–309. doi:10.1016/S0304-3959(99)00275-4.
80. ^ Gold, Jeffrey I.; Belmont, Katharine A.; Thomas, David A. (August 2007). “The Neurobiology of Virtual Reality Pain Attenuation”. CyberPsychology & Behavior 10 (4): 536–544. doi:10.1089/cpb.2007.9993.
81. ^ Meijer, Peter. “Augmented Reality for the Totally Blind”. Retrieved 4 February 2014.
82. ^ Striem-Amit, Ella; Guendelman, Miriam; Amedi, Amir; Serino, Andrea (16 March 2012). “‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution”. PLoS ONE 7 (3): e33136. doi:10.1371/journal.pone.0033136.
83. ^ Carmichael, Joey. “Device Trains Blind People To ‘See’ By Listening”. Retrieved 4 February 2014.
84. ^ Haigh, Alastair; Brown, David J.; Meijer, Peter; Proulx, Michael J. (2013). “How well do you see what you hear? The acuity of visual-to-auditory sensory substitution”. Frontiers in Psychology 4. doi:10.3389/fpsyg.2013.00330.
85. ^ Montandon, Adam. “Colourblind Eyeborg Colours to Sound”. Retrieved 4 February 2014.