“Before the development of more powerful telescopes, early astronomer Franz von Gruithuisen [March 19, 1774 – June 21, 1852] believed that Ashen light was from the fires from celebration of a new Venusian emperor, and later believed that it was the inhabitants burning vegetation to make room for farmland.”

Nice theories, don’t you think? It was likely the best speculation of its time, but just consider how human culture centric those thoughts really were.

1. “fires from celebration of a new Venusian emperor” = Venus has an emperor – implying a hierarchical society – celebrations are conducted on a primitive fashion through the lighting of massive planet-wide fires.

2. “inhabitants burning vegetation to make room for farmland” = there is a lot of vegetation on Venus, enough so that it needs to be burnt on a massive scale in order to conduct agriculture.

Basically these thoughts mirrored the current sociological-technological-philosophical environment surrounding von Gruithuisen and were projected into an alien environment. The tacit assumption seemed to be that the sociological-technological-philosophical status quo where von Gruithuisen was living in at the time (hierarchical society, all hail the leader, dependence on agriculture, etc.) was the most natural state of things and therefore it was reasonable to think that alien places, even civilizations on other planets, would follow this model.

Let’s extend these thoughts to SETI. What are we trying to do with SETI? We’re trying to pick up (radio) signals of an alien civilization.

Not withstanding arguments on the necessarily narrow time window when we could pick up anything in the first place, notice that there are many assumptions we’re making: radio signals are used, the “water hole” is preferred, potential willingness for the alien race to actively attempt contact, supposition that we could detect and distinguish an artificial signal from a natural one, etc.

What if we’re going the way of Baron von Gruithuisen here? What if what we’re trying to find is simply so alien, that we cannot comprehend it based on our sociological/technological/philosophical tradition and background? What if we’re trying to find traces of a hive mind of superintelligent translucent slime who thinks since they communicate by clanging on pipes of ice under liquid methane oceans, others must surely do the same?

My questions are not meant to imply that SETI is a waste of time and that we should stop it – on the contrary!

What I am saying is we should remember Baron von Gruithuisen, and try to think outside of the box, through de-assumptionizing (does this word win the Scrabble?) and re-thinking the “model of the space alien”, since we really have no good reason to assume anything specific in that area.

Here’s a cleaned up mail I sent, with some links added. What do you think?

There is a major problem with HFTs and fast algorithmic trading and markets due to full computerization (no human in the loop) and the massive speed obtained through those methods. This time, I don’t mean the obvious application of HFTs doing front-running.

Here I use the definition “HFT agents” to be any instance of an HFT/algorithmic trading algorithm running somewhere. Basically a computer reading and writing to the market.

So: I see the dynamics such that HFT agents both observe and influence the market. Yet they are part of that market as well. As they trade, they also influence each other (although indirectly)! And they do trading at such speed, that there is bound to be kind of “resonances” amplifying whatever direction was taken originally – sort of like everyone nudging the same direction a little bit. I think that this kind of system must be unstable.

Human traders can’t jump in and fix up the situation as they can’t keep up with the speed. So, in essence we get either flash bubbles or flash crashes. All this because the HFT agents actions influence each other, they end up modifying each other’s algorithms through application of stigmergy when they modify the market (=they modify the environment they’re in and read their own and others modifications as input to their algorithms).

It’s like the synchronizing of fireflies (see the video and read Strogatz’s book “Sync” for details), they adapt to each other until everyone blinks in unison. Except here the synchrony is a kind of “stochastic resonance” leading to spikes + valleys.

With slower trading the quick peaks and valleys even out, of course we still get bubbles and crashes over a longer period. But the same bubble/crash-creating phenomenon is at play, this time much quicker.

One example follows. There are surely other ways also, given that there is a big diversity in the algorithms.

• HFT agents x and y scans the market. Some condition c holds (e.g. break in 30 day SMA).
• HFT agent x triggers first and does whatever (sell/buy orders etc.), changing the market conditions.
• HFT agent y might do the same action immediately after x, reacting to condition c or action of x (stigmergy comes into play). This “boosts” the direction, be it up or down.
• Now imagine thousands of these agents. Some might react to the amplified direction signal (e.g. drop in price) further amplifying the system.
• Result: flash crash / flash bubble

As long as there would be an overall balance of HFT trading strategies such that some are bearish and some are bullish, we would not see nearly as much volatility. But as it is now, the balance is tilted and therefore I think insane volatility is inevitable.

Note: Sometime after the mail exchange, the Zero Hedge blog blogged about a paper from Reginald Smith examining similar ideas. It is a very interesting article that makes the paper more accessible.

-”Yo. Why is reality so hard?”, he asked.
-”I’ve no idea. Why is reality so hard?”, I replied.
-”It’s because there’s too many things to keep track of. I have a solution: micro-feng shui.”
-”Micro-feng shui? But isn’t micro less?”
-”Everything micro is good. For example: microchips. Very good.”
-”OK, so how does it work?”
-”Step 1: reduce spatial complexity. From now on, view the world as a planar space and ignore all height differences. That’s 30% less complexity. Then, step 2: happiness.”

Considering what’s published nowadays in various self-help books, packaging this idea into a 400-page book would probably reach New York Times bestseller list very fast. Not to mention the creation of some sort of a pseudo-religious movement, enabling my friend to be showered in non-Zimbabwean paper currency.

MP3 uses a form of lossy compression. This means that the original sound can not be perfectly recovered. What is recovered (sound you hear) is some kind of approximation of the original.

MP3 does this lossy compression by removing the sounds which a human ear cannot distinguish, using so-called “perceptual coding“. The sounds which are removed are specified by a psychoacoustic model, which is created using human input, for example listening tests.

In other words, this means that many people are all the time listening to music with “something missing”. But what is this something, what does this something sound like?

I woke up this Sunday morning and thought to myself, now there’s an interesting question! So, I set about to find out the lost sound.

The procedure to do this is quite simple: take an uncompressed original sound (e.g. WAV format with linear PCM encoding); call this a, encode it to MP3, decode it back to linear PCM; call this b. Then, subtract b from a. The result is the signal which MP3 encoding loses forever.

My test track was a Cuban hip-hop track “300 Kilos (Featuring Yuri Buenaventura)” by Orishas, from the album Emigrante. I chose this track because it’s has a variety of different sounds: human speech, high-hats, bass, etc. and besides, it’s kind of dope.

I did the test with two quality settings: 128 kbits per second, and with a variable bit rate (VBR). I used the Lame encoder, using the command line options “-b 128 -h” and “-V 3″, respectively.

I have provided below short 20-second test clips. In order to save space, I’ve encoded the test clips back to MP3 format. The sound quality was not altered in a very perceivable way for the new MP3 encoding (as MP3 encoding was truly doing its job!), the point is to demonstrate what this missing audio sounds like.

• Original clip, high-quality VBR MP3 (lame -V 5)
• Sounds lost when MP3 encoding with 128 kbps (lame -b 128 -h)
• Sounds lost when MP3 encoding with VBR (lame -V 3)

As you can hear, the better the quality of the MP3 encoding, the more silent the lost sound is. This makes sense. But note, even with the best setting you can get (not included in examples), MP3 is lossy – at least some of the original audio is lost.

If you want to create a digital collection of your own CDs, using formats such as FLAC is the way to go. FLAC is lossless, and therefore you can recover the original sound exactly. By having your files as FLAC, you can later decode the FLAC to linear PCM, and re-encode to whatever format is handy.

My test audience of one person thought the missing audio sounded very “satanic”, something like in 1970-1980s Hollywood horror movies. From this observation we can present a follow-up question: does MP3 encoding remove the evil from your audio files? If this hypothesis holds, then encoding reversed tracks of Led Zeppelin should produce silence or near-silence. Testing this hypothesis shall be left for future research.

Then, in a similar vein, I ran into this BBC article, which states: “Analysis of notes from a selection of Spain’s major cities showed that each one carried an average of 25.18 micrograms of cocaine.”

In other words, people snort so much of the illicit substance through rolled-up notes that the notes in circulation will have measurable amounts of cocaine in them. It’s unbelievable, but true.

Since people dig huge holes to the ground to get out minerals worth less, would this resource be somehow commercially exploitable?

Let’s do the math!

Assume you are in control of a shop selling relatively inexpensive goods at some busy area. Let’s assume, on average, there are 100 customers daily, each of who bring (on average) 2 notes to the shop. This means 100*2*25.18 micrograms = 0.005 grams of cocaine per day. The shop is run year-round, so let’s say about 330 days per year – this gives about 1.67 grams of cocaine per year per shop.

If you are a big franchise – similar to Subway, McDonald’s, Pizza Hut, whatever, and you’ll have, say, 1000 of specially equipped shops to “farm” or perhaps rather “wash” the cocaine out of the notes, you would be able to pull in a hefty 1.6 kg of cocaine every year, perhaps just in time for the company Christmas party.

Anyway, 1.6 kg may seem like a lot and I agree, it does. But you can fudge the numbers as you see fit, and see what the result is for you. Here’s the formula – similar to Drake Equation, but doesn’t involve little green men:

X * N * C * d * S

X = customers per day, on average
N = notes brought in by customers per day, on average
C = amount of cocaine in each note, on average, in grams
d = days in a year the shop operates
S = number of shops

In the example above, we’d get 100 * 2 * 0.00002518 * 330 * 1000 = 1662 (grams).

Here is the Gartner Group’s list in a nutshell:

• Multicore and hybrid processors
• Virtualisation and fabric computing
• Social networks and social software
• Cloud computing and cloud/Web platforms
• Web mashups
• User Interface
• Ubiquitous computing
• Contextual computing
• Augmented reality
• Semantics

The list seems to be mirroring the biggest trends which are going on already today. So I don’t consider the list so much as any kind of real prediction, but more a validation of how, according to Gartner, things will continue to grow (especially in the business setting).

I’m sure Gartner had to prune their original lists to get down to just ten. If I could add to their list, I would add the following.

Ultramobile computers – in a few years people will wonder how we could haul those monstrously-sized laptop computers with us. The more mobile computers are, the more people will be able to take their internet-connected laptops along truly wherever they go. Expect to see some very interesting progress on this front.

The open revolution – more and more hardware vendors will not try to hinder hobbyists from modifying the hardware or the software. Non-hobbyist consumers will be able to use the device as COTS devices for their intended purposes. Hobbyists will be able to expand the devices in new, unforeseen and innovative ways.

This will result in new applications and modifications, as the hobbyists scratch their personal itches to fullfill their bright ideas which would otherwise never see the daylight due to corporate inefficiencies in communication and decision-making, corporate bureocracy and the short-term profit motive thinking.

Smart companies can benefit from this open revolution, since if the hardware vendors allow selling third-party modifications as a service without harassing the modifiers, the hardware vendors will simply sell more devices without additional risk to themselves. The downside for the hardware vendors is the loss of control over how the device is used, and of course a slight dislocation in the value chain.

Brain-Computer Interfaces (BCI) – already now, there are consumer-grade EEG devices out on the market, for example the OCZ Neural Impulse Actuator. The Emotiv EPOC is following later this year. BCI devices are input devices which measure brainwave activity and (facial) muscle activation and interpret these signals to a form understood by computers. Essentially with these devices one can control the computer with “thinking” alone. There is a small learning curve, of course. The possibilities with these kind of interfaces are absolutely huge in many areas: home, office, military, medical, etc.

Sidenote: Currently, in a laboratory setting, it is possible to also alter the brain functionality in a non-permanent way with strong magnetic fields (so-called Transcranial Magnetic Stimulation, or TMS). When BCI devices become more ubiquitous it will naturally cause more research into devices which can be used to also “send” information directly to the brain (by exciting neurons in certain areas), i.e. consumer-grade TMS. This will bring the world closer to the direct neural jacks as popularized by William Gibson’s cyberpunk novels. I estimate this will happen in maybe 20 years, so don’t hold your breath.

There’s probably more I could come up with, but I think now it’s time to wrap this up. I might tell you, if you paid me as much as Gartner gets from each report ;)

Consider the following CAPTCHA image:

A human can easily see what the words are (in the example “Orange” and “Apple). But for a computer doing OCR, separating apples from oranges is not so easy, especially if the words are of the same color and overlap into weird non-character forms.

In principle the multivalued CAPTCHA picture should appear as if it were written with an unknown script. Think an English OCR program attempting to recognize Chinese, a Russian OCR attempting to recognize Arabic, and so on.

Note, if the words were partially transparently blended together, i.e. 50% grey + 50% grey, the places where the words intersect would be darker. Thus I would imagine one could separate the words more easily (upper part of the intersection belongs to the word stacked to the bottom part of picture, and vice-versa for the word at the upper part).

So, given a CAPTCHA image, there would be a maximum of four input boxes where to type all words which appear in the picture. The picture itself could have any number of overlapping words, from two to four. This would add an additional level of difficulty to the OCR algorithm; it would not even know how many words there are in the picture.

There is a CAPTCHA improvement to group different symbols (=characters of a word) more closely together, but as far as I know not this kind of multivalued CAPTCHA as presented here… If anyone knows or has seen such a thing, please post a comment.

The Drake Equation attempts to give an estimate to the number of intelligent civilizations in our galaxy which we might be able to contact. The Fermi Paradox provides a line of reasoning which implicitly gives a (high) number of such civilizations, but also notes that we’ve not found anyone else yet. And that is the core of the Fermi Paradox: there should be many others out there, so why don’t we see even traces?

If I were to explain the Fermi Paradox, I would do it like this: the presence of organic molecules in distant moons, planet(s), comets, nebulae, etc. speaks of the ubiquity of the building blocks of life in the known universe. Therefore it is likely that:

1. Life has emerged and will emerge outside Earth.
2. The process of evolution is a strong fundamental property of systems which contain agents capable of adapting to their environment, regardless of the environment being real or simulated. This property surely exists elsewhere outside Earth – therefore evolution surely is driving development elsewhere as well.
3. The presence of life together with evolution will, in time, evolve a collective of life-forms.
4. The presence of evolution working with a collective of life-forms will, in time, form collectives of higher organization, i.e. an intelligent civilization.

There might not be many civilizations which reach this level of intelligence, but it is bound to happen at some point in time (perhaps not simultaneously with another civilization!). As there are, due to the ubiquity of organic molecules, many possibilities where the whole process from organic molecules to intelligent civilizations might kickstart itself, we can assume that intelligent life could have emerged before us and can emerge after us. However, we could still be alone now.

In the case of previous intelligent civilizations, we might be able to pick up some transmissions (although this is not strictly necessary). This is what various forms of SETI aim to find out – to detect communications traces from the other civilizations. However, so far there has been no documented success.

Also, if we were the last of the intelligent civilizations, we should definitely be picking up some transmissions. However, SETI projects have not succeeded in this either.

Therefore, it seems strange that we have not actually detected any intelligent alien civilizations. This is a re-formulation of the Fermi Paradox.

Putting all these together, there can be only one conclusion: we are the FIRST intelligent civilization of the (known) universe. Every other civilization which follows us will be younger than us (maybe somehow even our descendants).

In other words, are the pioneers, the first ones, the ones those who come after us might some day call the Ancients. We’re the ones showing the way; the adults to the children. In all the science fiction stories, the mysterious “first ones” who created weird and fascinating technology for numerous deus ex machina moments – that would be us, the humans.

I find this an exhilarating thought. It puts things into perspective and gives the humankind a tremendous responsibility, but also a practical purpose and “meaning of life” as something concrete instead of the meanings dictated by visions of the divine.

One use for CAPTCHAs is to prevent spamming with the idea that anything automated (such as mass-spamming computers) won’t pass the puzzle, and therefore will not be able to inject their trash-messages into a service. However, CAPTCHAs don’t work so well in practice anymore.

As demonstrated by the recent break of Gmail CAPTCHA, even really good CAPTCHA systems are vulnerable to a “mechanical turk” attack. This is an attack where, instead of computers attempting to solve the problem, the attacker “outsources” the solving to a group of real people.

For example, one way to conduct such an attack is the following: to enter a porn site, one must solve a CAPTCHA puzzle. This puzzle is actually one which comes from a valid service, and is reproduced at the porn site. People who want to see porn solve the CAPTCHA, thus giving the attacker a proper and validated answer to the presented puzzle. (Of course the system relies on the people giving a correct answer)

An alternative way is to just pay the people to do such work. Generally, if there is some incentive which the attacker can provide to people, it is possible to employ a number of people for solving the CAPTCHAs without them possibly knowing the real use of the solved puzzles. This kind of attack is very hard to beat.

However, there is at least a way to make the problem harder simply by enforcing a strict time window in which the CAPTCHA must be solved. In a nutshell, the server must create a random ID and remember it along with a timestamp, and present the ID with the CAPTCHA. When the user solves it, he provides the ID to the server and the server notes the timestamp (from its own clock) when the solution (along with the ID) is received. The server can now look up the original time, based on the received ID. With knowledge of the current time, the server can deduce the amount of time it took for the user to answer. If the solution has not been given within a certain time window, the solution is rejected (even if it was correct).

For example, if a user has 20 seconds time to solve the CAPTCHA, it will give a very tight time window for an outsourcing/mechanical turk attack. The act of spamming and logging into a porn site would need to coincide within the 20 seconds, minus the time needed to solve the puzzle, minus the time for network latency. Surely, with today’s traffic amounts, obtaining such simultaneousness is possible, especially if a single spammer controls a large number of sites which can aid the mechanical turk attack. However, the amount of such opportunities would be less than now, when the CAPTCHA is not tied to time. As the amount of opportunities lessens, the amount of spam resulting from successful attacks would be cut down.

Another aspect of the time windowed CAPTCHA is that it raises the requirements for efficient logistics when moving the puzzle to the group of people. This translates to increased costs, which is undesired by the attacker.

The presented idea is not perfect by any means: it would still not solve the mechanical turk attacks where real people are coupled with the spamming system to solve the puzzles in real- or near-realtime.

Further Notes

There are some practical issues with the time-window approach. Obviously to prevent a denial of service the old, unused ID to timestamp mappings would need to be cleared from the database periodically.

Also, for practical reasons, the authentication would need to be two-step: the first step where a user writes a message (or fills in the user information when registering, or whatever the CAPTCHA puzzle protects), the second step to display and process the CAPTCHA. This is because a user can spend a lot of time actually writing the information, and that should not be penalized. Besides, proof of having solved the puzzle is not needed until the user actually attempts to submit the information.