Archive for December, 2009

Love is when the objective of the species overcomes the objective of the specimen

Author: Duane Johnson

20 Dec

Posted via email from Duane’s Quick Posts

0 Comments

Filed under: Uncategorized

Why not write log files in JSON format?

Author: Duane Johnson

16 Dec

Log files were invented long before the JSON data exchange format existed. In fact, they bring a sort of crisp 1970s feel whenever I look at them… text files streaming across a terminal screen. It’s almost like the Matrix—anyone who sees text files streaming across a terminal instantly believes we might be hackers. But I don’t think log files are that cool.

To be honest, I wonder why they are still in line-text format. I realize it’s good to be able to process log files quickly (grep, sed, perl), but imagine the useful metadata that could be gleaned from log files—if only they allowed for metadata. Is it time to start producing JSON log files? It might be worth a look. Here’s why I began moving in this direction.

I first started noticing a need for something more than a line of text when this sort of thing started showing up in my log files:

I, [2009-12-16T13:24:59.264678 #13623] INFO — : [app:des.rb, line:26] Encode file: blah ->

The area of focus here is the second set of square brackets, [app:des.rb, line:26]. I was essentially starting to “tag” my log output so that I could tell where the lines were coming from. Next, I realized some of the log lines that were reporting the results of executing external commands could benefit from metadata. I started tagging my log lines with things like “shell:true” with additional metadata such as “process_id:39212″ and “status_code:0″ (process return code for “success”).

Finally, I started to wonder, “Why am I not using JSON for this?” I couldn’t think of a good reason except that JSON requires an (opening square brace) [ to begin and a (closing square brace) ] to end. Why not add an open square bracket to the beginning of an empty log file, and let it hang open forever? Here’s my new Ruby logging code:

log(”Encoding file”, :level => “info”) # info is the default and can be omitted

log(”Unable to encode”, :level => “error”, :reason => “missing command ‘des’”)

The resulting log file looks like this:

[{"level":"info","msg":"Encoding file","time":"2009-12-16 14:15:10 -0600","caller":["\/Users\/duanejohnson\/bin\/test.rb:10:in `<main>'"]},

{”level”:”error”,”reason”:”missing command ‘des’”,”msg”:”Unable to encode”,”time”:”2009-12-16 14:15:10 -0600″,”caller”:["\/Users\/duanejohnson\/bin\/test.rb:11:in `<main>'"]},

The hanging comma allows new JSON objects (hashes, essentially) to be appended to the end of the file. A JSON reader then just has to replace the last comma with a closing square brace and it becomes a valid JSON file that can be imported to a database, or searched arbitrarily. It’s still quite easy to use awk and grep, too, since each “object” ends in both a comma *and* a newline.

Finally, there is a simple way with the common unix tool ’sed’ to replace the final comma with a closing brace, thus making it fairly easy to deal with the hanging comma (thanks to some helpful people at stackoverflow for this suggestion):

sed ‘$ s/,$/]/g’

So, for example, you could fix the hanging comma with the following:

cat ruby.hanging.json | sed ‘$ s/,$/]/g’ >ruby.log.json

I like being able to store more information in my log files this way. It seems to fit my need to zoom in on data subsets and find answers to meaningful questions. Is there anyone else who feels the same way?

Posted via email from Duane’s Quick Posts

0 Comments

Filed under: Uncategorized

Psychologists find repeatable way to unlock fears without drugs

Author: Duane Johnson

9 Dec

This looks like an important discovery in psychology:

In an exciting breakthrough for psychological science, researchers in the United States have demonstrated a drug-free way to prevent the return of a learned fear. Similar memory modification effects have been observed before, but these experiments have involved drugs such as the beta-blocker propranolol. It’s hoped the new drug-free procedure will lead to improved therapeutic techniques for people with phobias or intrusive traumatic memories.

Basically, there is a very high success rate for “unlearning” a fear if the fear is recalled 10 minutes prior to a training session in which the fear stimulus does not result in the feared outcome. The amygdala essentially allows feelings surrounding the trauma to be forgotten even though the memory of the experience persists.

– Duane

Posted via email from Duane’s Quick Posts

0 Comments

Filed under: Uncategorized

The Present is Future Crystallizing into Past?

Author: Duane Johnson

8 Dec

I like where this theory is going. I think it deserves more discussion:

Today, Ellis and Rothman introduce a significant new type of block universe. They say the character of the block changes dramatically when quantum mechanics is thrown into the mix. All of a sudden, the past and the future take on entirely different characteristics. The future is dominated by the weird laws of quantum mechanics in which objects can exist in two places at the same time and particles can be so deeply linked that they share the same existence. By contrast, the past is dominated by the unflinching certainty of classical mechanics.

…

They point out, for example, that this crystallization process doesn’t take place entirely in the present. In quantum mechanics the past can sometimes be delayed, for example in delayed choice experiments. This means the structure of the transition from future to past is more complex than a cursory thought might suggest.

The part that seems most significant is the way it ties quantum dynamics to classical or relativistic mechanics. The uncertainty that lies in the future just can’t be treated like a classical model, except in very narrow cases where all of the parameters are carefully controlled. In the large majority of cases, small quantum effects trigger larger effects that go on to affect the final outcome in unpredictable ways.

Posted via email from Duane’s Quick Posts

0 Comments

Filed under: Uncategorized

Image-Processing Hardware for Filtering Out Porn?

Author: Duane Johnson

2 Dec

Technology keeps getting faster, but I wonder if it is fast enough to do real-time image processing to filter out pornography? More on this idea soon, but let me first explain why the question came up.

My wife is studying CIPA (the Child Internet Protection Act) as part of her Marriage and Family Therapy degree. As she pointed out, it’s always a tricky bit of legislation to get right since it’s easy to be too lax for the sake of First Amendment rights or too strict for the sake of the children.

I suggested perhaps the solution to the ambiguity inherent in English words would be to create an open-source filtering package as part of the bill. Let concerned parents who are also programmers get together to define a specification and a software program that would be the Gold Standard. Vague words such as “obscene” would yield themselves to the specifics of an algorithm, and the software included in the bill could become a standard for libraries and schools.

So back to the idea of real-time image processing. I don’t think I’ve ever seen a product that even attempts this, but why not create an adapter that sits in between the VGA cord of a computer monitor and the video card itself? This “adapter” would have to be a computer in its own right, of course—it would have to process the incoming image in real-time at 60 or so frames per second, and match nudity patterns in the pixels. It would then draw black boxes in place of the obscene images. The pixel buffer, in its modified form, would then be delivered to the monitor, (sans the nudity).

The benefits of such a technique would be threefold: (1) the source of the obscenity would be irrelevant—whether it came from the internet, a disk, or even a drawing, it would be filtered out; (2) there would be no need to keep a long “white” or “black” list of internet sites because all obscene images would be filtered out in an even-handed way (3) as a hardware solution, there would be no tampering with it unless physically tampered with.

Figure 1: An attempt to show a VGA cable with a black box in the middle.

Note: There is a network-based hardware solution that is similar to this idea called iBoss, but this is not the same technology. iBoss seems like it’s worth a look, however, if you’re looking for a hardware solution and don’t need my vaporware

Posted via email from Duane’s Quick Posts

1 Comment

Filed under: Uncategorized

Numbers that Can’t Say No to Time

Author: Duane Johnson

2 Dec

Infinity is such a “number”. The irrational numbers are too, as is π (pi), and e, and so are a whole host of “numbers” derived through functions such as sin, cos, tan.

This is probably old news to most math people, but it was just such a neat realization that I had to share it: there is a class of numbers that can be represented in space, for example, 1.0 and 10^99; and then there are a class of “numbers” that can only be described through a process that must involve time. These other “numbers” are processes—that’s the key point. Their secrets cannot be revealed without the dimension of time and a repetitious loop headed toward infinity.

Posted via email from Duane’s Quick Posts

0 Comments

Filed under: Uncategorized