Archive for the 'Uncategorized' Category

Happy birthday, Honey!

Monday, September 21st, 2009
My wife, Pat, and I were married 25 years ago this month. And 25 years ago today we celebrated her birthday for the first time. We had just moved into Verano Place, graduate student housing at UC Irvine.

In any case, 25 years ago this evening, we left to meet another couple for Pat’s birthday — I think they lived in in Laguna Niguel, or Tustin, or …? Anyways, we pulled into a Trader Joe’s to pick something up for dinner, and I’d forgotten my wallet, so we turned around, me slightly perturbed to have forgotten my wallet back at the apartment.

We had to call the couple to tell them we’d be late, so I suggested that Pat come up with me. While we were gone about 25 of our friends had piled into our tiny one bedroom apartment, and when we opened the door and walked a ways in “SURPRISE!!”. What a trip — yes, I’d planned it, and my wallet had been tucked beside the front seat. It was genius!

Pat got a fish, and an aquarium, from me, and I’m sure some other neat stuff :-) That aquarium, which was opposite our bed, would confirm months later that we were indeed in an earthquake in the middle of the night.

I remember Pat tried to throw me a surprise party on my next birthday, but I found the reservation receipt for the Verano common room a few days before the party and it wouldn’t have mattered — in fact, the receipt only confirmed what I already suspected, but I played along. Rogers, whose job it was to get me to the party, asked later “Did you know?” — I said “no”, but yes I did :-) But the fact is that Pat had already surprised me — a few weeks before, there was a knock at the door and I opened it, to find Pat and a big French Lop bunny, who Pat had gotten from the Animal Shelter — his name was Jackson. He lived on our third-floor balcony and used a cat box (mostly). I built him a cage, with the door kept open during the day — it cost us more to fly Jackson to Nashville when we moved in 1987 than it cost to fly either one of us. It was a very sad day when Jackson died years later in Nashville, now buried at our home there.

Anyways — no surprise this year, but I’m waiting for the flower shop to open.

The Mohawks in High Steel

Monday, September 21st, 2009
I remember enjoying the article “The Mohawks in High Steel” many years ago, to some extent because of the detailed descriptions of bridge work, not unlike Moby Dick’s detailing of life at sea, but I was also fascinated with the fearless, steelworking nomads who I share some small amount of blood with. Reading this excerpt from the original article (I have the whole hardcopy back in Nashville), I particularly like the recounting of the Caughnawagas’ introduction to high steel (about 3 paragraphs down, beginning with “The records of the company …”) and the workers’ reaction to “the disaster” a bit farther down: http://www.brooklyn101.com/articles/boerumhill-mohawks.pdf

I was reminded of “The Mohawks in High Steel” at the American Indian Museum on Saturday – I got there before opening, walked up from Foggy Bottom, saw a multimedia presentation in the round and then a fictional account in the main theater on 4 Native American’s living in today, one a Mohawk stock trader in New York. Then I went to the cafeteria and had shank of wild boar from the Great Plains counter (I should have taken a panarama shot of the various counters — I keep forgetting that not everyone can visit the place on a whim :-) .

The American Indian Museum seemed to be the place to go while preparing for a talk on computing and the Environment, and I was right.

Grokking the Mall

Thursday, September 3rd, 2009
One of my friend Vivian’s favorite words was ‘grok’ (http://en.wikipedia.org/wiki/Grok). She didn’t use it often, but she spoke of it highly. She explained it’s origins to me and its meaning. When it came up again she might explain it (… again) or she’d ask if I knew what it meant to which I’d reply “Of course I do — you’ve explained it to me before!” In this latter case she’d give me a look of consideration before moving on. :-)         

Anyways, I grokked something Sunday and thought of Vivian — it had nothing to do with her other than I grokked. I had walked from Foggy Bottom to the Lincoln Memorial, then through the Vietnam War Memorial, which I have never seen so spare of people or of remembrances — there was only one little display in fact, and I photographed it.

 p8300015.JPG

The Mall was as quiet as Christmas morning. I stopped at the Washington Monument to appreciate the flags at half staff and an older couple who were flying kites. Then I walked up the center of the Mall to the reflection pool beneath the Capital, but instead of heading straight to the upper plaza for one of the great views in Washington, I went to the Grant Memorial for a close look.

As many times as I had been to the Mall, I’d never taken the modest detour around the pool to examine Grant’s Memorial (http://en.wikipedia.org/wiki/Ulysses_S._Grant_Memorial). I circled the statue of Grant; it’s dark — he looks tired and well worn, but resolute. Grant’s likeness reminded me of Clint Eastwood in “Unforgiven” (I’m serious — thats what I thought!) — a scene where Clint is riding out of town on a stormy night, having killed most of town’s men in an unbalanced gun battle at the bar just moments before, and now holding the remnant in the shadows through force of his presence and their memory.
If you asked me the delimiters of the Mall before this weekend I would have said the Capital Building marked the east — I wasn’t even conscious of the Grant Memorial. Now Grant and the Capital are together on the east, and of course the Lincoln Memorial marks the west — it’s roof is barely visable from ground level at Grant’s statue, but it’s integral to the east end nonetheless; I don’t know how the Mall unfolded, but there is a grand design to it and it’s length is meant to be walked. That Ulysses S Grant stands post in front of the Capital building makes perfect sense, as does (now) the White House’s northern view of foreign patriots of the American Revolution.     

For the past two years I’ve been drawn to the Mall over and over. There is clearly something about it that resonates with me. I’m grokking it Vivian. Thanks!

Goldbach’s Conjecture, Turing Machines, and Artificial Intelligence

Thursday, August 20th, 2009
When I was a graduate student I’d work on proving Goldbach’s Conjecture when I needed a break from my real research. I’d focus on what this Wikipedia article (http://en.wikipedia.org/wiki/Goldbach’s_conjecture) calls the strong form : every even natural number (aka even positive integer) greater than 5 can be expressed as the sum of two prime numbers. So, for example, 6 = 3 + 3, 8 = 5 + 3, 10 = 5 + 5 (and 7 + 3), 12 = 7 + 5, …. Again, this is a conjecture that is believed to be true by virtually everone and its truth has been demonstrated with computers up to huge even numbers, but no one has proved its truth for all even numbers, and there are an infinity of them.   

The really attractive thing about number theory is that so many of the problems are so easy to understand by so many — you may not be able to solve the problem, but you sure understand what’s being asked! An approach I hit upon to prove Goldbach’s conjecture (or I suppose disprove it, or perhaps that you could’nt prove it one way or the other!) was essentially this, write a computer program that ran forever (if you were to run it), generating the even natural numbers one after the other, and write another computer program that ran forever (again, only if you were to actually run it), that generated all the sums of two primes “in sequence”, and then show that the two programs were equivalent. Unfortunately, that last step is REALLY, REALLY hard, if doable at all, but fortunately my PhD research took off about this time and I did that instead, much to the relief of my wife, parents, and in-laws!

But now, just as I want my artificial intelligence students to find projects of interest, this is the project that I want to return to. Its been about 3 years since I’ve done my own substantive computer programming, and its probably been 15 years since I’ve done substantive programming in the LISP language. So this will be fun! I can trivially write a program that generates all even natural numbers greater than 5: (defun GenEven () (do ((i 3 (+ 1 i))) (t (princ (* 2 i))))). A program that generates the sum of all pairs of primes is a good deal more complicated, because in general each addend needs to be verified as prime (http://en.wikipedia.org/wiki/Prime_number). In fact, one way to write this second program is simply to write a program that generates all prime numbers, and then “append” it to a copy of itself, and as each copy produces a prime the sum is output. However we write the second, what we imagine is something remarkable — that the latter very complicated program is equivalent to the former very simple program. And if you’ve taken formal languages and automata theory, you probably know that this is an example of the concatenation of an unrestricted (and non-context free) language with itself being equivalent to a regular language!

It would be tempting to spend a good deal of time making each of these programs as concise or as efficient as possible, but you see, I am never going to run either program. If I am biased in any direction it is that each program be as “unstructured” and as “primitive” as possible, because once these programs are defined, a third program, an AI program, is going to search for a sequence of rewrites that will transform one program into the other, while provably maintaining the original functionality of each. The third (AI) program is the one that will actually be run, and I’ll be writing this program in Lisp. But the two programs, one for generating the even numbers and one for generating the sums of prime pairs, I’m imagining will be written in the most primitive of languages — the language for programming (or defining) a Turing Machine — a simple form of computer, but not a computer that you would ever power up — a Turing Machine is strictly a theoretical device (http://en.wikipedia.org/wiki/Turing_machine).
The reason for the bias of starting with as unstructured and primitive as programs as possible is that though there are optimizations in the test for primality, for example, which I could reflect in my initial programs, these optimizations reflect patterns that almost certainly have been exploited in explorations of Goldbach’s conjecture by better minds than mine. It may be that any proof, if one is possible, has to rely on reasoning that is just off (human-conceived) map.
I’d actually started this process as a grad, exploring the ways to bridge these two programs, via an AI program that searched through billions of possible rewrites. I’m essentially an experimentalist and I start with code and looking for data — that’s my bread and butter. I think that what I am really doing is shaping my retirement 20 years from now (or less, for Pete’s sake). When friends visit and ask Pat where I am, she’ll point to the shed and tell them that I’m working on “that proof”.  More likely, I’ll be tinkering with the AI program, making sure that there are no bugs in it — can you imagine my despair, if near the end of my life and after searching billions of rewrites, my program comes back with “Proof Found!”, and I didn’t correctly save the path my AI program took to get there!?
The older I get, the more I remind myself of my father. 

 

Artificial intelligence, critical thinking, and Facebook friend suggestions

Sunday, August 16th, 2009
This fall I’ll be “teaching” Vanderbilt’s undergraduate course in artificial intelligence (AI) at a physical distance — from Arlington, VA. It’s an opportunity to shatter the way I’ve taught AI in the past. As uncomfortable as giving up the ol’ PowerPoint presentations are, and as creative an outlet as it was to prepare and still is to refine/adapt those slides (I’m absolutely serious — those slides are sometimes art — I’m seriouuuuuuus!), I’ve fallen back on them too much — they’ve encouraged an automaticity of teaching and a psychological distance from students that is not healthy. Gutsy was/is the professor that attempts a proof or writes a program for the first time (perhaps, in a long time) on a board in front of a class. And so long as the class recognizes that they are witnessing a search for an answer rather than an answer, its a-OK: “I’m about to demonstrate what it is to search for a proof, with some commentary, I hope, about how this search is systematic in advancing towards an answer, and if I/we don’t find the proof today, you and I can race to see who can post a valid proof by next week.” It’s scary, quite frankly. I think its possible that a fair number of professors got into academia, in part, because they saw a safe psychological distance modeled in the classrooms in which they were students.

Prefabricated slides also add friction that opposes mobility — if my slides are synced to a textbook, for example, it’s harder to switch textbooks if I feel obligated to prepare yet another set of slides, or even to adapt my current slides to the new text.

When I started this post, I really wasn’t intending to get into the above, but rather launch directly into some ideas that are arising as I think about the AI course as critical thinking “lab” — as we study ways of designing ‘intelligent’ computers, we reflect on our own thinking — for inspiration, comparison, and validation. My study of AI, and in fact computer science generally, has certainly shaped the way I think about my thinking and the thinking of others, but frankly its shaped my thinking itself — I’m quite certain that I think differently than I would have if I had gone to Humboldt State to study forestry, though probably not radically different.

As critical thinking lab I want to reflect on everyday scenarios (and not so everyday scenarios too). Not long ago my sister posted a status report on Facebook on how eerie it was that Facebook was making friend suggestions that were spot on — that is, she knew them, but she had no idea how Facebook would know that! This is certainly the case with me. There are, of course, friend suggestions that seem no brainers — e.g., I and suggested person X have 2 friends in common. Sometimes these no brainer suggestions have a plausibility, but in other cases they are not terribly suggestions, and Facebook has enough information to infer this — if the two common friends are married and there is no other substantive link between me and the suggested person that I can see, this suggestion loses credibility as someone I might reflect on, and if Facebook knows of the marriage link between the two common friends, you would think that their algorithms could be written to take this into account (and perhaps the algorithms do!?).

The mystery of course is in the suggestions on which I am clueless on how Facebook made a connection — do they have some kind of crystal ball? :-) Two clues as to what Facebook could do if they chose to do so appear in the Facebook margins itself. One is this app that keeps popping up — “Find out who is searching for you!”, with a very good-looking young woman smiling out at me :-) ). Who knows, maybe the app is a scam, but is does suggest that who I search for and who’s profile I am clicking on (including perhaps through a Google search result) is recorded, and why wouldn’t it be!? It also seems clear that the ads that pop up in the margins are customized to me in some fashion (e.g., “Eating out in DC?” “Class of 1975 memories”). So, why couldn’t friend suggestions take into account who/what I’ve searched for and who’s (partal) profile I’ve gone too? — I just went to the “Glendora High School Class of 1975″ site and clicked on a couple of non-friend people (”Is this Betty Friedland the Betty Moorehead that I had a crush on in first grade??!!”) Now, it seems unlikely that having looked at Betty’s partial profile and having not friended her that she would be suggested as a friend for me (but I’ll let you know). But is Betty Friedland going to see my name as a suggested friend? Or more generally will that I looked at Betty Friedland be a factor that’s taken into account in friend suggestions to Betty? I don’t know, but I’d like my students to be asking themselves these kinds of questions on everyday kinds of scenarios like this. Exercises like this (a) cast a light on my own thinking, (b) suggest what others can infer and what can be automated (because whatever I can infer from Facebook data can be inferred by others and by properly designed automated methods), and (c) beg some interesting ethical discussions concerning AI (and in this case, online social networks). In the context of AI there are a lot of mundane things that one can infer about me — e.g., I’m not part of the Washington DC network, but given my photo gallery and many of my friends, it’s not hard (for a human) to realize where I am living right now, though to program a computer to take these modest inferential steps would be quite interesting. And there are even more ambitious inferential leaps that can be made, by human and (thus) ultimately by computer — there are all kinds of research going on into what can be inferred from social networks, and exploring designs that guard against those inferences.

At age 52, I am a lot less concerned with my own privacy than (I think) I used to be — life is short as I’ve been aware of lately, and while I am much more self-revealing than I once was, I keep a low profile on politics and religion, at least so long as I’m in this government gig, though you could dig, but I don’t think a simple story would be revealed (though the simplicity of the story often has more to do with the “reader” than the “author”). I think that I might prefer automated inference methods that include explicit representation of uncertainty to human inference methods that sometimes (incorrectly) dismiss the uncertainty — of late uncertainty has been much more a friend than an enemy. To return to my original thread, however, I see online social networks as a treasure trove for AI class discussions. Music and environment are others. More to come, I expect.

Computing and Activism

Sunday, July 12th, 2009
Friend Mary Lou pointed me at this video on philanthropy in the internet era:

http://www.ted.com/talks/katherine_fulton_you_are_the_future_of_philanthropy.html

Apropos this, I’ve been thinking about Loeb’s “Soul of a Citizen” (http://www.paulloeb.org/soul.html), and computing:

How does social computing diminish AND exacerbate impediments to individual entry/magnitude into/of activism? (and Fulton’s pair of 5-point frameworks gives us a initial language to talk about it)

Can intelligent computing allow for the hearing of all voices (how much injustice is traceable to human inability to deal with complexity)?

When Fulton talks about about acting our way into new ways of thinking, she is speaking both descriptively and prescriptively, I think, and the prescription isn’t wrong, its just that it requires a great deal of THOUGHT .. There is nothing to be proud of per se with being an integral part of a transformative technology, if you are clueless about the societal and environmental changes it will bring about beyond a short-term horizon. If you are clueless, or if you “know” but are very wrong and/or shortsighted, then you are just along for the ride, regardless of your perception (IMNSHO :-) . The tension between “Fast” and “Connected” (for example), also requires thought. Again, Fulton’s two 5-point frameworks offer a good initial language for thinking, not so much a contrast between right and wrong.

I’m a believer in “Just Do It” BTW — Habitat for Humanity, Alternative Spring Break, Engineers without Borders, Computational Sustainability, … if anything, I think that we need to start thinking about whether social computing will diminish the Just-Do-It’ness and in what ways, as well as increase it. For goodness sake, when it comes to transformational technology, start thinking your brains out even as you are doing, at least if you are worried about 100+ years out.

My wife’s enthusiasms not (necessarily) contagious

Sunday, July 5th, 2009

Yesterday the two of us tromped all over the DC Mall, from McPherson Square Metro to the July 4th P-rade along Constitution, to the folk festival on the Mall, extending from Smithsonian Station to the Capital Building. Then after a “break” for work and a nap, back on the Metro that evening to Capital South and the concert at the foot of the Capital Building and fireworks over the Washington Monument, then walked to Foggy Bottom, finding that every Metro station along the way was hopelessly backed up. We got back to the apartment at midnight, dragging, and this morning it felt like I’d been dropped from our 18th-floor apartment.
 
My wife’s enthusiasms for parades, festivals, jazzy/poppy concerts, and fireworks are not contagious, so it was with trepidation that I ventured out yesterday, but not overt resistance, because I love seeing her enthusiastic, and I know that she has trekked out with me plenty of times despite her inclinations to do something else.

After getting off at McPherson Square in the morning, we walked through Lafayette Park, around the White House, and found a nice spot towards the end of the parade route, near the corner of Constitution and 15th. The Washington July 4 parade is remarkably small-town, and when I realized that I was watching a small-town parade in the nation’s capital, my mild chagrin turned to appreciation. I could have been watching the July 4th parade on main street in Glendora back in the 60’s. I so loved the reenactment of the raising of the flag at Iwo Jima by a band of not-so-prime-time, but nonetheless proud veterans on the back of a flat-bed truck. I didn’t tear up then, but I am now.

The folk festival was dominated by a huge Welsh presence — Welsh music, Welsh story-telling, Welsh food, craftmanship of numerous forms — slate and stone, fabric, … Its actually a 10-day festival, and July 4 was the second to last day. After a late lunch at my favorite eatery on the Mall — the cafeteria at the National Gallery of Art — we returned to the Festival. A Hare Krishna contingent was second in size to the Welsh, their tents occupying the upper part of the Mall nearest the Capital Building, with their crafts, foods, philosophies on display, with other Eastern-influenced traditions. At the Hare Krishna “Questions and Answers” tent, Patricia and I stopped to listen to a fundamentalist Christian “argue” with the Krishnas — “You’re going to Hell!”, “Well, if it doesn’t include you, that’ll be fine!” There was something about it .. I didn’t find it disturbing.

It was an incredibly perfect day for July 4 — almost cool. We watched and listened to a performance of four young men, rapping under the tent on the role of spoken word in African-American culture. They spoke of hip hop culture, Martin Luther King, President Obama, suicide, prejudice, justice, the worth of men and women in the black Christian tradition, …. It was a large, racially mixed crowd … an older white couple …. about our age (good grief) … sat in front of us, and kept looking at each other, nodding.

It was a short walk to the Welsh music tent, and we stood there a while, listening to the reels, before committing to seats, and listened to the four-man band, the corny but likable front man, and watching the dancers of all types, couples and soloists, do their thing on the large wooden dance floor under the big tent — I thought of friend, Wendy, and her Celtic dancing, and Franko too, and their music.

That night, we got off at Capital South, and went to the far side of the Capital — I was really surprised at the access we had. We went through a security checkpoint, and Patricia tried to navigate us up towards the Capital Building, where she thought viewing would be best, but a police officer directed the incoming crowd to “go down to the bottom of the hill, there is no seating up here”, Pat tried to navigate around the officer, but was blocked by another, who directed her personnally “Ma’am, down the hill please.” She turned, and I followed, but within a few steps, Pat saw a cubbyhole, quite amazing actually, and we darted into it, for as good as a spot as we were going to get for having come in after the concert had started. We arrived just in time to hear Jimmy Smits announce Aretha Franklin, and the crowd rose as one. Aretha isn’t twenty any more, but Aretha is Aretha, and it was a girating couple of songs. We could see her on a big screen in the distance, partially occluded by trees, and I spotted her person, a speck moving in the distance, and pointed her out to Patricia. We had a great view of the fireworks, which from our vantage point exploded at the tip of the Washington Monument.

On the walk home I reflected aloud on how tenacious Patricia had been at finding our spot and she talked about her family of ten: “We never challenged authority, but we never let it stop us.” :-)

I took 205 pictures yesterday — two seem worthy of posting.

p7050086.JPG

And yesterday was my father’s birthday — born on the 4th of July — happy birthday, Smith! RIP

Personal Data Mining

Friday, July 3rd, 2009

Data mining looks for patterns in data, typically in a (semi-)automated fashion with ‘intelligent’ computer software. Examples of data mining include seeking patterns in genomic and demographic data of cancer patients (http://www.vuse.vanderbilt.edu/~dfisher/Papers/RuleBasedLearning.html) and searching for patterns in the observed behavior of students working math problems. Discovered patterns are ‘gems’, and thus the mining metaphor, because patterns can be exploited to the good, we hope. For example, discovered patterns in genomic data can be used to better target treatments (e.g., chemo therapies) to patients exhibiting different patterns, while discovered patterns in student problem-solving data can guide educational remediation. Data mining can certainely benefit medicine, environment, education, and business, but there are ethical concerns/cautions concerning data mining too — e.g., genomic patterns can be used to fit patients to treatments, but like patterns can predict risk too, and do we want insurance companies to pay for tests that will identify personal patterns, while trusting them not to use this knowledge to exclude coverage?

I first did PERSONAL data mining, albeit “by hand”, after being diagnosed with diabetes. The data in this case consisted of (a) blood glucose readings, at least 8 a day and spaced appropriately, (b) morning and evening weighings, (c) food label data to include calories, carbs (total and broken down into sugars, fiber, others), proteins, fats, etc of what I was eating and timestamps of when I was eating it, and (d) exercise timestamps and intensities. This all seemed a bit over-the-top to some, but not my doctor at the time, Ben, who loved it, and I quickly converged on good glucose levels without medication. Some interesting anecdotes about this whole process are that the data collection was informed by knowledge — for example, glucose readings were spaced in a manner guided by my research into when glucose was likely to peak after a meal, and this illustrates the more general point that data collection/mining typically isn’t done in a conceptual vacuum. Also my processes of data collection, pattern discovery, and responses such as changes in diet, exercise, etc, were more tightly coupled than these processes are in most other data mining contexts. That first month or two was highly beneficial and I still reap the benefits — when I saw my doctor in Arlington a few months ago, she said that I was the best controlled diabetic she’d ever seen, and I beamed of course :-) . Later I also got an inkling into the allure, probably felt by body-builders and perhaps those with eating disorders, of being in control of my body. In my case the feeling of control felt mainly healthy (because control is critical in effective management of diabetes), but there was an allure to the control that went beyond the strictly healthy — I remember hitting 140 lbs, high-school wrestling weight, and thinking “135 would be easy”, and the goal had appeal, I think, only from a feeling of control when other life circumstances felt out of control. That said, one thing that Doc Ben advised me upon diagnosis was “get skinny” and he wasn’t talking “middle-age skinny”.

My ‘data mining’ of glucose, diet, weight, and exercise data was the topic of an invited talk back in the late 90’s, as well as a (declined) proposal to customize ‘generic’ (i.e., population-wide) mathematical models of glucose/insulin dynamics to fit an individual’s particular dynamics based on individual data — it’s a topic that I want to return to again, and now with renewed excitment.

My interests in personal-data mining have continued and recently I found data from iTunes. When I discovered the Top-25 playlist on my iPod nano, I knew that certain data had to be recorded, and sure enough its easy to get to on iTunes by viewing my music library in ‘list’ format. An important attribute is ‘Play Count’ — the total number of times the song has been played on my iPod as of the last synchronization. It is possible to reset (zero-out) this attribute, which could compromise data accuracy, and I don’t know how long the song has to play before its counted as played — I’m certain for example that if I forward to the next song with 5 seconds REMAINING on the current song then the current song is still counted as played, and likewise I’m guessing that if a song plays only two seconds before I forward to the next, its NOT counted as played, but rather counted as SKIPPED (total times a song is skipped is another attribute), but these are just guesses — these and other questions might be answerable from research on the Web, patent records, or other resources. Generally, its often the case that data collection, interpretation, and coding are not trivial.

Its not hard to spot at least one “pattern” if you look at my top 10 (out of about 216 songs) — even the Genius recommender system can catch the one I have in mind:

       Song Name                         Artist                       Play Count
1   I Know You Rider               Hot Tuna                        193   (off Classic Hot Tuna Electric)
2   I Know You Rider               Half Day Bluegrass Band      80
3   The Touch of the Master’s Hand    Laurie Lewis              78
4   Under God’s Light                 Rare Earth                       78
5   I Know You Rider (Live In Paris)  Grateful Dead               76
6   Tears Of A Clown                  The English Beat               67
7   Downtown                          Neil Young                         65
8   I Know You Rider (1966)           Grateful Dead               62
9   Girlfriend                        Matthew Sweet                       61
10  Well…All Right                  Buddy Holly                         57

The count of 193 is not a typo, and it would be tempting to call it an “outlier”, which is something so anomolous that it should be regarded as outside the scope of analysis, but in this case of looking at song listening behavior omitting this point would be like trying to understand solar system behavior while disregarding a black hole that was approaching the heliosphere because it was so much more distant than our other heavenly neighbors. And, as it turns out, this 193-count song is an integral part of a pattern, and thus not an outlier at all. When I rank the songs from most played to least played and plot them, I get a graph very like in shape to the one in the first figure below. The first graph actually shows the ‘average number of plays per day’, which isn’t an attribute that iTunes actually gives me, but it was an attribute I computed — after all, it might be that a song is played more only because I’ve owned it longer (which caries information to be sure), but by looking at the per-day average of each song since my ownership of it began I am ‘normalizing’ it. iTunes tells me the total number of plays, and the date of acquisition in my library (by purchase or copy from CD), and Microsoft Excel includes a function, DAYS360, for approximating the number of days between two dates — in this case the current date and the date of acquisition. The graph of average plays-per-day shows some of the same songs at the top, including my musical black hole, but other songs are now at the top after compensating for days owned. And of course, this new attribute reflects other biases, like my tendancy to play a song more often earlier in ownership — I’m sure that I’d verify this pattern by looking at my iTunes library statistics over time — and there is much activity on many forms of temporal data mining.

(CLICK ON THE IMAGE)

figure1.jpg

The second figure below shows a curve fit to the data on plays per day — in this case the curve is a power function and it fits the data well (and better than other common functional forms, including logarithmic). A power function, suffice it to say, is a functional form characterized by rapidly diminishing returns. I first learned about power functions in Tarow Indow’s UCI class on human cognition and memory, where behaviors characterized by rapidly diminishing returns are ubiquitous. For example, if I practice some new behavior, I become competent rapidly, and continued practice brings further improvement, though in smaller and smaller increments (or decrements) — this is called the “power law of practice”. But power laws manifest in many other contexts too, like rate of memory retrieval from a given category and the learning behavior of machine learning programs, which Lewis Frey and I published on years ago (http://www.vuse.vanderbilt.edu/~dfisher/Papers/ModelingTree.pdf).

figure2.jpg

I looked at skips too, notably skips per day (figure 3, in red), ordered from most to least, and asked whether more frequently listened to songs tended to be more rarely skipped songs, with the apparent answer being ‘NO’ (see figure 4), but I think that there are varying reasons for the relative lack of correlation. For example, there are some songs that I have listened to rarely, but that I have never skipped (Skip Count = 0) and I may never skip them. Also, there is a Top-25 list that is maintained on my iPod, and there is this musical black hole that I choose to listen to often, and thus the second most listened to song (and reliably second on my Top-25 playlist), is among my most frequently skipped songs!  This latter example highlights the importance of the iPod nano interface and functionality in getting the data that I’m getting. The iPod shuffle would show a much more linear relationship (of near zero slope) I expect, because my ability to chose next song would be much more limited (though I am probably ignorant of functions of the shuffle like an ability to create playlists). And on the side of greater functionality, if I could enter a song at a middle point, I might listen to some songs even more, such as Under God’s Light (number 4 on my list of total plays), with a final instrumental section that I like better than say that of Stairway to Heaven, and ranking right up there with Black Magic Woman, Freebird, and Green Grass and High Tides, and besides Under God’s Light was on the first album I ever bought, One World, so it was imprinted on me early (note the cool cover, particularly appealing to 13 year old, but heck it still is, who am I fooling: http://www.amazon.com/One-World-Rare-Earth/dp/B000K7BPYI#moreAboutThisProduct. And the first customer review is right on).

figure3.jpgfigure4.jpg

I am left with a lot of questions. I wonder whether the existence of a top-25 list that I often use causes the diminishing-returns characteristic to be exaggerated or lessoned? It would be interesting to look more at skips, normalize the “raw data” in other ways, guess as to what iTunes might do with the data they receive, speculate on real “genious” music recommender systems, and elaborate on mining for temporal patterns. In this latter case, I wonder, for example, whether the rate that I listen to a given song falls off according to a power law — maybe in some cases, but in others like the black hole, it might fall off linearly (and it is falling off) … but if I had the time to collect and look at the data, I wouldn’t have to guess.

Finally, there appear to be power laws of musical listening behavior, and possibly other manifestations of the soul, almost certainly aspects of friendship. But not only are there important differences in interface (e.g., nano factionality versus shuffle versus radio) that determine the data, but there are individual personal differences too. The returns (as in rapidly-diminishing returns) in cases like music and friendship  include sadness and joy.  There are many musical expressions that bring these returns, and its fair to say that in a radio-listening context I would not station-hop past the large majority of the songs on my iPod with say greater than 10 plays (most of my single-digit played songs were those I downloaded for my friend Vivian when she was in hospice, but some of this latter set I have adopted as my own), but the additional returns I get beyond those of higher ranking seem to fall off rapidly in an environment in which I have a choice, but I don’t think its (necessarily) because they are of lesser importance to me, but some/many simply fill a more specialized niche. And here is an aside — if power laws, which seem inherently unbalanced to me, come with mechanisms that allow greater individual choice, then what are the implications for sustainable decision making?

Clearly, in the diabetic-related data case, I learned actionable patterns of great value. What have I learned from the iTunes-data mining case beyond what I already knew? In some ways nothing (yet) beyond some details on numbers of plays and skips, and the magnitudes of some of these were surprising, and I suppose the power law in this context was a surprise too. Beyond this, it has caused me to reflect and it highlights certain things, like the cluster of lesser-played but zero-skipped songs (zero is a very special number), which nobody but me could possibly understand, unless they had done their own mining and reflecting to understand what such a cluster meant to them, which might give them just enough insight into me to ask me the question — what does this cluster mean for you?

(Plots and curve fits were done by open source software Graph 4.3 (http://www.padowan.dk/graph/)).

All 216

    Song Name                               Artist              Play Count
1   I Know You Rider               Hot Tuna                    193
2   I Know You Rider               Half Day Bluegrass Band      80
3   The Touch of the Master’s Hand    Laurie Lewis                 78
4   Under God’s Light                 Rare Earth                   78
5   I Know You Rider (Live In Paris)  Grateful Dead                76
6   Tears Of A Clown                  The English Beat             67
7   Downtown                          Neil Young                   65
8   I Know You Rider                  Grateful Dead                62
9   Girlfriend                        Matthew Sweet                61
10  Well…All Right                  Buddy Holly                  57
11  Fortunate Son                     U2                           55
12  Save It For Later                 The English Beat             54
13  Beautiful Day                     U2                           54
14  There Goes Another Love Song      The Outlaws                  53
15  Celebrate                         Sam Bush                     52
16  Castanets                         Alejandro Escovedo           51
17  In a Big Country (Radio Edit)     Big Country                  48
18  Gloria                            Van Morrison with Them       47
19  Praise You                        Fatboy Slim                  45
20  Down On Me (Live)                 Janis Joplin                 45
21  Get Ready  (21 min)               Rare Earth                   43
22  Cinnamon Girl                     Type O Negative              42
23  In God’s Country                  U2                           40
24  Born On the Bayou                 Creedence Clearwater Revival 37
25  Our Lips Are Sealed               The Go-Go’s                  37
26  Good King Wenceslaus              Melanie                      36
27  Scarlet Begonias                  Grateful Dead                35
28  You Wreck Me                      Tom Petty                    34
29  Time Has Come Today               Joan Jett                    33
30  Spirit In the Sky                 Plumb featuring Mikeschair   33
31  It’s A Long Way To The Top (If… AC/DC                        32
32  A Mighty Fortress Is Our God      Mormon Tabernacle Choir      32
33  She’s a Mystery to Me             Roy Orbison                  32
34  New World Man                     Rush                         32
35  Great White Buffalo (Live)        Ted Nugent                   32
36  My Love Will Not Change           The Del McCoury Band         31
37  Hoedown (Taken from Rodeo)        Emerson, Lake & Palmer       31
38  John Henry                        Harry Belafonte              31
39  Where Are We Runnin’?             Lenny Kravitz                31
40  Magic Carpet Ride                 Steppenwolf                  31
41  Jessica (Single Version)          The Allman Brothers Band     30
42  Green Grass And High Tides        The Outlaws                  30
43  Limelight                         Rush                         30
44  Best of Both Worlds               Van Halen                    30
45  Johnny Strikes Up The Band        Warren Zevon                 30
46  Werewolves Of London              Warren Zevon                 30
47  I Can Love You Better             Dixie Chicks                 29
48  Tuff Enuff                        The Fabulous Thunderbirds    29
49  I’m a Believer                    The Monkees                  29
50  Finest Worksong                   R.E.M.                       29
51  Lucky Never Had It So Good        Ashley Cleveland             28
52  Ramble Tamble                     Creedence Clearwater Revival 28
53  In the Evening                    Led Zeppelin                 28
54  Rocky Top                         The Osborne Brothers         28
55  Every Picture Tells a Story       Rod Stewart                  28
56  Time To Start                     Blue Man Group               27
57  1952 Vincent Black Lightning      The Del McCoury Band         27
58  Grey Seal                         Elton John                   27
59  Cold Rain and Snow                Grateful Dead                27
60  The Safety Dance                  Men Without Hats             27
61  I Know You Rider                  Phil Lesh & Friends          27
62  Desire                            U2                           27
63  Hush                              Deep Purple                  26
64  You Can Close Your Eyes (Live)    James Taylor                 26
65  Somebody to Love                  Jefferson Airplane           26
66  Someone To Love                   Rare Earth                   26
67  What’d I Say                      Rare Earth                   26
68  What I Like About You             The Romantics                26
69  My Maria                          B.W. Stevenson               25
70  Stage Fright                      The Band                     25
71  Twist and Shout                   David Lindley & El Rayo-X    25
72  Shapes of Things                  Jeff Beck                    25
73  Express Yourself                  Madonna                      25
74  Jet Airliner (Live)               Steve Miller Band            25
75  867-5309/Jenny                    Tommy Tutone                 25
76  Magic Bus                         The Who                      25
77  Thank You                         Alanis Morissette            24
78  Down In The Hollow                Leftover Salmon              24
79  Cripple Creek                     Leo Kottke                   24
80  What A Crying Shame               The Mavericks                24
81  Tennessee Stud                    The Nitty Gritty Dirt Band   24
82  Girl of the North Country         Sam Bush                     24
83  Rider                             The Seldom Scene             24
84  Kentucky Woman                    Deep Purple                  23
85  Introduction/Darlin’ Cora         Harry Belafonte              23
86  Free Bird                         Lynyrd Skynyrd               23
87  Gloria                            Patti Smith                  23
88  I Just Want to Celebrate          Rare Earth                   23
89  Excitable Boy                     Warren Zevon                 23
90  I’m So Glad                       Cream                        22
91  Playing in the Band               Grateful Dead                22
92  The Golden Road (To Unlimited…  Grateful Dead                22
93  La Bamba                          Los Lobos                    22
94  Magic Key                         Rare Earth                   22
95  Marianne                          Stephen Stills               22
96  White Rabbit                      Blue Man Group Feat. Esthero 21
97  But Anyway                        Blues Traveler               21
98  Crossroads (Live At Winterland)   Cream                        21
99  Mercury Blues                     David Lindley                21
100 Love Is A Long Road               The Del McCoury Band         21
101 Hocus Pocu (US Single)            Focus                        21
102 Johnny B. Goode                   Grateful Dead                21
103 Angel to Be                       Sam Bush                     21
104 Hello Mary Lou                    The Seldom Scene             21
105 Jungle Love                       Steve Miller Band            21
106 Lawyers, Guns And Money           Warren Zevon                 21
107 One                               Deirdre Jenkins              20
108 Best Friend                       The English Beat             20
109 Would I Lie to You?               Eurythmics                   20
110 Stand                             R.E.M.                       20
111 Poor Poor Pitiful Me              Warren Zevon                 20
112 I Feel Love                       Blue Man Group Feat. Venus H 19
113 Time Has Come Today               The Chambers Brothers        19
114 Werewolves Of London              David Lindley & El Rayo-X    19
115 Happiness (I’m So Glad)           Deep Purple                  19
116 Hocus Pocus (Long)                Focus                        19
117 Who Do You Love [Live]            George Thorogood             19
118 Mama Tried                        Grateful Dead                19
119 With a Little Help from My Fri… Jim Sturgess & Joe Anderson  19
120 (I Know) I’m Losing You           Rare Earth                   19
121 Hey Big Brother (Single)          Rare Earth                   19
122 Roundabout                        Yes                          19
123 L.A. Woman                        Billy Idol                   18
124 Light My Fire                     The Doors                    18
125 A Better Man                      Keb’ Mo’                     18
126 Gallows Pole                      Led Zeppelin                 18
127 Pamela Brown                      Leo Kottke                   18
128 Colorful                          Rocco DeLuca & The Burden    18
129 Great White Buffalo               Ted Nugent                   18
130 Who Are You                       The Who                      18
131 Follow You Down                   Alejandro Escovedo           17
132 Blue Sky                          The Allman Brothers Band     17
133 All Right Now                     Copycats                     17
134 The Cold Hard Facts               The Del McCoury Band         17
135 L.A. Woman                        The Doors                    17
136 I Know You Rider                  Joan Baez                    17
137 Nobody Told Me                    John Lennon                  17
138 God Trying To Get Your Attention  Keb’ Mo’                     17
139 Can’t You See [Live]              The Marshall Tucker Band     17
140 Middle of the Road                The Pretenders               17
141 Get Ready (radio edit)            Rare Earth                   17
142 I Am A Man Of Constant Sorrow     The Soggy Bottom Boys        17
143 Once In a Lifetime                Talking Heads                17
144 Mercury Blues                     David Lindley & El Rayo-X    16
145 Easy to Slip/I Know You Rider     Little Feat                  16
146 Pop Song 89                       R.E.M.                       16
147 Undone (The Sweater Song)         Weezer                       16
148 The Goddess                       Deirdre Jenkins              15
149 Bertha                            Grateful Dead                15
150 Me & My Uncle                     Grateful Dead                15
151 Mexico                            James Taylor                 15
152 Bye Bye Love                      David Lindley & El Rayo-X    14
153 Wharf Rat                         Grateful Dead                14
154 Not Fade Away/Goin’ Down the Road Grateful Dead                14
155 Get Up                            R.E.M.                       14
156 Feelin’ Alright                   Rare Earth                   14
157 Time Has Come Today               The Chambers Brothers        13
158 I Feel Love (12″ Version)         Donna Summer                 13
159 Fortunate Son                     John Fogerty                 13
160 U Got the Look                    Prince                       13
161 It’s the End of the World …     R.E.M.                       13
162 Miss Me but Let Me Go             The Rarely Herd              13
163 And She Was Talking Heads         The Best of Talking Heads    13
164 Won’t Get Fooled Again            The Who                      13
165 Ramblin’ Man                      The Allman Brothers Band     12
166 School of Rock                    Karaoke All Stars            12
167 Brandy (You’re A Fine Girl)       Looking Glass                12
168 I Am A Man Of Constant Sorrow     The Soggy Bottom Boys        12
169 Rock and Roll, Pt. 2              Gary Glitter                 11
170 Bad To The Bone [Live]            George Thorogood             11
171 You’ve Got a Friend               James Taylor                 11
172 Vertigo                           U2                           11
173 God Will Take Care of You         Aretha Franklin               9
174 Singing in My Soul                Sister Rosetta Tharpe         9
175 Climbing Higher Mountains         Aretha Franklin               8
176 I Wouldn’t Mind Dying             Dorothy Love Coates & …     8
177 I Know You Rider                  Roger “Hurricane” Wilson      8
178 Didn’t It Rain                    Sister Rosetta Tharpe         8
179 On Our Way (1-13-1972 Opening…  Aretha Franklin               7
180 Climbing Higher Mountains …     Aretha Franklin               7
181 My Sweet Lord (1-14-1972 In…    Aretha Franklin               7
182 Lord, Don’t Forget About Me       Dorothy Love Coates & …     7
183 One Bourbon, One Scotch, One Beer George Thorogood              7
184 Wholy Holy (1-13-1972 Version)    Aretha Franklin               6
185 Give Yourself to Jesus            Aretha Franklin               6
186 There’s a God Somewhere           Dorothy Love Coates & …     6
187 How I Got Over                    Aretha Franklin               5
188 My Sweet Lord (1-13-1972 Instrumental Version) Aretha Franklin  5
189 Old Landmark                      Aretha Franklin, James …    5
190 That’s Enough                     Dorothy Love Coates & …     5
191 You’ll Never Walk Alone           Aretha Franklin               4
192 What a Friend We Have In Jesus    Aretha Franklin               4
193 Aretha’s Introduction …         Aretha Franklin               4
194 Mary, Don’t You Weep              Aretha Franklin               4
195 I Won’t Let Go                    Dorothy Love Coates           4
196 Opening Remarks By Reverend C … Aretha Franklin               3
197 Aretha’s Introduction …                                       3
198 Medley: Precious Lord, Take My… Aretha Franklin               3
200 Wholy Holy                       Aretha Franklin               3
201 On Our Way (1-13-1972 Version)    Aretha Franklin               2
202 Never Grow Old                    Aretha Franklin               2
203 Precious Memories                 Aretha Franklin               2
204 I Drink Alone [Live]              George Thorogood              2
205 Shenandoah                        Harry Belafonte               2
206 I Am A Man Of Constant Sorrow (Instrumental) John Hartford      2
207 Henry                             Keb’ Mo’                      2
209 Precious Memories                 Aretha Franklin               1
210 On Our Way (1-14-1972 Opening…  Aretha Franklin               1
211 On Our Way (1-14-1972 Version)    Aretha Franklin               1
212 What a Friend We Have In Jesus… Aretha Franklin               1
213 She Took Off My Romeos David Lindley & El Rayo-X                1
214 I Was Wrong                       Keb’ Mo’                      1
215 Highway Blues                     Marc Seales                   1
216 Down by the Riverside             Sister Rosetta Tharpe         1
 

Madeleine Peyroux at GW’s Lisner Auditorium

Sunday, June 21st, 2009

My wife, Patricia, got a call yesterday from one of our Nashville friends, Pat. He was backing Madeleine Peyroux at George Washington University’s Lisner Auditorium, just a couple of blocks from Foggy Bottom last night and wanted us to come to the show. Neither of us had heard of Madeleine Peyroux, but Pat is one of those special people you want to see — in fact, we are blessed to have great friends — they have bent over backwards for us, and I hope that I/we am/are the same.

We got to the Lisner just before showtime, a long but fast moving line, picked up the tickets and backstage passes (!) at the Will Call window — our seats were 7 rows back from the stage, on the center aisle. The Lisner reminds me of Langford at Vanderbilt, but about half again bigger, presumably to accommodate DC crowds, and it was sold out. The opening act, an acoustic guitarist from Portland, OR was great — but I can’t recall his name, and surprisingly can’t easily find him on the Web, though you would think that there would be enough constraints to nail it quickly, and I’m not a naive user. Apparently, there is a lot that publicists (who are wanting to promote new artists) still don’t get about the Web — they introduce a new act, apparently expect people to remember the name, because they (at least in this case) don’t set up the online constraints to find the name quickly (in about 2 minutes) when you look on impulse afterwards. Or heck, maybe I just missed it — it wouldn’t be the first time, but … :-) .

Madelein Peyroux was outstanding. As I experience over and over, there is something about a live performance that is much different — I don’t think that I’d listen to her regularly in the iTunes context. Tony Bennet, Keb Mo, Leftover Salmon, Buddy Rich, Minton Sparks ( I held her purse during the show at The Basement — I was in the front, and she threw me the purse and told me to watch it) and the Nashville Symphony for that matter — all performers that have mesmerized me live, but a slew of CDs bought on impulse immediately afterwards, certainly in the case of Keb Mo and Leftover Salmon, but rarely played are evidence that the peak live experience doesn’t necessarily transfer. It’s not that I don’t like the music, but when I have a choice, and with an iPod the choice is as easy as with remote control (almost), I choose something else. This isn’t all that different from back-in-the day — you could still pull out my vinyl record collection and easily — very easily — spot the tracks that I regularly listened to, or even the segments within tracks. I’m not really a fan of artists, but a fan of songs, and even covers — that’s not bragging — I think that it may be related to attention span!

“A Little Bit” is but one song I really liked, even the studio version below, but the live version rocked a bit more, with Pat going off on a build with his guitar — wonderful.

“A Little Bit”: http://www.youtube.com/watch?v=mhkXPoHxx4s

I am pretty sure that we’ll be going back to the Lisner for this show :

Yamato: Demon Drummers of Japan: http://lisner.org/eventdetails.asp?id=521

I think that I’m a drummer actually.

Plugged in and zoned out in Arlington

Thursday, June 18th, 2009

It’s amazing how many people around here have got something in their ears, generally some kind of mp3 player (http://en.wikipedia.org/wiki/MP3), and I’m one of them. It seems that every pedestrian on the two-block walk from NSF to my apartment is in their own little world, and on the elevator it’s not unusual for every rider to be plugged in. If the gadget people don’t yet outnumber the others on the Metro into DC, they will soon. Maybe a little more than a month ago, one Saturday or Sunday morning, I was walking into the office to do some work before Pat got up and a youngish gal and guy were walking towards me arm in arm, talking lightly, smiling, I was listening to music on my iPod, but still aware, and as they passed me, the gal gets in close to my face and says loudly (but not shouting) “HeLLOOOOOOOOO” or some such thing, and I smiled and responded “Hey, there!” Some other time it might have caught me by surprise, but not that morning, and I wasn’t bothered, which I certainly could have been under other circumstances (“The NERVE!”) and she actually seemed to be the surprised one! It’s a sign too that I’m getting older, despite my self-conception – the one that is shattered each morning — because the guy didn’t seem unnerved – I was no threat :-) , and besides I was wearing my hat, and the greatest thing about my hat is that strangers smile at me … a lot. But back to the no-threat-to-the-boyfriend observation — I think that it’s helpful to understand that some (many?) men are sometimes controlling, I suspect, because at some primal level, possibly beneath consciousness, but often not, they are simply worried about getting their butt’s kicked.

I wasn’t upset, I think, because I’m pretty sure that my gut knew an approximation of what that gal was thinking because I’ve often thought it myself and smiled amongst the crowd of similarly-adorned people …. “Are you even present?!” … “Well, more than you thought apparently, but no, not really, and not as present as if I was in the jungle getting stalked.”

And as I write I am appreciating what my little iPod has done, bought by my wife, Patricia, as a birthday present almost one year ago to the day, with encouragement of friend/colleague Mary Lou to get WITH IT. It’s changed the way I exercise – exercise used to be a chore, but I tear it up now listening to Papa Creech’s fiddle on Hot Tuna’s I Know You Rider, Ted Nugent’s guitar on The Great White Buffalo, David Lindley and El Rayo X’s Mercury Blues, and quite a few of my other 210 songs. One late night at Alive Hospice in Nashville, the patients, including my friend Vivian, were in the hall during a tornado warning, almost a party and certainly fun, Vivian with my iPod, flipping through the songs and stopping on the Monkey’s I’m a Believer (this is a big confession for me) and she started singing along out loud, head swaying, and I tried to shush her, but she looked straight at me, nonchalant as you please, and started singing louder – I smiled. In fact, that Saturday morning when the gal had said hello, I was remembering Vivian to James Taylor, which I do now when I want to shed a few — You Can Close Your Eyes, Mexico, You’ve Got a Friend. So the iPod isn’t only about isolation. In fact, in writing I am wondering if that’s why I got the hello.

Nonetheless, I have made some changes in response to recent observations – I turn the iPod off as I approach the Apartment building’s reception desk, the guard’s station at NSF, get on an elevator, etc, in case anyone says “Hi”, which the guards always do (“Happy, Monday!” in a Caribbean accent) or as a prompt to say “Hi” myself, and it goes without saying (?) that when Metro’ing into DC with Patricia on a Saturday that I’m not listening unless she’s wearing one of the plugs, but generally to Bill Moyers and not Ted Nugent.