What Does Qualitative Data Look Like? and Other Issues with this Much-Needed Genre of Data for [Jewish] Non-Profits

Yes, I've got a lot to say on this matter, so here is a table of contents: I. Introduction, II. Qualitative Data vs. Quantitative Data, III.  [Some] Types of Qualitative Data, IV.  [Some] Problems with Collecting Qualitative Data, V.  Moving Forward

 - - - - - 

I. Introduction
       Before the school year began, I had some time to put down some of my thoughts regarding a sadly much-overlooked piece in the Jewish (perhaps the non-profit world more broadly?) world: "Qualitative Metrics for the Jewish Community?" (which I was then able to get re-posted at eJewish Philanthropy the following week).  My main point I tried pushing in that post is that we should not be as "concerned with how many people show up as a marker of programmatic success" than "with how the program actually went and its impact upon the participants/attendees."  Granted, coming from me, that's not surprising since my view on matters is to be concerned more about quality than quantity, but this view should not be exclusive to me within the Jewish world (and, perhaps, the non-profit world more generally).
     Having posted it, I received a variety of feedback, some not sure what to do with it (okay, largely the organizations I mentioned, who thought I was criticizing them), but also many who thanked me for opening up this conversation and that they had been wondering about this matter, as well.  My goal, as I stated in the original post was "to push the conversation not only toward thinking about the importance and need of qualitative data amongst participants of Jewish communal programs, but also to see if we can develop methods of such data collection as well as to be able to get it into the minds of funders that a program's impact is more than just number of feet who walk into a program."
      As I mentioned in the follow-up post, I was glad that I felt I achieved the goal of people discussing qualitative data and its importance rather than an almost exclusive focus on quantitative data (I was super happy with the response by Renée Rubin Ross and Matthew Grossman (especially since they start off with "Drew Kaplan is right"), wherein they acknowledge "Funders and organizations offering Jewish experiences must pay attention to qualitative assessment. While it is challenging to measure the effects that unique Jewish experiences have on the lives of individuals, thanks to partnerships between funders and Jewish educational organizations, new approaches are emerging to do just that."  Fortunately, it seems that the Jim Joseph Foundation gets it and is really working toward integrating quantitative data in their considerations of developing young Jewish people), however I don't know or think there was much further discussion of the matter (let alone philanthropically-minded folks changing their orientation from quantitatively-focused to also considering (or mostly considering) qualitative matters.  Part of that is my fault for not having continued posting on the matter and continuing to keep it on people's minds.
     I also mentioned in that follow-up post that my ideas had been further shaped/advanced through being in dialogue with people about this topic and that was tremendously delightful.  I wrote down notes in August about writing further about this matter and never seemed to grab the time to do so.  Now that the school year is winding down, I am grabbing some time now to write about this once again.

 - - - - - 

II.  Qualitative Data vs. Quantitative Data

     Although people may be quite familiar with quantitative data, that is numbers that inform about the amount of something; although in this case of the non-profit Jewish world, I mean how a measurement of how many people are participating in something: it could be an event, it could be a variety of events, etc.  It seems that the Jewish world is largely concerned with the numbers of feet that make it through the door at a given event or over the course of a given time-span.  While this can indicate marketing/promotional success, it does not indicate how successful the program they are attending is.  While it can indicate how good previous programs/events produced by that organization/group are, they don't bear directly on this particular program.  Yes, it's further true that attendance can indicate how excellently-targeted this particular demographic is and the type of program is, it may also be about the timing and location (vice versa for low attendance).
      But what about the program, itself?  How does one measure this?  Also, which type of program is it?  For example, if it's a shabbat dinner, we may measure one type of qualitative element; if it's a Jewish textual learning experience, it's another; and if it's a leadership-development program, perhaps it's another.  Perhaps we have different expectations of what we want to measure: are we measuring short-term effects, medium-term effects, or long-term effects?

 - - - - - 

III.  [Some] Types of Qualitative Data
      In August, through the discussions I was having with people, I realized that there are different types of qualitative data.  Here are some: specific quantified qualitative data, oblique quantified data about participation in event or practice, binary quantified qualititative data of beliefs or activities, and vignettes/stories as non-quantitative data.
      Here is what I mean about that linguistic gobbledy-gook above (perhaps they're neologistic terminology...): specific quantified qualitative data - if you can pardon the seeming paradoxical element - is measuring how good the event was or how much a given person (or certain ones or all of them) enjoyed or developed due to the event.  This can be measured on a particular numerical scale (1-5, 1-10, 1-100, etc.) and crunching that data.
      Oblique quantified data about participating in event or practice is measuring how many people are now doing some particular practice; this can include (but in now way limited to) lighting shabbat candles, observing shabbat, marrying a Jew/Jewess, or giving tzedakah.  This isn't so much qualitative, per se, exactly, but can reflect how impactful a particular activity/event/experience was.
      Binary quantified qualitative data is similar to the above, (and, frankly, I don't remember what I meant when I wrote this down 9 months ago...).
      Vignettes/stories as non-quantitative data is interesting in that it's not numerical data, but rather can be sharing a story about someone who's life was greatly effected/enhanced/ameliorated by the participation in the event/program/etc.

 - - - - - 

IV.  [Some] Problems with Collecting Qualitative Data
      As I mentioned in August, that

even if there is a way to measure it, when and how ought we go about doing it?  Is it simply an entry survey and an exit survey to see if their leadership skills or Jewish identity has increased?  If so, is it a binary measurement of yes/no or is it more complex, even something as seemingly simple as measuring it on a scale of 1-10.  Let's say you try this and you are able to get information such as "There was a 56% increase in Jewish identity from this event", what does that mean?  Furthermore, how does that translate?  
But the biggest question mark is that this work generally is actually development work (no, not fundraising), such that we are seeing how we can provide these ideas, activities, etc. to develop in these [young (although not necessarily so)] people.  Thus, even if they can tell us how they were affected, impacted, or helped through such programming immediately afterwards, what about six months down the road?  One year later?  What about 5, 10, 15 years later?

There are certainly issues with not only the data that result through the arbitrary self-reporting of those attending at the moment, but also how is the data collected weeks, months, or years later?  How many of those who attended will respond?  How much can the attendees attribute their growth (or, perhaps, lack of it) to that particular program (especially years later)?  
    But this is only just the beginning of the problem: do the staff members of those groups/organizations spend the time and resources on gathering and going back and continually gathering this information?  That could be time and labor-intensive.  What about hiring outside evaluators, which will be also money-intensive?  And how will those evaluators be able to stick to that group and how will they be remunerated for those programs?  Also, can we say 10-15 years later: that particular program was great, but that other one wasn't based on longitudinal study?  Even if longitudinal studies are launched for the Jewish community, how sustainable are they?  Lastly, are funders able/willing to wait a few years to hear about their investment in the development of these [young (but not necessarily)] Jews?  I would imagine they not only want [more] immediate results, but also immediate data.  For those working with young Jews, whether in their teens, college years (or emerging adulthood), or 20s-30s, we know that their development doesn't just happen at a shabbat dinner; it takes time.  

 - - - - - 

V.  Moving Forward
     My vision is two-fold: the first of which is further discussion of not only the need and desirability (usefulness, even?) of qualitative data, but also the collection of it and publication of it.  The second is for donors to have a greater desiring of qualitative data over simply quantitative data.  Fortunately, since August when I first wrote about this topic, I have been happy to see Andrés Spokoiny writing about this need in various writings of his, whether in The Jewish Week, The Jewish Journal, or eJewish Philanthropy.  He is well-situated as not only a Jewish communal professional, but also someone who has access and influence to donors to encourage them to consider qualitative data.  For a couple of his examples, he wrote in eJewish Philanthropy that "Good measures are those that measure impact, especially long-term impact. Granted, this can be hard and expensive to measure. But measuring the wrong things is not the solution. Investing in capacity to allow organizations to measure themselves is critical for high-performing philanthropy."  He also wrote that

The tendency is to measure what we can (objective metrics), instead of what we should. For example, while this country was founded based on the concept that we all have the right to the “pursuit of happiness”, this is not anything we can objectively measure. We often need to be more creative and subjective in measuring impact. It is not only important how many kids attend Jewish camps, but also the quality of their experience.

  There is hope.