Organise the ‘messiness’ of qualitative analysis: demonstrating the audit trail in NVivo

In this post, I will discuss how a well organised NVivo project can be used to provide an audit trail in qualitative data analysis.

Specifically, I will focus on what to include in the audit trail and provide ideas on how NVivo can be used to construct the audit trail.  To add clarity to the discussion, I will use examples from my own PhD research.

What is an audit trail?

An audit trail can add methodological rigour to a study.  This is because it allows you to show (sceptics, reviewers, examiners) a clear series of events that led to the emergent findings.  These steps should be retractable to the original raw data to enlighten potential auditors on how you came to your final conclusions—the audit trail adds integrity to the analysis.

To me, the whole NVivo project has the potential to effectively serve as the audit trail.

As guidance, Lincoln & Guba (1985) have suggested six elements that comprise an audit trail.  These are outlined in Table 1 below, which also provides examples on each element.

 Table 1. What is an audit trail? (Adapted from Lincoln & Guba, 1985)

In consideration of the above elements, I feel that organisation and a systematic approach in the NVivo project are essential to the development of an audit trail.  Otherwise, a messy NVivo project exacerbates the messiness of the analysis process.

Organising your project: dropping the ‘bread crumbs’

The first step is to structure your NVivo folders to correspond with the audit trail elements.

This can be achieved by the systematic organisation of the NVivo structural folders (e.g. sources, nodes, models), to create sub folders and sub-sub folders.   I would name sub-folders as if they were headings in a thesis: each folder numbered and named in a chronological fashion.  Imagine linking this to the same chapters in a thesis – it would actually bridge your NVivo project closer to your thesis.  I would repeat this process for the internals, memos, and nodes (see Figure 1 for example).

Figure 1. Organized sources and nodes

Within each ‘chapter’ folder I would then create sub-sub folders.  The content here would of course vary across the internals, memos and nodes.  These sub-sub folders would represent chapter sections in a thesis and would contain varying information to support the audit trail.  They would be numbered and named as if they were sub-sections within a chapter.

It is within each relevant sub-sub folder that you can begin to create, or import relevant research documents to support the audit trail.  For example, you could import into NVivo your reflexive diary, method protocol, participant demographics, interview questions, etc.

To demonstrate, Figure 1 presents a screen shot from my NVivo project that I used in my thesis.  To add context, at a voluntary sector mental health organisation, I ran three qualitative studies: (a) semi-structured interviews, (b) focus groups, and (c) a photo elicitation method.

I included all three studies within the same project because I believed that this would allow a more analytical conception on the differences and similarities among the studies, which I could then draw upon in my discussion chapter. Furthermore, this project was located from a constructivist research perspective (Lincoln, Lynham, & Guba, 2011), which also utilised a thematic analysis (Braun & Clarke, 2006).

Spotting the ‘bread crumbs’ in the emergent findings

To elaborate further on Figure 1, the photo elicitation study has been chosen to provide a specific example to highlight the audit trail in the emergent qualitative analysis.  Because I had adopted Braun & Clarke’s (2006) recommendations on thematic analysis, the sub-sub folders were created to demonstrate each step taken during this method.  This is shown below in Table 2.

 Table 2. Demonstrating the audit trail during a thematic analysis

As the analyst, I have found that such organisation was really useful to help me understand and make sense of the qualitative data. Analysis is often a process of moving backwards and forwards.

This structure facilitated the fluidity of movement as it allows my previous thinking (memos, codes) to influence my current emergent thinking.  For instance, I could revisit earlier codes to reconsider them in later stages of analysis.  Such important decisions are further logged and compiled in the audit trail.  In addition, I would also code all relevant text from my earlier memos on reflexivity and emergent themes, into the final constructed themes.  Including these notes here, within the same node, invites the reader to view all the ‘evidence’ in the same place.

In addition, to benefit the potential auditor, this structure provides a user friendly map of the analysis to allow someone easily to evaluate the emergent findings that occurred during all phases of analysis.  Otherwise, a messy and unclear project will fail to highlight important elements to satisfy the audit trail.  When organised effectively, for example, the properties of NVivo might enable someone to see:

Raw data

• Easy access to the raw data (transcript, audio, photos, videos) to support each code/theme (“open referenced source”, or from the sources and internals organised sub-folders)

 Data reduction and analysis products, data reconstruction and synthesis products

• My analytical thinking and decision making during the analysis (memos)
• Easy access to linked memos from nodes to view my reflexive notes per phase of the analysis, or my notes on emergent themes
• Easy access to view the segmented text that characterises each code/theme.
• See how my initial open codes developed during each phase of the analysis

 Process notes, materials relating to intentions and dispositions & Instrument development

• See how the data analysis links to the wider aspects of a research project (methods, criteria of rigour, discussion, etc).  This can be done by coding your reflexive memos and emergent theme memos that correspond to sections of writing that you might use elsewhere in your project, other than the findings.  For instance, some of my reflexive notes talked about the authenticity of my findings (my criteria of rigour).  I coded these under nodes, 5.6 authenticity.  Subsequently, this also helped me to write this section in my thesis.
• Additionally, for example, I imported my interview questions, research questions, and methods procedure into sources, memos 5.1 photo project information.

Closing thoughts

This post presented one way of implementing an audit trail using NVivo.  The organisation and structure of any project will of course vary from person to paradigm to analysis method.  In my approach, I have related my NVivo project to the audit trail recommendations of Lincoln & Guba (1985).  It is hoped that this post can inspire people’s organised creativity to shape their project in an individual and rigorous manner.

However, some critics might question the suggested structure and organisation of a project as time consuming, unnecessary and thereby perhaps methodolatry (Chamberlain, 2000).  It is possible that all this hard work is potentially in vain; no one might come to view your NVivo project to provide an external audit.  However, for me, the most important thing that I have experienced from this procedure, has been the increased confidence that I have developed in my findings and NVivo competence.  A most valued and appreciated feeling when trying to understand the messiness of qualitative research.

References

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3, 77-101.

Chamberlain, K. (2000). Methodolatry and qualitative health research. Journal of health psychology, 5, 285-296.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry California: SAGE.

Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2011). Paradigmatic controversies, contradictions, and emerging confluences, revisited. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (4 ed.). California: Sage.

Further suggested reading

Bringer, J. D., Johnston, L. H., & Brackenridge, C. H. (2004). Maximizing transparency in a Doctoral Thesis1: The complexities of writing about the use of QSR*NVIVO within a grounded theory study. Qualitative research, 4, 247-265.

Sinkovics, R. R., & Alfoldi, E. A. (2012). Progressive focusing and trustworthiness in qualitative research: the enabling role of computer-assisted qualitative data analysis software (CAQDAS). Management international review, 52, 817-845.