| Title: | Methods for Deliberation Analysis |
|---|---|
| Description: | An implementation of deliberative reasoning index (DRI) and related tools for analysis of deliberation survey data. Calculation of DRI, plot of intersubjective correlations (IC), generation of large-language model (LLM) survey data, and permutation tests are supported. Example datasets and a graphical user interface (GUI) are also available to support analysis. For more information, see Niemeyer and Veri (2022) <doi:10.1093/oso/9780192848925.003.0007>. |
| Authors: | Gustavo Kreia Umbelino [aut, cre] (ORCID: <https://orcid.org/0000-0001-7754-9606>), Francesco Veri [aut] (ORCID: <https://orcid.org/0000-0002-9295-6776>), Swiss National Science Foundation (SNSF) [fnd] |
| Maintainer: | Gustavo Kreia Umbelino <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.2 |
| Built: | 2026-06-02 09:33:58 UTC |
| Source: | https://github.com/gumbelino/deliberr |
format_dri_survey helps transform raw survey data into
useful objects for further manipulation with deliberr
format_dri_survey( survey_info = list(type = NA_character_, order = NA_integer_, statement = NA_character_, name = NA_character_, scale_max = NA_integer_, q_method = NA) )format_dri_survey( survey_info = list(type = NA_character_, order = NA_integer_, statement = NA_character_, name = NA_character_, scale_max = NA_integer_, q_method = NA) )
survey_info |
survey information needed to format DRI survey |
A list of survey info, including name, considerations
data, policies data, scale_max or the upper bound of
Likert-scale survey questions, and q_method which flags whether the
survey uses Q methodology
surveys for raw survey data formatting
dri_survey <- format_dri_survey(surveys[surveys$name == "acp", ]) dri_survey$name dri_survey$considerationsdri_survey <- format_dri_survey(surveys[surveys$name == "acp", ]) dri_survey$name dri_survey$considerations
get_dri calculates the deliberation reasoning index (DRI) for a group of
deliberation participants
get_dri(ic, adjusted = TRUE)get_dri(ic, adjusted = TRUE)
ic |
dataframe generated by |
adjusted |
a logical indicating whether to use the original or adjusted DRI calculation formula |
the group-level DRI value
get_dri_ic() to generate ic parameter
Other IC methods:
get_dri_ind()
# get pre-deliberation (stage_id == 1) data from BEP case data <- human_data[human_data$stage_id == 1 & human_data$case == "BEP", ] # calculate IC ic <- get_dri_ic(data) # generate DRI get_dri(ic) # same as the mean of individual DRIs mean(get_dri_ind(ic)$dri)# get pre-deliberation (stage_id == 1) data from BEP case data <- human_data[human_data$stage_id == 1 & human_data$case == "BEP", ] # calculate IC ic <- get_dri_ic(data) # generate DRI get_dri(ic) # same as the mean of individual DRIs mean(get_dri_ind(ic)$dri)
get_dri_alpha calculates the internal consistency of DRI survey responses
using Cronbach's alpha
get_dri_alpha(data)get_dri_alpha(data)
data |
the raw DRI survey response data |
a dataframe with alpha_c, alpha_p, and alpha_all with values
of Cronbach's alpha for considerations, policy preferences, and both,
respectively
human_data for raw survey response data formatting
psych::alpha() for details on Cronbach's alpha calculation
Other DRI survey methods:
get_dri_case(),
get_dri_ic()
# get pre-deliberation (stage_id == 1) data from Mayo case data <- human_data[human_data$stage_id == 1 & human_data$case == "Mayo", ] get_dri_alpha(data)# get pre-deliberation (stage_id == 1) data from Mayo case data <- human_data[human_data$stage_id == 1 & human_data$case == "Mayo", ] get_dri_alpha(data)
get_dri_case calculates the pre- and post-deliberation DRI from a specific
deliberation case
get_dri_case( case, adjusted = TRUE, method = "wilcox", alternative = "greater", data = NULL )get_dri_case( case, adjusted = TRUE, method = "wilcox", alternative = "greater", data = NULL )
case |
a character string specifying the name of the case in human_data |
adjusted |
a logical indicating whether you want the original or adjusted DRI formula |
method |
a character string specifying the method for statistical testing, must be one of "wilcox" (default) or "t.test" |
alternative |
a character string specifying the alternative hypothesis, must be one of "greater" (default), "two.sided" or "less". You can specify just the initial letter |
data |
a survey responses dataframe, must include pre-
( |
a tibble with with the following components: case, pre,
post, delta, p_value, and significance
human_data for raw survey response data formatting
Other DRI survey methods:
get_dri_alpha(),
get_dri_ic()
get_dri_case("Activate") # same as get_dri_case("Activate", data = human_data)get_dri_case("Activate") # same as get_dri_case("Activate", data = human_data)
get_dri_ic calculates the intersubjective consistency (IC) between
deliberation participants using their DRI survey responses
get_dri_ic(data)get_dri_ic(data)
data |
the raw DRI survey response data |
dataframe with pnums or participant numbers, pnum1 and pnum2,
or the unique number of participant 1 and 2, respectively, ccor and pcor, or
the correlation between considerations statement ratings and policy
preference rankings, respecively, and dj, or the modal orthogonal distance
d for a given pair j
human_data for raw survey response data formatting
Other DRI survey methods:
get_dri_alpha(),
get_dri_case()
# get post-deliberation (stage_id == 2) data from Zukunft case data <- human_data[human_data$stage_id == 2 & human_data$case == "Zukunft", ] get_dri_ic(data)# get post-deliberation (stage_id == 2) data from Zukunft case data <- human_data[human_data$stage_id == 2 & human_data$case == "Zukunft", ] get_dri_ic(data)
get_dri_ind calculates the DRI for each individual participant at a
deliberation
get_dri_ind(ic, adjusted = TRUE)get_dri_ind(ic, adjusted = TRUE)
ic |
dataframe generated by |
adjusted |
a logical indicating whether to use the original or adjusted DRI calculation formula |
tibble with pnum as participant number and their respective
individual dri
get_dri_ic() to generate ic parameter
Other IC methods:
get_dri()
# get post-deliberation (stage_id == 2) data from Zukunft case data <- human_data[human_data$stage_id == 2 & human_data$case == "Zukunft", ] # generate IC ic <- get_dri_ic(data) # get individual DRIs get_dri_ind(ic)# get post-deliberation (stage_id == 2) data from Zukunft case data <- human_data[human_data$stage_id == 2 & human_data$case == "Zukunft", ] # generate IC ic <- get_dri_ic(data) # get individual DRIs get_dri_ind(ic)
get_dri_llm_response uses https://openrouter.ai to generate artificial LLM
responses to DRI survey questions
get_dri_llm_response( model_id, survey_info = list(type = NA_character_, order = NA_integer_, statement = NA_character_, name = NA_character_, scale_max = NA_integer_, q_method = NA), api_key = Sys.getenv("OPENROUTER_API_KEY"), role_info = list(uid = NA_character_, role = NA_character_, description = NA_character_), n = 1, temperature = NA_real_, reasoning = NULL, request_log_path = NA_character_ )get_dri_llm_response( model_id, survey_info = list(type = NA_character_, order = NA_integer_, statement = NA_character_, name = NA_character_, scale_max = NA_integer_, q_method = NA), api_key = Sys.getenv("OPENROUTER_API_KEY"), role_info = list(uid = NA_character_, role = NA_character_, description = NA_character_), n = 1, temperature = NA_real_, reasoning = NULL, request_log_path = NA_character_ )
model_id |
a model_id string from openrounter.ai |
survey_info |
a list with survey question information, including |
api_key |
the API key generated by OpenRouter |
role_info |
a named list with basic data of a role (i.e., |
n |
the number of responses requested (default = 1) |
temperature |
an optional numeric value between 0 and 2 controlling
the randomness of model output. When |
reasoning |
an optional named list controlling extended thinking for
models that support it. Supply exactly one of |
request_log_path |
an optional path to a file where the request texts are saved |
a dataframe with n survey responses by model_id, including a
unique identifier, uuid, a creation timestamp, created_at_utc, the time
it took to generate the response, time_s, the estimated cost in USD,
est_cost_usd, the reasoning configuration used, reasoning_effort and
reasoning_max_tokens (NA when disabled), whether the response is valid,
is_valid, and the reason it is not, invalid_reason
get_model_ids() for all currently available model ids from
openrouter.ai
Other LLM methods:
get_model_ids(),
make_dri_llm_prompts()
# get DRI survey survey_info <- surveys[surveys$name == "acp",] # select a model from openrouter model_id <- "google/gemini-2.5-flash-lite" # send request to openrouter API ## Not run: llm_data <- get_dri_llm_response(model_id, survey_info) # with reasoning via effort level (OpenAI-style, e.g. openai/o4-mini) llm_data_r <- get_dri_llm_response(model_id, survey_info, reasoning = list(effort = "low")) # with reasoning via token budget (Anthropic-style, e.g. anthropic/claude-opus-4-5) llm_data_r <- get_dri_llm_response(model_id, survey_info, reasoning = list(max_tokens = 2000)) ## End(Not run)# get DRI survey survey_info <- surveys[surveys$name == "acp",] # select a model from openrouter model_id <- "google/gemini-2.5-flash-lite" # send request to openrouter API ## Not run: llm_data <- get_dri_llm_response(model_id, survey_info) # with reasoning via effort level (OpenAI-style, e.g. openai/o4-mini) llm_data_r <- get_dri_llm_response(model_id, survey_info, reasoning = list(effort = "low")) # with reasoning via token budget (Anthropic-style, e.g. anthropic/claude-opus-4-5) llm_data_r <- get_dri_llm_response(model_id, survey_info, reasoning = list(max_tokens = 2000)) ## End(Not run)
get_llm_response sends a prompt to a specified large language model (LLM)
through the OpenRouter.ai API and returns the text response. It can also
manage conversation history
get_llm_response( user_prompt, model_id = "x-ai/grok-3-mini", system_prompt = NA_character_, context = NULL, temperature = NA_real_, reasoning = NULL, api_key = Sys.getenv("OPENROUTER_API_KEY") )get_llm_response( user_prompt, model_id = "x-ai/grok-3-mini", system_prompt = NA_character_, context = NULL, temperature = NA_real_, reasoning = NULL, api_key = Sys.getenv("OPENROUTER_API_KEY") )
user_prompt |
a string containing the prompt or question for the model |
model_id |
a string specifying the model to use (e.g., "google/gemini-flash-1.5"). You can find model names on the OpenRouter.ai website |
system_prompt |
A string defining the role or behavior of the model. This is only used for the first message in a conversation (when 'context' is NULL) |
context |
a list representing the conversation history. If provided, the 'system_prompt' is ignored, as the context is assumed to contain the full history. Defaults to NULL for a new conversation |
temperature |
an optional numeric value between 0 and 2 that controls the
randomness of the model's output. Higher values mean more "creative"
responses. When |
reasoning |
an optional named list controlling extended thinking for models that support it. Supply exactly one of:
When |
api_key |
a string containing your OpenRouter.ai API key. It is strongly
recommended to use the default, which retrieves the key from an
environment variable named |
a list containing four elements: response, context, usage, and
reasoning. usage is the raw usage object returned by OpenRouter,
including usage$cost (total USD charged, accounting for prompt,
completion, reasoning, and cached tokens) and token count fields.
reasoning is a character string with the model's reasoning summary
when reasoning is set and the model returns reasoning content;
otherwise NA_character_
## Not run: # Make sure to set your API key first # Sys.setenv(OPENROUTER_API_KEY = "your_api_key_here") # First turn of the conversation first_turn <- get_llm_response( user_prompt = "What are the three main benefits of using R for data analysis?", model_id = "x-ai/grok-3-mini", system_prompt = "You are a helpful assistant who provides concise answers." ) cat("--- Initial Response ---\n") cat(first_turn$response) # With reasoning enabled using effort level (OpenAI-style) first_turn_reasoning <- get_llm_response( user_prompt = "What are the three main benefits of using R for data analysis?", model_id = "openai/o4-mini", system_prompt = "You are a helpful assistant who provides concise answers.", reasoning = list(effort = "low") ) cat("--- Reasoning Summary ---\n") cat(first_turn_reasoning$reasoning) cat("\n--- Response ---\n") cat(first_turn_reasoning$response) # With reasoning enabled using token budget (Anthropic-style) first_turn_reasoning_tokens <- get_llm_response( user_prompt = "What are the three main benefits of using R for data analysis?", model_id = "anthropic/claude-opus-4-5", system_prompt = "You are a helpful assistant who provides concise answers.", reasoning = list(max_tokens = 2000) ) # Follow-up question using the context from the first turn second_turn <- get_llm_response( user_prompt = "Can you elaborate on the second benefit you mentioned?", model_id = "x-ai/grok-3-mini", context = first_turn$context ) cat("\n\n--- Follow-up Response ---\n") cat(second_turn$response) ## End(Not run)## Not run: # Make sure to set your API key first # Sys.setenv(OPENROUTER_API_KEY = "your_api_key_here") # First turn of the conversation first_turn <- get_llm_response( user_prompt = "What are the three main benefits of using R for data analysis?", model_id = "x-ai/grok-3-mini", system_prompt = "You are a helpful assistant who provides concise answers." ) cat("--- Initial Response ---\n") cat(first_turn$response) # With reasoning enabled using effort level (OpenAI-style) first_turn_reasoning <- get_llm_response( user_prompt = "What are the three main benefits of using R for data analysis?", model_id = "openai/o4-mini", system_prompt = "You are a helpful assistant who provides concise answers.", reasoning = list(effort = "low") ) cat("--- Reasoning Summary ---\n") cat(first_turn_reasoning$reasoning) cat("\n--- Response ---\n") cat(first_turn_reasoning$response) # With reasoning enabled using token budget (Anthropic-style) first_turn_reasoning_tokens <- get_llm_response( user_prompt = "What are the three main benefits of using R for data analysis?", model_id = "anthropic/claude-opus-4-5", system_prompt = "You are a helpful assistant who provides concise answers.", reasoning = list(max_tokens = 2000) ) # Follow-up question using the context from the first turn second_turn <- get_llm_response( user_prompt = "Can you elaborate on the second benefit you mentioned?", model_id = "x-ai/grok-3-mini", context = first_turn$context ) cat("\n\n--- Follow-up Response ---\n") cat(second_turn$response) ## End(Not run)
get_model_ids uses OpenRouter to get provider and model names. The
model_id can be recreated as provider/model. It also returns one binary
column for each of the key supported parameters: temperature, reasoning,
include_reasoning, reasoning_effort, and max_tokens
get_model_ids()get_model_ids()
a dataframe with columns provider, model, and one logical column
per tracked parameter (temperature, reasoning, include_reasoning,
reasoning_effort, max_tokens), indicating whether the model supports it
Other LLM methods:
get_dri_llm_response(),
make_dri_llm_prompts()
get_model_ids()get_model_ids()
Pre- and post-deliberation DRI survey data from 24 deliberation cases around the world. Some cases used the same survey.
human_datahuman_data
A data frame with 67 variables, including
survey, case, stage_id, C1...C50 and P1...P10.
make_dri_llm_prompts creates the system and user prompts used for
generating LLM DRI survey data
make_dri_llm_prompts( dri_survey, role_info = list(uid = NA_character_, role = NA_character_, description = NA_character_) )make_dri_llm_prompts( dri_survey, role_info = list(uid = NA_character_, role = NA_character_, description = NA_character_) )
dri_survey |
a list of formatted DRI survey questions |
role_info |
information about a specific role, including unique
identifier |
a list of lists with four variables: system,
considerations, policies, and reason prompts
format_dri_survey() for how to format dri_survey
prompts for how prompts are formatted
Other LLM methods:
get_dri_llm_response(),
get_model_ids()
# get ccps as an example survey dri_survey <- format_dri_survey(surveys[surveys$name == "ccps",]) # create an example role from scratch role_info <- list( uid = "sur", role = "surfer", description = "likes the ocean" ) make_dri_llm_prompts(dri_survey, role_info)# get ccps as an example survey dri_survey <- format_dri_survey(surveys[surveys$name == "ccps",]) # create an example role from scratch role_info <- list( uid = "sur", role = "surfer", description = "likes the ocean" ) make_dri_llm_prompts(dri_survey, role_info)
open_gui uses shiny to open an interactive interface for code-free DRI
analysis
open_gui()open_gui()
NA
## Not run: open_gui() ## End(Not run)## Not run: open_gui() ## End(Not run)
permute_dri tests whether the links between considerations and policy
preferences are consistent or likely due to chance
permute_dri(data, iterations = 10000, verbose = FALSE, summary = TRUE)permute_dri(data, iterations = 10000, verbose = FALSE, summary = TRUE)
data |
raw DRI survey dataframe |
iterations |
number permutations to generate |
verbose |
a logical flag to print time of permutation |
summary |
a logical indicating whether to return the raw data or summary of test results; raw data is optimal for plotting permutation results |
dataframe with permutation test results, raw or summarized.
Summarized results include the number of participants, n, the observed DRI,
obs_dri, the number of permutations conducted, n_perm, the mean
permutation DRI, mean_perm_dri, and the frequency which the permutation DRI
is greater or equal to the observed DRI, p
# get pre-deliberation (stage_id == 1) data from Zukunft case data <- human_data[human_data$stage_id == 1 & human_data$case == "Zukunft", ] # permute DRI 100 times permute_dri(data, iterations = 100)# get pre-deliberation (stage_id == 1) data from Zukunft case data <- human_data[human_data$stage_id == 1 & human_data$case == "Zukunft", ] # permute DRI 100 times permute_dri(data, iterations = 100)
plot_dri_ic creates a dot plot of deliberation IC where each dot represents
a pair of participants
plot_dri_ic( ic, title = NA_character_, suffix = NA_character_, dri = NA_real_, caption = NULL )plot_dri_ic( ic, title = NA_character_, suffix = NA_character_, dri = NA_real_, caption = NULL )
ic |
dataframe generated by |
title |
title of the plot |
suffix |
string to be added after the title separated by |
dri |
numeric value generated by |
caption |
a string to be displayed under the plot |
an IC plot
get_dri_ic() for how to generate the ic parameter
get_dri() for how to generate the dri parameter
# get post-deliberation (stage_id == 2) data from Zukunft case data <- human_data[human_data$stage_id == 2 & human_data$case == "Zukunft", ] # set plot optional parameters title <- "Case Zukunft" suffix <- "Post-Deliberation IC Plot" caption <- "this is an example plot" # calculate ic ic <- get_dri_ic(data) plot_dri_ic(ic, title, suffix, caption = caption)# get post-deliberation (stage_id == 2) data from Zukunft case data <- human_data[human_data$stage_id == 2 & human_data$case == "Zukunft", ] # set plot optional parameters title <- "Case Zukunft" suffix <- "Post-Deliberation IC Plot" caption <- "this is an example plot" # calculate ic ic <- get_dri_ic(data) plot_dri_ic(ic, title, suffix, caption = caption)
Prompts used in DRI surveys for humans and LLMs.
promptsprompts
A data frame with two variables:
type and prompt
Roles used to generate LLM role-playing data.
rolesroles
A data frame with five variables:
uid, type, article, role and description.
summarize_perm_dri summarizes the results of a permutation test done using
permute_dri(..., summary = FALSE); useful for summarizing results after
plotting permutation results
summarize_perm_dri(perms, type = "common")summarize_perm_dri(perms, type = "common")
perms |
results of the permutation test generated by |
type |
which type of statistics to summarize (e.g., |
summary of permutation test
permute_dri() for generating the perms parameter
rstatix::get_summary_stats() for values of type
# get pre-deliberation (stage_id == 1) data from Zukunft case data <- human_data[human_data$stage_id == 1 & human_data$case == "Zukunft", ] # create permutations perms <- permute_dri(data, iterations = 100, summary = FALSE) summarize_perm_dri(perms)# get pre-deliberation (stage_id == 1) data from Zukunft case data <- human_data[human_data$stage_id == 1 & human_data$case == "Zukunft", ] # create permutations perms <- permute_dri(data, iterations = 100, summary = FALSE) summarize_perm_dri(perms)
All survey data used in deliberations.
surveyssurveys
A data frame with six variables:
type, order, statement, name, scale_max and q_method.