Import Data

Jana Welling, Christian Diegel

2026-09-23

Required Data and Files

Scaling requires multiple datasets that usually need to be assembled manually. This section provides an overview of the most commonly required data and their typical locations. Exceptions may apply depending on the specific study.

Most relevant files can be found in:

//wipo.lifbi.de/daten/Projektgruppen_(08)/Kompetenzen_BA_Hiwi_(p000012)/Methoden/Skalierung/HE

Within each sub-study folder, SPSS files and codebooks are typically located in the rawdata folder (sometimes within preloads).

If a required file is missing, it should be requested.

Competence Data

Competence data are contained in SPSS files with xTargetCompetencies in the file name.

Required variables include the ID variable (ID_t) and all competence items. Item names can be found in the respective study documentation or codebook. Only data from the relevant wave should be retained.

Information on Competence Items

For scaling, additional item-level information is required, including:

This information must typically be extracted from test booklets, study design documentation, or survey materials and ideally compiled into an Excel file.

Relevant documents can often be found here:

//wipo.lifbi.de/daten/Projekte_(07)/NEPS-Kompetenzen_(14)/Netzwerk_(05)

Sociodemographic Data

In addition to competence items, sociodemographic variables are required for descriptive analyses and DIF analyses. Depending on the study type, these may include gender, migration background, cultural capital (e.g., number of books at home as part of socioeconomic status), school type (for students in secondary education), educational attainment (for adults), and age (for adults; in school samples this is typically homogeneous due to grade-level structure).

These variables can be found in SPSS files such as CohortProfile, pParent (children), or pTarget (adults).

Common variable names (may vary by study):

All other variables should be verified using the respective codebook.

Test Booklet Design Information

In addition to competence and sociodemographic data, information on survey design and test booklet structure is required.

Many studies use multiple test booklets (e.g., easy vs. difficult forms) and/or rotational designs in which different groups receive different test orders (e.g., Group 1: Reading → Mathematics; Group 2: Mathematics → Reading).

This information is typically stored in the SPSS file xTargetCompetencies in a variable whose name starts with tx and whose label contains “survey/test instrument”.

Since values are encoded numerically, value labels must be used to interpret them. In SPSS, these can be viewed in variable view.

In R, labels can be retrieved (after importing the data with haven) using:

attributes(dat$name_variable)$labels

Import Data

Information on Competence Items

First, the required data must be imported. We initially need certain information about the competence items (e.g., correct response, item type, position, test booklet; see Section Required Data and Files). This information should be identified beforehand for each unscored item and compiled in an Excel table so that it can now be loaded into the R environment as the data.frame vars. Pay attention to the naming conventions of the NEPSroutines package for the variables in the data.frame vars (see Section Required Data Structure in vignette package_introduction). Which items are included in the competency test can be found either in the codebook or in a document corresponding to the respective test (see Section Required Data and Files).

vars <- openxlsx::read.xlsx(file = "path/filename.xlsx") 

The NEPSroutines package additionally requires logical variables in the dataset that contain information about the item version (raw, dich, poly). Since only unscored items are available at this point, the assignment is straightforward and can be applied to all items simultaneously. Note that the variable poly is only required if the test also contains polytomous items.

vars$raw <- TRUE
vars$dich <- FALSE
vars$poly <- FALSE 

Competence Data

The raw competency item data can be found in the SPSS dataset xTargetCompetencies. This dataset is loaded into the R environment as a data.frame named resp using the function read_sav(). The argument use_na = FALSE prevents missing values specified in SPSS from being imported as NA. Very large datasets may cause memory issues; in such cases, using foreign::read.spss may help.

resp <- haven::read_sav(file = "path/filename.sav", use_na = FALSE) 

We only need those items that occurred in the competency test to be scaled and, for the DIF analyses, the variable indicating the test booklet design (tx...). These can be selected using the function select().

item_names <- vars$item 
design <- "NameDesignVariable"
resp <- dplyr::select(resp, ID_t, design, all_of(item_names)) 

Sociodemographic Data

In the DIF analyses, it is examined whether the items elicit comparable response behavior across different groups (e.g., test booklets A and B). Several sociodemographic and methodological variables are used as so-called DIF variables. The variables typically used in our scaling procedures are explained in Sections Sociodemographic Data and Test Booklet Design Information in vignette import_data. The raw data must be assembled from various SPSS files.

As with the competency data, the SPSS files are loaded into R using read_sav(), and the required variables (ID_t, DIF variables) are selected using select() (see previous section). In general, data from the current wave are used; however, missing values may also be supplemented using information from earlier waves. If multiple values are available (e.g., school-leaving qualification or socioeconomic status of the parents), the higher value is used.

The data.frame containing the new variables is merged with the data.frame resp using the merge() function.

resp <- merge(
  resp, 
  dif_vars[, c("ID_t", "dif_var1", "dif_var2", etc.)],  
  by = "ID_t",  
  all.x = TRUE
)