Creating and Managing Weight Models

A normal survey will elicit responses from a very small (relatively) sub-set of the general population. Also, in most cases, not all of the intended respondents will reply to the survey. The resulting respondent data may therefore need to be weighted to adjust the data presented in the report such that it reflects the "real world" situation. Authoring enables you to create weight models for a survey that can then be applied to the reports based on that survey. The applied weight models are included in the result calculations between the real respondent data and the report, to adjust the data as required.

Weight models may be applied to the entire report, or to individual elements in the report such as folders, pages and tables.

Note: Weight models applied lower in the report hierarchy override those that may be applied higher up. For example a model applied to a folder will override one applied to the report.

It is not possible to apply weighting on questions that were answered only by a part of the respondents, for example questions in conditions.

Weighting is only applied to complete records.

Creating a New Weight Model

To create a new weight model for a survey:

1. Go to the Reporting > Weighting menu command.

The Weight Model List window opens with the weight models currently available for this survey - . If no weight models currently exist, then the list will be empty. To edit an existing weight model, click on the appropriate blue Weight ID link to open it.



Figure 1 - Example of the Weight Model List

2. Click the New Weight Model button located towards the right end of the window's toolbar.

The Weight Model page opens at the Overview tab - .



Figure 2 - The Weight Model page

On this tab you define an ID and a Name as required. The system will define an ID by default; this can be changed as required.

Note: A weight model ID must be all lower case, it must be a maximum of 13 characters including any separators, and it cannot have the same ID as a question ID in the survey. If a weight model has the same ID as a question ID used in the survey, the weight model generation will fail and an error will be generated.

3. On completion, click Save to save the changes.

The weight model is created and the remaining tabs are activated.

Note: Once the weight model is created, you can save changes in the Weight Model window either by clicking the Save button in the window's toolbar or by moving to a different tab; changes are then saved automatically.

4. Go to the Questions tab - .

On this tab you select which questions are to be used in the weight model. Note that the questions listed in this tab will depend on the model Type selected in the Overview tab. For Sample Balancing and Cell Matching, only Single questions can be used in a weight model; for Survey Question, only Open Numeric questions can be used. Questions that reside within loops cannot be selected.



Figure 3 - Example of the Questions tab

5. Select the questions to be used.

There are two methods of selecting the questions; click on the question in the Available Questions list to select it then click the >> button to move it to the Weight Model Questions list, or double-click on the question in the Available Questions list. Use the same methods to remove an unwanted question from the Weight Model Questions list.

Note: The order in which the questions are selected will be the order in which the questions are used in the weight model.

6. On completion click Save then go to the Settings tab.

The options on this tab will depend on the Type selected on the Overview tab - .



Figure 4 - The Settings tab when Sampling Balancing is the selected Type

A Target set will be presented for each question selected in the Questions tab. The weights input for each Target set must total 100%.



Figure 5 - The Settings tab when Cell Matching - Percentage is the selected Type



Figure 6 - The Settings tab when Cell Matching - Weight is the selected Type

Note: If the Survey Question type is selected, there are no further settings to make.

7. Add the appropriate values to the fields in the Settings tab, then click Save to save the changes.
8. Click the Calculate button in the window toolbar to open the Calculate Weights dialog - .



Figure 7 - The Calculate Weights dialog

9. Either leave the Schedule as ASAP and click OK to add the task to the queue immediately, or select Schedule for Later Execution and then click OK to open the Recurring Task window (go to Recurrence Tab for more information). Here you can set up the task to run at a specific date and time, and/or repeat as required.
10. On completion, click Close Window (located towards the lower edge of the Weight Model page) to return to the Weight Model List.

Overview

The properties and fields on the Overview tab are as follows:

• ID - the identification number for the weight model. This is defined automatically by the system when the model is created, but can be changed by the user as required.

Note: A weight model cannot have the same ID as a question ID in the survey. If a weight model has the same ID as a question ID used in the survey, the weight model generation will fail and an error will be generated (go to Question ID for more information).

• Name - the "logical" name for the weight model. This is defined automatically by the system when the model is created, based on the next available ID number. (Note that if you change the ID from the system-generated value, then that number will become available to be used in the Name the next time a model is created). The Name can be changed by the user as required.
• Type - specify the type of weight model to be used. The Type is selected in the Weight Matrix tab. Four types of weight models are available:
• Sample balancing - used for weighting along multiple dimensions when only the marginal reference percentages along each dimension are known. For example: It is known that the reference population is 30% male and 70% female and that 60% of the population own a car while 40% do not. From these facts the actual weights are calculated according to the standard sample-balancing algorithm. Max iterations and deviation limit for this algorithm decide how accurate the calculations are required to be. Note that the percentage values entered must total to 100%.
• Cell matching - percentage - used for weighting along multiple dimensions when all the matrix cell percentages are known. For example, it is known that 20% are male with cars, 10% are male without cars, 40% are female with cars, 30% are female without cars. From these facts, the weights can be calculated by dividing the sample percentages by the reference percentages. Note that the percentage values entered must total to 100%.
• Cell matching – weight - with this method all the weights are entered directly and not calculated by comparing a reference with the sample. The method may for example be used if the weights are calculated by another system.
• Survey Question - this method provides the user with the ability to directly use weight factors stored in a survey question for each respondent, instead of factors resulting from the other weight model types. The weight factors can for example be imported into a hidden question in the response data set. This enables you to calculate weight factors outside Authoring and apply them directly to reports in Reportal, for types of weighting not supported by the current weight models in Authoring.
• Status - the status of the weight model. Once the model has been generated then the Last Log field will be populated with the appropriate data.
• Last Log - shows the details of the last weight model generation task that has been performed on the current model.

Settings

The properties and fields on the Settings tab are as follows:

• Set First Question Fixed - in the event you wish to report on data divided into time periods, for example months, then you will probably wish to weight the data for each time period separately rather than weight the data set as a whole. Check this box to keep the first question with fixed values.
• Max Iterations - the maximum number of times the iteration is to run. This will restrict the time taken for the process to run in the event the desired values are difficult to achieve.
• Deviation Limit - the acceptable accuracy tolerance. Once the system has come within this limit the iteration process will stop.
• Targets - set the target values the system is to aim for during the iteration process.

Applying a Weight Model to a Report

Weight models in Authoring are created for a specific survey. Once created they can then be applied to reports or elements in reports (folders, pages or tables) that are based on that survey. The weight models must be defined in Authoring before they can be used in the report.

Each element in the report that can have a weight model applied to it has the Weight Model property in the element's Properties sheet. The drop-down in this property contains a list of the weight models defined for the survey on which the report is based. In Reportal, go to the appropriate element's Properties sheet, select the required weight model from the drop-down list and save the change to apply the model to that element in the report. Refer to the Reportal documentation for further information on using weight models.

Note: Weighting is only applied to complete records.

Important
In the event a weighting model is based on a survey question that for some data records is not answered (has the value NULL), then the results published in a report will differ depending on whether the BitStream query engine or the SmartHub query engine is used. If weights are missing, the BitStream engine defaults to 1 while the SmartHub engine defaults to 0 and therefore excludes these records from aggregated tables.

Duplicating a Weight Model

To duplicate a weight model:

1. When in a survey, go to the Reporting > Weighting menu command.

The Weight Model List page opens.

2. Check the boxes for the weight models you wish to duplicate, then click Duplicate.

The selected weight models are duplicated, and given the name "Copy of [original name]". You can now edit the new weight model as required.

Weighting Different Sub-samples Separately

In continuous surveys where you report on different time periods (for example week, month, quarter or year) you may wish to weight each time period separately to match the same target distribution on some variables, keeping those calculated weights for each period. This can be achieved by using the "First question fixed" property on the weight model.

For example, assume that you are weighting on age and gender to match the distribution shown below:

Figure 8 - Distribution targets

Note: Under 18s have been screened in the example survey.

When reporting on this survey, we will be looking at results per month. In the survey there is a "month" question where the month of the response is set. This question will be used to indicate the sub samples to which the weighting should be applied. We want to apply weights to achieve the same distribution on gender and age for each month - .

Figure 9 - Extract from the answer list of the month question

In the weight model, month will then have to be included as the first object - :

Figure 10 - Weight model order of questions selected

In weight model properties, the "First question fixed" property is set. This means that weights will be calculated for gender and age for each of the values of the month question separately.

Now, if you double-click on the month question in the weight model, you can set which codes (which months) are to be available for weight calculation. Typically the answer list will include the entire year, but weights should only be calculated for months that have passed - .

Note: It is only possible to calculate a sub-sample for which data has already been collected.

Figure 11 - Set active codes in the "fixed question", month

When calculating a weight model with a "fixed question", you can select one of three options - :

• Calculate weights for all of the codes selected.
• Calculate weights for a subset of them (only specific codes).
• Calculate weights for those codes that have been activated since the last weight calculation (only new codes), keeping the existing weights for the other codes.

Figure 12 - Select which codes the weights are to be calculated for in the "fixed question"